Apache Spark aims to solve the problem of working with large scale distributed data -- and with access to over 500 million leaked passwords we have a lot of data to dig through.
Advancements in the API make running Spark with Scala, Python, or even SQL smoother and faster than ever. This talk will introduce you to Spark and the new way to run queries on structured, distributed data by looking at breached credentials. We'll walk through how to get started with Spark and discuss the tradeoffs for using different abstractions provided by the framework. With the help of live code, we'll find patterns in the password data and look at how you can encourage your users to be more secure. You will see how easy and fast it is to both explore and process data using Spark SQL and leave with the tools to get started with your own distributed data...and a password manager.
This talk assumes knowledge of Scala and SQL programming, but otherwise will be an introduction to the Spark and data engineering concepts.
Attendees will learn how to get started with Spark, the difference between 1.x and 2.x APIs, and gain familiarity with how to manipulate data. This is a practical, hands on talk that is designed to inspire attendees to play around with data themselves.
I've included an example setup for attendees of the presentation to play along with after they get the basics. You can download the project here: www.krobinson.me/files/intro-spark.zip. Once you unzip the folder, setup instructions are in the README.md. Setup may take about 15 minutes (longer if you need to install Java).
Kelley has worked in a variety of engineering roles, ranging from trading live cattle derivatives to building production data pipelines in Scala. She spends a lot of time thinking about how to make technical concepts accessible to new audiences. In her spare time, Kelley spends a lot of time cooking and greatly enjoys reorganizing her tiny kitchen to accommodate completely necessary small appliance purchases.