In this lesson, you see how Spark makes large scale data analysis much more accessible by providing application programming interfaces in languages familiar to data scientists.
We start by using Python and the PySpark interface to perform basic data analysis on a large dataset.
Throughout the lesson you learn how PySpark and SparkR interface with a Spark cluster and the Spark core framework, which is written in Scala.
With the SparkR API, you see how to reason about your data from a higher level using the DataFrame abstraction.
And finally, you see how by adding structure to your data with Spark SQL, you can construct complex queries to efficiently analyze your data.
- Understand how the extensibility the core Spark framework enables programming APIs to be developed in other languages
- Work with the PySpark API to perform an end-to-end analysis of flight delay data
- See how the DataFrame API in Spark allows for efficient analyses, both in terms of developer productivity and execution performance
- Aggregate and filter a large dataset with Spark to visualize locally with ggplot2
- Understand how to integrate existing SQL workflows with Spark
- 2.2 - 2.4: pyspark.ipynb
- 2.5 - 2.7: sparkR.ipynb
- 2.9: sparkSQL.ipynb