Apache Spark™ with Scala for PySpark Developers Workshop 5 Days

@jaceklaskowski / StackOverflow / GitHub
The "Internals" Books: Apache Spark / Spark SQL / Spark Structured Streaming

Jacek is best known by the online "Internals" books:

## Day 3. Spark SQL / Aggregration and Joins

1. Exercises
                * [Exercise: Difference in Days Between Dates As Strings](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Difference-in-Days-Between-Dates-As-Strings.html)
                * [Exercise: Using UDFs](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Using-UDFs.html)
              1. [Basic Aggregation](spark-sql-basic-aggregation.html)
                * [Exercise: Finding maximum values per group (groupBy)](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Finding-maximum-values-per-group-groupBy.html)
                * [Exercise: Collect values per group](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Collect-values-per-group.html)
                * [Exercise: Multiple Aggregations](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Multiple-Aggregations.html)
              1. [Joins](spark-sql-joins.html)
                * [Exercise: Finding Most Populated Cities Per Country](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Finding-Most-Populated-Cities-Per-Country.html)

## Day 4. Advanced Spark SQL

1. [The Internals of Structured Query Execution](spark-sql-internals-of-structured-query-execution.html)
              1. Exercises
                * [Exercise: Flattening Array Columns (From Datasets of Arrays to Datasets of Array Elements)](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Flattening-Array-Columns-From-Datasets-of-Arrays-to-Datasets-of-Array-Elements.html)
                * [Exercise: Using pivot for Cost Average and Collecting Values](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Using-pivot-for-Cost-Average-and-Collecting-Values.html)
              1. [Caching and Persistence](spark-sql-dataset-caching-and-persistence.html)
              1. [Windowed Aggregation](spark-sql-windowed-aggregation.html)
                * [Exercise: Finding 1st and 2nd Bestsellers Per Genre](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Finding-1st-and-2nd-Bestsellers-Per-Genre.html)
                * [Exercise: Calculating Gap Between Current And Highest Salaries Per Department](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Calculating-Gap-Between-Current-And-Highest-Salaries-Per-Department.html)
                * [Exercise: Calculating Running Total / Cumulative Sum](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Calculating-Running-Total-Cumulative-Sum.html)

Prerequisities

Be prepared to get the most out of the workshop

Prerequisities / Experience

Hands-on programming experience using Scala
Experience developing Spark applications

Prerequisities / To Be Installed

Java Platform, Standard Edition (Java SE) 8
Apache Spark
IntelliJ IDEA Community Edition with Scala plugin
sbt
Apache Kafka
Linux or macOS preferred
- Consult Running Spark Applications on Windows when on MS Windows

In-Class Preparations

Make Instructor's Life Slightly Easier. Thanks!

Introduce Yourself

First name
What's your experience with Spark?
Any production experience with Spark?
What do you expect from the workshop?
Where do you want to be with Spark after 5 days?

Addendum

Put your name on paper in front of you
- Stick to your laptop if possible
- Use LARGE fonts
1-hour lunch break at 12:45pm