Day 1 — The Internals of Spark SQL

1. (Warm-up) Exercise: Finding Ids of Rows with Word in Array Column
2. Spark SQL
3. Data Structures (Dataset, DataFrame, RDD)
4. The Internals of Structured Query Execution
5. Columns and Dataset Operators

Day 2 — Spark SQL

1. Standard and User-Defined Functions
   • Exercise: How to add days (as values of a column) to date?
2. Basic Aggregation
   • Exercise: Multiple Aggregations
3. Joins
   • Exercise: Finding Most Populated Cities Per Country

Day 3 — Advanced Spark SQL

1. Windowed Aggregation
   • Exercise: Finding 1st and 2nd Bestsellers Per Genre
   • Exercise: Calculating Gap Between Current And Highest Salaries Per Department
   • Exercise: Calculating Difference Between Consecutive Rows Per Window

Day 4 — Spark SQL

1. Spark SQL / Exercises
   • split function with variable delimiter per row
   • Selecting the most important rows per assigned priority
   • Adding count to the source DataFrame

Day 5 — Spark SQL and Delta Lake

1. Spark SQL / Exercises
   • Limiting collect_set Standard Function
   • Structs for column names and values
   • Exercise: Using pivot for Cost Average and Collecting Values
2. Multi-Dimensional Aggregation
3. Introduction to Delta Lake

Prerequisities / Experience

1. Hands-on programming experience using Scala
2. Experience developing Spark applications

Prerequisities / To Be Installed

• Java Platform, Standard Edition (Java SE) 8
• Apache Spark
• IntelliJ IDEA Community Edition with Scala plugin
• sbt
• git
• Linux or macOS preferred
  • Consult Running Spark Applications on Windows when on MS Windows

Introduce Yourself

1. Say "Hi!" in the chat window

Schedule

1. 5 x 45-minute blocks / day
2. Mon-Tue
   • 9:00 - 11:00, 15' break, 11:30 - 13:00
3. Wed-Fri
   • 10:00 - 11:45, 15' break, 12:00 - 14:00

Addendum

1. Ask questions in the chat window