Apache Spark™
Advanced for Developers Workshop 5 Days

@jaceklaskowski / StackOverflow / GitHub
The "Internals" Books: Apache Spark / Spark SQL / Spark Structured Streaming

Jacek is best known by the online "Internals" books:

Professional Objectives

Advance in solving analytical problems using Spark SQL
Explore the recent features of Apache Spark 2.4
Deep dive into the internals of Apache Spark and the modules (Spark SQL, Spark Structured Streaming and Spark MLlib)
Understand performance tuning of Apache Spark applications and the advanced features of Apache Spark

Training content

Anatomy of Spark Core Data Processing Platform
Foundations of Spark SQL
Internals of Structured Query Execution
Standard, User-Defined and User-Defined Aggregate Functions (Spark SQL)
Basic, Windowed and Multi-Dimensional Aggregations (Spark SQL)
Monitoring Spark Applications Using web UI and SparkListeners
Join Optimization with Bucketing (Spark SQL)
Stream Processing with Spark Structured Streaming
Machine Learning with Spark MLlib

## Day 2 — Spark SQL

1. [Standard and User-Defined
              Functions](spark-sql-standard-functions-udfs.html)
                * [Exercise: Using upper Standard Function](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Using-upper-Standard-Function.html)
                * [Exercise: Using explode Standard Function](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Using-explode-Standard-Function.html)
                * [Exercise: Difference in Days Between Dates As Strings](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Difference-in-Days-Between-Dates-As-Strings.html)
                * [Exercise: Using UDFs](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Using-UDFs.html)
              1. [Basic Aggregation](02-spark-sql-basic-aggregation.html)
                * [Exercise: Finding maximum value (agg)](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Finding-maximum-value-agg.html)
                * [Exercise: Finding maximum values per group (groupBy)](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Finding-maximum-values-per-group-groupBy.html)
                * [Exercise: Collect values per group](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Collect-values-per-group.html)
                * [Exercise: Multiple Aggregations](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Multiple-Aggregations.html)
              1. [Joins](spark-sql-joins.html)
                * [Exercise: Finding Most Populated Cities Per Country](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Finding-Most-Populated-Cities-Per-Country.html)

## Day 3 — Advanced Spark SQL

1. Exercises
                * [Exercise: Flattening Array Columns (From Datasets of Arrays to Datasets of Array Elements)](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Flattening-Array-Columns-From-Datasets-of-Arrays-to-Datasets-of-Array-Elements.html)
                * [Exercise: Reverse-engineering Dataset.show Output](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Reverse-engineering-Dataset-show-Output.html)
                * [Exercise: Using pivot to generate a single-row matrix](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Using-pivot-to-generate-a-single-row-matrix.html)
                * [Exercise: Using pivot for Cost Average and Collecting Values](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Using-pivot-for-Cost-Average-and-Collecting-Values.html)
              1. [Windowed Aggregation](spark-sql-windowed-aggregation.html)
                * [Exercise: Finding 1st and 2nd Bestsellers Per Genre](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Finding-1st-and-2nd-Bestsellers-Per-Genre.html)
                * [Exercise: Calculating Gap Between Current And Highest Salaries Per Department](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Calculating-Gap-Between-Current-And-Highest-Salaries-Per-Department.html)
                * [Exercise: Calculating Running Total / Cumulative Sum](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Calculating-Running-Total-Cumulative-Sum.html)
                * [Exercise: Calculating Difference Between Consecutive Rows Per Window](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Calculating-Difference-Between-Consecutive-Rows-Per-Window.html)
              1. [Working with Missing Data](spark-sql-Working-with-Missing-Data.html)
                * [Exercise: Finding First Non-Null Value per Group](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Finding-First-Non-Null-Value-per-Group.html)

Prerequisities

Be prepared to get the most out of the workshop

Prerequisities / Experience

Hands-on programming experience using Scala
Experience developing Spark applications

Prerequisities / To Be Installed

Java Platform, Standard Edition (Java SE) 8
Apache Spark
IntelliJ IDEA Community Edition with Scala plugin
sbt
Apache Kafka
Linux or macOS preferred
- Consult Running Spark Applications on Windows when on MS Windows

In-Class Preparations

Make Instructor's Life Slightly Easier. Thanks!

Introduce Yourself

First name
What's your experience with Spark?
Any production experience with Spark?
What do you expect from the workshop?
Where do you want to be with Spark after 5 days?

Addendum

Put your name on paper in front of you
- Stick to your laptop if possible
- Use LARGE fonts
1-hour lunch break at 12:45pm

Apache Spark™Advanced for Developers Workshop 5 Days

@jaceklaskowski / StackOverflow / GitHub The "Internals" Books: Apache Spark / Spark SQL / Spark Structured Streaming