Apache Spark™
Advanced for Developers Workshop 5 Days

@jaceklaskowski / StackOverflow / GitHub
The "Internals" Books: Apache Spark / Spark SQL / Spark Structured Streaming

Jacek is best known by the online "Internals" books:

Professional Objectives

Advance in solving analytical problems using Spark SQL
Explore the recent features of Apache Spark 2.4.1
Deep dive into the internals of Apache Spark and the modules (Spark SQL, Spark Structured Streaming and Spark MLlib)
Understand performance tuning of Apache Spark applications and the advanced features of Apache Spark

Training content

Anatomy of Spark Core Data Processing Platform
Foundations of Spark SQL
Internals of Structured Query Execution
Standard, User-Defined and User-Defined Aggregate Functions (Spark SQL)
Basic, Windowed and Multi-Dimensional Aggregations (Spark SQL)
Monitoring Spark Applications Using web UI and SparkListeners
Join Optimization with Bucketing (Spark SQL)
Stream Processing with Spark Structured Streaming
Machine Learning with Spark MLlib

## Day 3 — Advanced Spark SQL

1. [Basic Aggregation](spark-sql-basic-aggregation.html)
                * [Exercise: Finding maximum values per group (groupBy)](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Finding-maximum-values-per-group-groupBy.html)
                * [Exercise: Collect values per group](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Collect-values-per-group.html)
                * [Exercise: Multiple Aggregations](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Multiple-Aggregations.html)
                * [Exercise: Calculating aggregations](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Finding-maximum-value-agg.html)
              1. [Joins](spark-sql-joins.html)
                * [Exercise: Finding Most Populated Cities Per Country](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Finding-Most-Populated-Cities-Per-Country.html)
              1. [The Internals of Structured Query Execution](spark-sql-internals-of-structured-query-execution.html)
              1. Exercises
                * [Exercise: Flattening Array Columns (From Datasets of Arrays to Datasets of Array Elements)](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Flattening-Array-Columns-From-Datasets-of-Arrays-to-Datasets-of-Array-Elements.html)
                * [Exercise: Reverse-engineering Dataset.show Output](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Reverse-engineering-Dataset-show-Output.html)
                * [Exercise: Using pivot for Cost Average and Collecting Values](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Using-pivot-for-Cost-Average-and-Collecting-Values.html)

## Day 4 — Spark SQL / Spark Structured Streaming

1. [Windowed Aggregation](spark-sql-windowed-aggregation.html)
                * [Exercise: Finding 1st and 2nd Bestsellers Per Genre](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Finding-1st-and-2nd-Bestsellers-Per-Genre.html)
                * [Exercise: Calculating Gap Between Current And Highest Salaries Per Department](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Calculating-Gap-Between-Current-And-Highest-Salaries-Per-Department.html)
                * [Exercise: Calculating Running Total / Cumulative Sum](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Calculating-Running-Total-Cumulative-Sum.html)
                * [Exercise: Calculating Difference Between Consecutive Rows Per Window](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Calculating-Difference-Between-Consecutive-Rows-Per-Window.html)
              1. [Spark Structured Streaming](spark-structured-streaming.html)
                * [Exercise: Streaming CSV Datasets](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-structured-streaming-exercise-Streaming-CSV-Datasets.html)
                * [Exercise: Stateless Streaming Aggregation](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-structured-streaming-exercise-Streaming-Aggregation-with-groupBy.html)

## Day 5 — Spark (All Modules)

1. Spark Structured Streaming cntd.
                1. [Developing Custom Data Source](spark-sql-Developing-Custom-Data-Source.html)
                1. [Exercise: Developing Custom Data Source](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Developing-Custom-Data-Source.html)
                1. [Exercise: Using Kafka Data Source](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-structured-streaming-exercise-Using-Kafka-Data-Source.html)
              1. Spark SQL / Exercises
                1. [Exercise: Generating Exam Assessment Report](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Generating-Exam-Assessment-Report.html)
                1. [Exercise: Finding Longest Sequence](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Finding-Longest-Sequence.html)
                1. [Exercise: Calculating percent rank](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Calculating-percent-rank.html)
              1. Spark MLlib
                1. [Machine Learning with Spark MLlib](spark-mllib.html)
                1. [ML Pipelines](spark-mllib-ml-pipelines.html)
                1. [Exercise: Email Classification](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-mllib-exercise-Email-Classification.html)
              1. [web UI](spark-core-webui.html)

Prerequisities

Be prepared to get the most out of the workshop

Prerequisities / Experience

Hands-on programming experience using Scala
Experience developing Spark applications

Prerequisities / To Be Installed

Java Platform, Standard Edition (Java SE) 8
Apache Spark
IntelliJ IDEA Community Edition with Scala plugin
sbt
Apache Kafka
Linux or macOS preferred
- Consult Running Spark Applications on Windows when on MS Windows

In-Class Preparations

Make Instructor's Life Slightly Easier. Thanks!

Introduce Yourself

First name
What's your experience with Spark?
Any production experience with Spark?
What do you expect from the workshop?
Where do you want to be with Spark after 5 days?

Addendum

Put your name on paper in front of you
- Stick to your laptop if possible
- Use LARGE fonts
1-hour lunch break at 12:45pm

Apache Spark™Advanced for Developers Workshop 5 Days

@jaceklaskowski / StackOverflow / GitHub The "Internals" Books: Apache Spark / Spark SQL / Spark Structured Streaming