Apache Spark™
Advanced for Developers Workshop 5 Days

@jaceklaskowski / StackOverflow / GitHub / LinkedIn
The "Internals" Books: Apache Spark / Spark SQL / Spark Structured Streaming / Delta Lake

"The Internals Of" Online Books:

Professional Objectives

Advance in solving analytical problems using Spark SQL
Explore the recent features of Apache Spark 2.4
Deep dive into the internals of Apache Spark and the modules (Spark SQL, Spark Structured Streaming and Spark MLlib)
Understand performance tuning of Apache Spark applications and the advanced features of Apache Spark

Training content

Anatomy of Spark Core Data Processing Platform
Foundations of Spark SQL
Internals of Structured Query Execution
Standard, User-Defined and User-Defined Aggregate Functions (Spark SQL)
Basic, Windowed and Multi-Dimensional Aggregations (Spark SQL)
Monitoring Spark Applications Using web UI and SparkListeners
Join Optimization with Bucketing (Spark SQL)
Stream Processing with Spark Structured Streaming
Machine Learning with Spark MLlib

## Day 1 — The Essentials of Spark SQL

1. (Warm-up) [Exercise: Finding Ids of Rows with Word in Array Column](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Finding-Ids-of-Rows-with-Word-in-Array-Column.html)
              1. [Spark SQL](spark-sql.html)
                * [Exercise: Standalone Spark Application to Display Spark SQL Version](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Standalone-Spark-Application-to-Display-Spark-SQL-Version.html)
                * [Exercise: Using CSV Data Source](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Using-CSV-Data-Source.html)
              1. [Columns and Dataset Operators](spark-sql-columns-and-dataset-operators.html)
                * [Exercise: Using Dataset.flatMap Operator](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Using-Dataset-flatMap-Operator.html)
                * [Exercise: Difference in Days Between Dates As Strings](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Difference-in-Days-Between-Dates-As-Strings.html)
                * [Exercise: Why are all fields null when querying with schema?](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Why-are-all-fields-null-when-querying-with-schema.html)

## Day 2 — Spark SQL

1. [Standard and User-Defined Functions](spark-sql-standard-functions-udfs.html)
                * [Exercise: How to add days (as values of a column) to date?](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/How-to-add-days-as-values-of-a-column-to-date.html)
              1. [Basic Aggregation](spark-sql-basic-aggregation.html)
                * [Exercise: Finding maximum values per group (groupBy)](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Finding-maximum-values-per-group-groupBy.html)
                * [Exercise: Collect values per group](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Collect-values-per-group.html)
                * [Exercise: Multiple Aggregations](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Multiple-Aggregations.html)
                * [Exercise: Calculating aggregations](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Finding-maximum-value-agg.html)
              1. [Joins](spark-sql-joins.html)
                * [Exercise: Finding Most Populated Cities Per Country](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Finding-Most-Populated-Cities-Per-Country.html)
              1. [The Internals of Structured Query Execution](spark-sql-internals-of-structured-query-execution.html)
              1. Exercises
                * [Exercise: Flattening Array Columns (From Datasets of Arrays to Datasets of Array Elements)](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Flattening-Array-Columns-From-Datasets-of-Arrays-to-Datasets-of-Array-Elements.html)

## Day 3 — Advanced Spark SQL

1. Exercises
                * [Exercise: Reverse-engineering Dataset.show Output](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Reverse-engineering-Dataset-show-Output.html)
                * [Exercise: Using pivot for Cost Average and Collecting Values](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Using-pivot-for-Cost-Average-and-Collecting-Values.html)
              1. [Windowed Aggregation](spark-sql-windowed-aggregation.html)
                * [Exercise: Finding 1st and 2nd Bestsellers Per Genre](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Finding-1st-and-2nd-Bestsellers-Per-Genre.html)
                * [Exercise: Calculating Gap Between Current And Highest Salaries Per Department](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Calculating-Gap-Between-Current-And-Highest-Salaries-Per-Department.html)
                * [Exercise: Calculating Running Total / Cumulative Sum](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Calculating-Running-Total-Cumulative-Sum.html)
                * [Exercise: Calculating Difference Between Consecutive Rows Per Window](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Calculating-Difference-Between-Consecutive-Rows-Per-Window.html)
              1. [Multi-Dimensional Aggregation](spark-sql-multi-dimensional-aggregation.html)
                * [Exercise: Using rollup Operator for Total and Average Salaries by Department and Company-Wide](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Using-rollup-Operator-for-Total-and-Average-Salaries-by-Department-and-Company-Wide.html)
              1. [Developing Custom Data Source](spark-sql-Developing-Custom-Data-Source.html)

## Day 4 — Spark Structured Streaming

1. [Exercise: Developing Custom Data Source](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Developing-Custom-Data-Source.html)
              1. [Spark Structured Streaming](spark-structured-streaming.html)
                * [Exercise: Your First Standalone Structured Streaming Application](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-structured-streaming-exercise-Your-First-Standalone-Structured-Streaming-Application.html)
                * [Exercise: Streaming CSV Datasets](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-structured-streaming-exercise-Streaming-CSV-Datasets.html)
                * [Exercise: Stateless Streaming Aggregation](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-structured-streaming-exercise-Streaming-Aggregation-with-groupBy.html)
              1. Spark Structured Streaming / Kafka Data Source
                * [The Essentials of Apache Kafka™](https://jaceklaskowski.github.io/kafka-workshop/slides/apache-kafka-essentials.html)
                * [Exercise: Using Kafka Data Source](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-structured-streaming-exercise-Using-Kafka-Data-Source.html)

## Day 5 — Spark (All Modules)

1. Spark Structured Streaming / Exercises
                * [Exercise: Using foreach Operator (and ForeachWriter)](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-structured-streaming-exercise-Using-foreach-Operator-and-ForeachWriter.html)
                * [Exercise: Executing SQL Statements from Kafka](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-structured-streaming-exercise-Executing-SQL-Statements-from-Kafka.html)
                * [Exercise: Writing Selected Columns to Kafka](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-structured-streaming-exercise-Writing-Selected-Columns-to-Kafka.html)
              1. Delta Lake
              1. Spark MLlib
                1. [Machine Learning with Spark MLlib](spark-mllib.html)
                1. [ML Pipelines](spark-mllib-ml-pipelines.html)
                1. [Exercise: Email Classification](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-mllib-exercise-Email-Classification.html)
              1. [web UI](spark-core-webui.html)
              1. Spark SQL / Exercises
                1. [Exercise: Generating Exam Assessment Report](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Generating-Exam-Assessment-Report.html)
                1. [Exercise: Finding Longest Sequence](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Finding-Longest-Sequence.html)
                1. [Exercise: Calculating percent rank](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Calculating-percent-rank.html)

Prerequisities

Be prepared to get the most out of the workshop

Prerequisities / Experience

Hands-on programming experience using Scala
Experience developing Spark applications

Prerequisities / To Be Installed

Java Platform, Standard Edition (Java SE) 8
Apache Spark
IntelliJ IDEA Community Edition with Scala plugin
sbt
git
Apache Kafka
Linux or macOS preferred
- Consult Running Spark Applications on Windows when on MS Windows

In-Class Preparations

Make Instructor's Life Slightly Easier. Thanks!

Introduce Yourself

First name
What's your experience with Spark?
Any production experience with Spark?
What do you expect from the workshop?

Addendum

Put your name on paper in front of you
- Stick to your laptop if possible
- Use LARGE fonts
1-hour lunch break at 12:45pm

Apache Spark™Advanced for Developers Workshop 5 Days

@jaceklaskowski / StackOverflow / GitHub / LinkedIn The "Internals" Books: Apache Spark / Spark SQL / Spark Structured Streaming / Delta Lake