Big Data Solutions
using Apache Spark 2 Days

@jaceklaskowski / StackOverflow / GitHub / LinkedIn
The "Internals" Books: books.japila.pl

"The Internals Of" Online Books:

Goal

Developing hands-on experience in
Big Data solutions using
Apache Spark
(Spark SQL and Structured Streaming)
for Scala developers

## Agenda (1 of 2)

1. [Spark SQL](https://jaceklaskowski.github.io/spark-workshop/slides/spark-sql.html)
                  * [Exercise: Standalone Spark Application to Display Spark SQL
                  Version](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Standalone-Spark-Application-to-Display-Spark-SQL-Version.html)
                  * [Exercise: Using CSV Data
                  Source](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Using-CSV-Data-Source.html)
              1. [Columns and Dataset Operators](https://jaceklaskowski.github.io/spark-workshop/slides/spark-sql-columns-and-dataset-operators.html)
                  * [Exercise: Using Dataset.flatMap
                  Operator](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Using-Dataset-flatMap-Operator.html)
              1. [Standard and User-Defined Functions](https://jaceklaskowski.github.io/spark-workshop/slides/spark-sql-standard-functions-udfs.html)
                  * [Exercise: Difference in Days Between Dates As Strings](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Difference-in-Days-Between-Dates-As-Strings.html)
              1. [Basic Aggregation](https://jaceklaskowski.github.io/spark-workshop/slides/spark-sql-basic-aggregation.html)
                  * [Exercise: Finding maximum values per group
                  (groupBy)](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Finding-maximum-values-per-group-groupBy.html)
                  * [Exercise: Collect values per
                  group](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-sql-exercise-Collect-values-per-group.html)

## Agenda (2 of 2)

1. [Joins](https://jaceklaskowski.github.io/spark-workshop/slides/spark-sql-joins.html)
                  * [Exercise: Finding Most Populated Cities Per
                  Country](https://jaceklaskowski.github.io/spark-workshop/exercises/sql/Finding-Most-Populated-Cities-Per-Country.html)
              1. web UI
              1. [Spark Structured Streaming](spark-structured-streaming.html)
                  * [Exercise: Your First Standalone Structured Streaming Application](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-structured-streaming-exercise-Your-First-Standalone-Structured-Streaming-Application.html)
                  * [Exercise: Streaming CSV Datasets](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-structured-streaming-exercise-Streaming-CSV-Datasets.html)
                  * [Exercise: Stateless Streaming Aggregation](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-structured-streaming-exercise-Streaming-Aggregation-with-groupBy.html)
                  * [Exercise: Using foreach Operator (and ForeachWriter)](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-structured-streaming-exercise-Using-foreach-Operator-and-ForeachWriter.html)
              1. (?) Spark Structured Streaming / Kafka Data Source
                  * [Exercise: Using Kafka Data Source](https://jaceklaskowski.github.io/spark-workshop/exercises/spark-structured-streaming-exercise-Using-Kafka-Data-Source.html)

Prerequisities

Be prepared to get the most out of the workshop

Prerequisities / To Be Installed

Java Platform, Standard Edition (Java SE) 11
IntelliJ IDEA Community Edition with Scala plugin
sbt
git
Linux or macOS preferred
- Consult Running Spark Applications on Windows when on MS Windows

Prerequisities / To Be Downloaded

The latest version of Apache Spark

Big Data Solutionsusing Apache Spark 2 Days