Agenda

1. Spark Architecture
2. Dynamic Allocation of Executors
3. Dynamic Allocation Demo

Spark Architecture

1. Driver is a JVM process that hosts SparkContext for a Spark application
2. Executor is a distributed agent that executes Spark tasks
3. Master and Slaves / Workers
4. Switch to Mastering Apache Spark 2
   • Spark Architecture

Dynamic Allocation of Executors

1. Dynamic Allocation is a Spark feature that allows for adding or removing executors dynamically based on the workload
2. Switch to Mastering Apache Spark 2
   • Dynamic Allocation (of Executors)

Recap

1. Spark Architecture
2. Dynamic Allocation of Executors
3. Dynamic Allocation Demo

Questions?

• Read Mastering Apache Spark 2
  • https://bit.ly/mastering-apache-spark
• Follow @jaceklaskowski on twitter
• Upvote my questions and answers on StackOverflow
• Use Jacek's code at GitHub
• Read Jacek Laskowski @ Medium
• Visit Jacek Laskowski's blog