Spark and Cluster Managers
Apache Spark 2.4.1 / Spark Core
@jaceklaskowski / StackOverflow / GitHub
The "Internals" Books: Apache Spark / Spark SQL / Spark Structured Streaming
Agenda
Spark Application = The Driver + Executors
- A Spark application consists of the driver and executors (and your code, too!)
- Switch to The Internals of Apache Spark
Master URL
- Master URL points at the cluster manager to deploy your Spark application to.
- Switch to The Internals of Apache Spark
Deploy Mode
- Deploy mode specifies the location of where the driver executes when deployed to a deployment environment
- Switch to The Internals of Apache Spark
spark-submit Command-Line Tool
- spark-submit shell script allows you to manage your Spark applications.
- Switch to The Internals of Apache Spark
Spark Standalone
- Spark Standalone is Spark's own built-in clustered environment.
- Switch to The Internals of Apache Spark
ExternalClusterManager — Pluggable Cluster Managers
- ExternalClusterManager is a contract for pluggable cluster managers.
- Switch to The Internals of Apache Spark
Spark on YARN (1 of 2)
- Hadoop YARN is the Resource Negotiator from Apache Hadoop project.
- Switch to The Internals of Apache Spark
Spark on YARN (2 of 2)
From the official documentation of Apache Hadoop YARN
Spark on Mesos (1 of 2)
- Apache Mesos is a distributed systems kernel with API's for resource management and scheduling across entire datacenter and cloud environments.
- Switch to The Internals of Apache Spark
Spark on Mesos (2 of 2)
From the official documentation of Apache Mesos