ExecutorPodsWatchSnapshotSource¶

Creating Instance¶

ExecutorPodsWatchSnapshotSource takes the following to be created:

• ExecutorPodsSnapshotsStore
• KubernetesClient

ExecutorPodsWatchSnapshotSource is created when:

• KubernetesClusterManager is requested for a SchedulerBackend

Starting¶

start(
  applicationId: String): Unit

start prints out the following DEBUG message to the logs:

Starting watch for pods with labels spark-app-selector=[applicationId], spark-role=executor.

start requests the KubernetesClient to watch pods with the following labels using ExecutorPodsWatcher:

• spark-app-selector with the given applicationId
• spark-role as executor

start is used when:

• KubernetesClusterSchedulerBackend is requested to start

Logging¶

Enable ALL logging level for org.apache.spark.scheduler.cluster.k8s.ExecutorPodsWatchSnapshotSource logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.scheduler.cluster.k8s.ExecutorPodsWatchSnapshotSource=ALL

Refer to Logging.