ExecutorPodsWatchSnapshotSource¶

ExecutorPodsWatchSnapshotSource is given KubernetesClient to a Kubernetes API server to watch for status updates of the executor pods of a given Spark application when started (that ExecutorPodsWatcher passes along to the ExecutorPodsSnapshotsStore).

Creating Instance¶

ExecutorPodsWatchSnapshotSource takes the following to be created:

• ExecutorPodsSnapshotsStore
• KubernetesClient

ExecutorPodsWatchSnapshotSource is created when:

• KubernetesClusterManager is requested for a SchedulerBackend

watchConnection¶

watchConnection: Closeable

ExecutorPodsWatchSnapshotSource defines watchConnection internal registry to be a "watch connection" to a Kubernetes API server to watch any status updates of the executor pods of a given Spark application (using ExecutorPodsWatcher).

ExecutorPodsWatchSnapshotSource uses watchConnection internal registry as an indication of whether it has been started already or not (and throws an IllegalArgumentException when it has).

ExecutorPodsWatchSnapshotSource requests the watchConnection to close and nulls it when requested to stop.

Starting¶

start(
  applicationId: String): Unit

start prints out the following DEBUG message to the logs:

Starting watch for pods with labels spark-app-selector=[applicationId], spark-role=executor.

start requests the KubernetesClient to watch pods with the following labels and values and pass pod updates to ExecutorPodsWatcher.

start is used when:

• KubernetesClusterSchedulerBackend is requested to start

Logging¶

Enable ALL logging level for org.apache.spark.scheduler.cluster.k8s.ExecutorPodsWatchSnapshotSource logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.scheduler.cluster.k8s.ExecutorPodsWatchSnapshotSource=ALL

Refer to Logging.