Skip to content

ExecutorPodsWatchSnapshotSource

ExecutorPodsWatchSnapshotSource is given KubernetesClient to a Kubernetes API server to watch for status updates of the executor pods of a given Spark application when started (that ExecutorPodsWatcher passes along to the ExecutorPodsSnapshotsStore).

Creating Instance

ExecutorPodsWatchSnapshotSource takes the following to be created:

ExecutorPodsWatchSnapshotSource is created when:

watchConnection

watchConnection: Closeable

ExecutorPodsWatchSnapshotSource defines watchConnection internal registry to be a "watch connection" to a Kubernetes API server to watch any status updates of the executor pods of a given Spark application (using ExecutorPodsWatcher).

ExecutorPodsWatchSnapshotSource uses watchConnection internal registry as an indication of whether it has been started already or not (and throws an IllegalArgumentException when it has).

ExecutorPodsWatchSnapshotSource requests the watchConnection to close and nulls it when requested to stop.

Starting

start(
  applicationId: String): Unit

start prints out the following DEBUG message to the logs:

Starting watch for pods with labels spark-app-selector=[applicationId], spark-role=executor.

start requests the KubernetesClient to watch pods with the following labels and values and pass pod updates to ExecutorPodsWatcher.

Label Name Value
spark-app-selector the given applicationId
spark-role executor

start is used when:

  • KubernetesClusterSchedulerBackend is requested to start

Logging

Enable ALL logging level for org.apache.spark.scheduler.cluster.k8s.ExecutorPodsWatchSnapshotSource logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.scheduler.cluster.k8s.ExecutorPodsWatchSnapshotSource=ALL

Refer to Logging.


Last update: 2021-01-23