ExecutorPodsWatchSnapshotSource¶
Creating Instance¶
ExecutorPodsWatchSnapshotSource
takes the following to be created:
- ExecutorPodsSnapshotsStore
-
KubernetesClient
ExecutorPodsWatchSnapshotSource
is created when:
KubernetesClusterManager
is requested for a SchedulerBackend
Starting¶
start(
applicationId: String): Unit
start
prints out the following DEBUG message to the logs:
Starting watch for pods with labels spark-app-selector=[applicationId], spark-role=executor.
start
requests the KubernetesClient to watch pods with the following labels using ExecutorPodsWatcher:
spark-app-selector
with the givenapplicationId
spark-role
asexecutor
start
is used when:
KubernetesClusterSchedulerBackend
is requested to start
Logging¶
Enable ALL
logging level for org.apache.spark.scheduler.cluster.k8s.ExecutorPodsWatchSnapshotSource
logger to see what happens inside.
Add the following line to conf/log4j.properties
:
log4j.logger.org.apache.spark.scheduler.cluster.k8s.ExecutorPodsWatchSnapshotSource=ALL
Refer to Logging.
Last update: 2021-01-10