ExecutorPodsPollingSnapshotSource¶

ExecutorPodsPollingSnapshotSource manages pollingFuture (for PollRunnables) to synchronize executor pods state every spark.kubernetes.executor.apiPollingInterval (by requesting the ExecutorPodsSnapshotsStore to replace executor pods snapshot).

Creating Instance¶

ExecutorPodsPollingSnapshotSource takes the following to be created:

• SparkConf
• KubernetesClient
• ExecutorPodsSnapshotsStore
• Java ScheduledExecutorService

ExecutorPodsPollingSnapshotSource is created when:

• KubernetesClusterManager is requested for a SchedulerBackend (and creates a KubernetesClusterSchedulerBackend)

spark.kubernetes.executor.apiPollingInterval¶

ExecutorPodsPollingSnapshotSource uses spark.kubernetes.executor.apiPollingInterval configuration property when started (to schedule a PollRunnable for regular executor pod state synchronization).

pollingFuture¶

pollingFuture: Future[_]

pollingFuture...FIXME

Starting¶

start(
  applicationId: String): Unit

start prints out the following DEBUG message to the logs (with the pollingInterval):

Starting to check for executor pod state every [pollingInterval] ms.

start throws an IllegalArgumentException when started twice (i.e. pollingFuture has already been initialized):

Cannot start polling more than once.

start is used when:

• KubernetesClusterSchedulerBackend is requested to start

Logging¶

Enable ALL logging level for org.apache.spark.scheduler.cluster.k8s.ExecutorPodsPollingSnapshotSource logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.scheduler.cluster.k8s.ExecutorPodsPollingSnapshotSource=ALL

Refer to Logging.