Skip to content

ExecutorPodsPollingSnapshotSource

ExecutorPodsPollingSnapshotSource manages pollingFuture (for PollRunnables) to synchronize executor pods state every spark.kubernetes.executor.apiPollingInterval (by requesting the ExecutorPodsSnapshotsStore to replace executor pods snapshot).

Creating Instance

ExecutorPodsPollingSnapshotSource takes the following to be created:

ExecutorPodsPollingSnapshotSource is created when:

ExecutorPodsPollingSnapshotSource

spark.kubernetes.executor.apiPollingInterval

ExecutorPodsPollingSnapshotSource uses spark.kubernetes.executor.apiPollingInterval configuration property when started (to schedule a PollRunnable for regular executor pod state synchronization).

pollingFuture

pollingFuture: Future[_]

pollingFuture...FIXME

Starting

start(
  applicationId: String): Unit

ExecutorPodsPollingSnapshotSource and KubernetesClusterSchedulerBackend

start prints out the following DEBUG message to the logs (with the pollingInterval):

Starting to check for executor pod state every [pollingInterval] ms.

start throws an IllegalArgumentException when started twice (i.e. pollingFuture has already been initialized):

Cannot start polling more than once.

start is used when:

  • KubernetesClusterSchedulerBackend is requested to start

Logging

Enable ALL logging level for org.apache.spark.scheduler.cluster.k8s.ExecutorPodsPollingSnapshotSource logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.scheduler.cluster.k8s.ExecutorPodsPollingSnapshotSource=ALL

Refer to Logging.


Last update: 2021-01-25