Skip to content

ExecutorPodsPollingSnapshotSource

ExecutorPodsPollingSnapshotSource manages pollingFuture (with a PollRunnable) to synchronize executor pods state every pollingInterval (and requests the ExecutorPodsSnapshotsStore to replaceSnapshot).

Creating Instance

ExecutorPodsPollingSnapshotSource takes the following to be created:

ExecutorPodsPollingSnapshotSource is created when:

spark.kubernetes.executor.apiPollingInterval

ExecutorPodsPollingSnapshotSource uses spark.kubernetes.executor.apiPollingInterval configuration property when started (to schedule a PollRunnable for regular executor pod state synchronization).

pollingFuture

pollingFuture: Future[_]

pollingFuture...FIXME

Starting

start(
  applicationId: String): Unit

start prints out the following DEBUG message to the logs:

Starting to check for executor pod state every [pollingInterval] ms.

start throws an IllegalArgumentException when started twice (i.e. pollingFuture has already been initialized):

Cannot start polling more than once.

start is used when:

  • KubernetesClusterSchedulerBackend is requested to start

Logging

Enable ALL logging level for org.apache.spark.scheduler.cluster.k8s.ExecutorPodsPollingSnapshotSource logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.scheduler.cluster.k8s.ExecutorPodsPollingSnapshotSource=ALL

Refer to Logging.


Last update: 2021-01-06