Skip to content

PollRunnable

PollRunnable is a Java Runnable that ExecutorPodsPollingSnapshotSource schedules for full executor pod state snapshots from Kubernetes every spark.kubernetes.executor.apiPollingInterval in the Spark application.

PollRunnable is an internal class of ExecutorPodsPollingSnapshotSource with full access to its internals.

Creating Instance

PollRunnable takes the following to be created:

  • Application Id

PollRunnable is created when:

  • ExecutorPodsPollingSnapshotSource is requested to start

Starting Thread

run(): Unit

run prints out the following DEBUG message to the logs:

Resynchronizing full executor pod state from Kubernetes.

run requests the KubernetesClient for Spark executor pods with the following labels and values.

Label Name Value
spark-app-selector application Id
spark-role executor
spark-exec-inactive any value but true

In the end, run requests the ExecutorPodsSnapshotsStore to replace the snapshot.

Logging

PollRunnable uses org.apache.spark.scheduler.cluster.k8s.ExecutorPodsPollingSnapshotSource logger for logging.


Last update: 2021-01-23