Skip to content

ExecutorPodsSnapshot

ExecutorPodsSnapshot is an immutable view (snapshot) of the executor pods running in a Kubernetes cluster.

ExecutorPodsSnapshot tracks the state of executor pods and can be updated partially or fully.

Creating Instance

ExecutorPodsSnapshot takes the following to be created:

ExecutorPodsSnapshot is created (indirectly using apply utility) when:

Full Snapshot Timestamp

ExecutorPodsSnapshot keeps track of the time of the full snapshot of executor pods.

The time is given (via apply) when:

  • ExecutorPodsSnapshotsStoreImpl is requested to replace a snapshot (and basically do a full synchronization)

shouldCheckAllContainers Flag

shouldCheckAllContainers: Boolean

ExecutorPodsSnapshot uses shouldCheckAllContainers internal flag to control whether or not to include all containers in a running pod when requested for the ExecutorPodState.

shouldCheckAllContainers is controlled by spark.kubernetes.executor.checkAllContainers configuration property (when KubernetesClusterManager is requested for a SchedulerBackend).

apply Utility

apply(): ExecutorPodsSnapshot
apply(
  executorPods: Seq[Pod],
  fullSnapshotTs: Long): ExecutorPodsSnapshot

apply creates a ExecutorPodsSnapshot with the given arguments.

apply is used when:

Updating ExecutorPodsSnapshot

withUpdate(
  updatedPod: Pod): ExecutorPodsSnapshot

withUpdate creates a new ExecutorPodsSnapshot with the executorPods updated based on the given executor pod update (converted).

withUpdate is used when:

  • ExecutorPodsSnapshotsStoreImpl is requested to updatePod

toStatesByExecutorId

toStatesByExecutorId(
  executorPods: Seq[Pod]): Map[Long, ExecutorPodState]

toStatesByExecutorId...FIXME

toStatesByExecutorId is used when:

toState

toState(
  pod: Pod): ExecutorPodState

toState...FIXME

isDeleted

isDeleted(
  pod: Pod): Boolean

isDeleted...FIXME


Last update: 2021-01-22