ExecutorPodsSnapshot¶

ExecutorPodsSnapshot is an immutable view (snapshot) of the executor pods running in a Kubernetes cluster.

ExecutorPodsSnapshot tracks the state of executor pods and can be updated partially or fully.

Creating Instance¶

ExecutorPodsSnapshot takes the following to be created:

• ExecutorPodState by Executor ID (Map[Long, ExecutorPodState])
• Full Snapshot Timestamp

ExecutorPodsSnapshot is created (indirectly using apply utility) when:

• ExecutorPodsAllocator is created
• ExecutorPodsSnapshotsStoreImpl is created and requested to replace a snapshot
• ExecutorPodsSnapshot is requested to update

Full Snapshot Timestamp¶

ExecutorPodsSnapshot keeps track of the time of the full snapshot of executor pods.

The time is given (via apply) when:

• ExecutorPodsSnapshotsStoreImpl is requested to replace a snapshot (and basically do a full synchronization)

shouldCheckAllContainers Flag¶

shouldCheckAllContainers: Boolean

ExecutorPodsSnapshot uses shouldCheckAllContainers internal flag to control whether or not to include all containers in a running pod when requested for the ExecutorPodState.

shouldCheckAllContainers is controlled by spark.kubernetes.executor.checkAllContainers configuration property (when KubernetesClusterManager is requested for a SchedulerBackend).

apply Utility¶

apply(): ExecutorPodsSnapshot
apply(
  executorPods: Seq[Pod],
  fullSnapshotTs: Long): ExecutorPodsSnapshot

apply creates a ExecutorPodsSnapshot with the given arguments.

apply is used when:

• ExecutorPodsAllocator is created
• ExecutorPodsSnapshotsStoreImpl is created and requested to replace a snapshot

Updating ExecutorPodsSnapshot¶

withUpdate(
  updatedPod: Pod): ExecutorPodsSnapshot

withUpdate creates a new ExecutorPodsSnapshot with the executorPods updated based on the given executor pod update (converted).

withUpdate is used when:

• ExecutorPodsSnapshotsStoreImpl is requested to updatePod

toStatesByExecutorId¶

toStatesByExecutorId(
  executorPods: Seq[Pod]): Map[Long, ExecutorPodState]

toStatesByExecutorId...FIXME

toStatesByExecutorId is used when:

• ExecutorPodsSnapshot is requested to withUpdate and apply

toState¶

toState(
  pod: Pod): ExecutorPodState

toState...FIXME

isDeleted¶

isDeleted(
  pod: Pod): Boolean

isDeleted...FIXME