BasicExecutorFeatureStep¶

BasicExecutorFeatureStep is a KubernetesFeatureConfigStep.

Creating Instance¶

BasicExecutorFeatureStep takes the following to be created:

• KubernetesExecutorConf
• SecurityManager (Apache Spark)
• ResourceProfile (Apache Spark)

BasicExecutorFeatureStep is created when:

• KubernetesExecutorBuilder is requested for a pod spec for executors

Executor Resources¶

BasicExecutorFeatureStep creates an ExecutorResourcesOrDefaults when created (cf. ResourceProfile) with the following:

• ID and executorResources of the ResourceProfile
• spark.kubernetes.memoryOverheadFactor as the overheadFactor

The ExecutorResourcesOrDefaults is used for the following:

• executorMemoryString
• executorCoresRequest
• Configuring Pod (for executorMemoryQuantity, executorResourceQuantities and SPARK_EXECUTOR_CORES environment variable)

Executor Container Image Name¶

BasicExecutorFeatureStep asserts that spark.kubernetes.executor.container.image configuration property is defined or throws a SparkException:

Must specify the executor container image

Executor Pod Name Prefix¶

BasicExecutorFeatureStep requests the KubernetesExecutorConf for the resourceNamePrefix.

Configuring Pod¶

configurePod(
  pod: SparkPod): SparkPod

configurePod builds the name of the pod (with the executorPodNamePrefix and the executorId of the KubernetesExecutorConf):

[executorPodNamePrefix]-exec-[executorId]

configurePod configures a PodBuilder and a ContainerBuilder, and in the end, creates a SparkPod.

configurePod is part of the KubernetesFeatureConfigStep abstraction.

buildExecutorResourcesQuantities¶

buildExecutorResourcesQuantities(
  customResources: Set[ExecutorResourceRequest]): Map[String, Quantity]

For every ExecutorResourceRequest (in the given customResources), buildExecutorResourcesQuantities builds a Kubernetes resource name (based on the vendor and the resourceName of the ExecutorResourceRequest) and maps it to the amount (of the ExecutorResourceRequest).