BasicDriverFeatureStep¶

BasicDriverFeatureStep is a KubernetesFeatureConfigStep.

Creating Instance¶

BasicDriverFeatureStep takes the following to be created:

KubernetesDriverConf

BasicDriverFeatureStep is created when:

KubernetesDriverBuilder is requested for the driver pod spec

Name of Driver Pod¶

BasicDriverFeatureStep uses the spark.kubernetes.driver.pod.name configuration property (if defined) or the resourceNamePrefix (of the given KubernetesDriverConf) as the name of the driver pod:

[resourceNamePrefix]-driver

Driver Memory¶

BasicDriverFeatureStep calculates the driver memory for the driver pod based on the Base Driver Memory with the extra Non-Heap Memory Overhead.

Driver memory is the quantity (in Mis) of the driver memory resource (the request and limit).

Demo: Running Spark Examples on minikube

Use kubectl get po [driverPod] to review the memory resource spec. Learn more in Demo: Running Spark Examples on minikube.

Base Driver Memory¶

BasicDriverFeatureStep uses spark.driver.memory configuration property (default: 1g) for the base driver memory.

Memory Overhead Factor¶

BasicDriverFeatureStep uses different memory overhead factors based on the MainAppResource (of the given KubernetesDriverConf):

For JVM applications (Java, Scala, etc.), it is spark.kubernetes.memoryOverheadFactor configuration property
For non-JVM applications (Python and R applications), it is spark.kubernetes.memoryOverheadFactor configuration property (if defined) or 0.4

The memory overhead factor is used for the following:

Additional System Properties
Memory Overhead

Non-Heap Memory Overhead¶

BasicDriverFeatureStep defines the memory overhead for a non-heap memory to be allocated per driver in cluster mode (in MiBs) based on the following:

spark.driver.memoryOverhead configuration property (if defined)
Maximum of the Memory Overhead Factor multiplied by the Base Driver Memory and 384

Additional System Properties¶

getAdditionalPodSystemProperties(): Map[String, String]

getAdditionalPodSystemProperties is part of the KubernetesFeatureConfigStep abstraction.

getAdditionalPodSystemProperties sets the following additional properties:

Name	Value
spark.kubernetes.submitInDriver	`true`
spark.kubernetes.driver.pod.name	driverPodName
spark.kubernetes.memoryOverheadFactor	overheadFactor
`spark.app.id`	appId (of the KubernetesDriverConf)

In the end, getAdditionalPodSystemProperties uploads local files (in spark.jars and spark.files configuration properties) to a Hadoop-compatible file system and adds their target Hadoop paths to the additional properties.

spark.kubernetes.file.upload.path Configuration Property

The Hadoop DFS path to upload local files to is defined using spark.kubernetes.file.upload.path configuration property.

Driver Container Image Name¶

BasicDriverFeatureStep uses spark.kubernetes.driver.container.image for the name of the container image for drivers.

The name must be defined or BasicDriverFeatureStep throws an SparkException:

Must specify the driver container image

driverContainerImage is used when requested for configurePod.

Configuring Driver Pod¶

configurePod(
  pod: SparkPod): SparkPod

configurePod is part of the KubernetesFeatureConfigStep abstraction.

configurePod...FIXME