Skip to content

BasicDriverFeatureStep

BasicDriverFeatureStep is a KubernetesFeatureConfigStep.

Creating Instance

BasicDriverFeatureStep takes the following to be created:

BasicDriverFeatureStep is created when:

Name of Driver Pod

BasicDriverFeatureStep uses the spark.kubernetes.driver.pod.name configuration property (if defined) or the resourceNamePrefix (of the given KubernetesDriverConf) as the name of the driver pod:

[resourceNamePrefix]-driver

Driver Memory

BasicDriverFeatureStep calculates the driver memory for the driver pod based on the Base Driver Memory with the extra Non-Heap Memory Overhead.

Driver memory is the quantity (in Mis) of the driver memory resource (the request and limit).

Demo: Running Spark Examples on minikube

Use kubectl get po [driverPod] to review the memory resource spec. Learn more in Demo: Running Spark Examples on minikube.

Base Driver Memory

BasicDriverFeatureStep uses spark.driver.memory configuration property (default: 1g) for the base driver memory.

Memory Overhead Factor

BasicDriverFeatureStep uses different memory overhead factors based on the MainAppResource (of the given KubernetesDriverConf):

The memory overhead factor is used for the following:

Non-Heap Memory Overhead

BasicDriverFeatureStep defines the memory overhead for a non-heap memory to be allocated per driver in cluster mode (in MiBs) based on the following:

Additional System Properties

getAdditionalPodSystemProperties(): Map[String, String]

getAdditionalPodSystemProperties is part of the KubernetesFeatureConfigStep abstraction.

getAdditionalPodSystemProperties sets the following additional properties:

Name Value
spark.kubernetes.submitInDriver true
spark.kubernetes.driver.pod.name driverPodName
spark.kubernetes.memoryOverheadFactor overheadFactor
spark.app.id appId (of the KubernetesDriverConf)

In the end, getAdditionalPodSystemProperties uploads local files (in spark.jars and spark.files configuration properties) to a Hadoop-compatible file system and adds their target Hadoop paths to the additional properties.

spark.kubernetes.file.upload.path Configuration Property

The Hadoop DFS path to upload local files to is defined using spark.kubernetes.file.upload.path configuration property.

Driver Container Image Name

BasicDriverFeatureStep uses spark.kubernetes.driver.container.image for the name of the container image for drivers.

The name must be defined or BasicDriverFeatureStep throws an SparkException:

Must specify the driver container image

driverContainerImage is used when requested for configurePod.

Configuring Driver Pod

configurePod(
  pod: SparkPod): SparkPod

configurePod is part of the KubernetesFeatureConfigStep abstraction.

configurePod...FIXME


Last update: 2021-01-16