LocalDirsFeatureStep¶

LocalDirsFeatureStep is a KubernetesFeatureConfigStep.

Creating Instance¶

LocalDirsFeatureStep takes the following to be created:

• KubernetesConf
• Default Local Directory (default: /var/data/spark-[randomUUID])

LocalDirsFeatureStep is created when:

• KubernetesDriverBuilder is requested to build a driver pod
• KubernetesExecutorBuilder is requested for a pod spec for executors

spark.kubernetes.local.dirs.tmpfs¶

LocalDirsFeatureStep uses spark.kubernetes.local.dirs.tmpfs configuration property when configuring a pod.

Configuring Pod¶

configurePod(
  pod: SparkPod): SparkPod

configurePod is part of the KubernetesFeatureConfigStep abstraction.

configurePod finds mount paths of the volume mounts with spark-local-dir- prefix name of the input SparkPod (localDirs).

If there are no local directory mount paths, configurePod...FIXME

configurePod adds the local directory volumes to a new pod specification (there could be none).

configurePod defines SPARK_LOCAL_DIRS environment variable as a comma-separated local directories and adds the local directory volume mounts to a new container specification (there could be none).

In the end, configurePod creates a new SparkPod with the new pod and container.