Skip to content

LocalDirsFeatureStep

LocalDirsFeatureStep is a KubernetesFeatureConfigStep.

Creating Instance

LocalDirsFeatureStep takes the following to be created:

  • KubernetesConf
  • Default Local Directory (default: /var/data/spark-[randomUUID])

LocalDirsFeatureStep is created when:

spark.kubernetes.local.dirs.tmpfs

LocalDirsFeatureStep uses spark.kubernetes.local.dirs.tmpfs configuration property when configuring a pod.

Configuring Pod

configurePod(
  pod: SparkPod): SparkPod

configurePod is part of the KubernetesFeatureConfigStep abstraction.

configurePod finds mount paths of the volume mounts with spark-local-dir- prefix name of the input SparkPod (localDirs).

If there are no local directory mount paths, configurePod...FIXME

configurePod adds the local directory volumes to a new pod specification (there could be none).

configurePod defines SPARK_LOCAL_DIRS environment variable as a comma-separated local directories and adds the local directory volume mounts to a new container specification (there could be none).

In the end, configurePod creates a new SparkPod with the new pod and container.


Last update: 2021-02-01