Skip to content

Storage Volumes

Storage Volumes (Kubernetes) are configured using MountVolumesFeatureStep and KubernetesVolumeSpec.

Types of Volumes

Spark on Kubernetes supports the following types of volumes:

  • emptyDir
  • hostPath
  • nfs
  • persistentVolumeClaim

Volume Configuration Prefixes

Spark on Kubernetes uses the prefixes for volume-related configuration properties for driver and executor pods:

  • spark.kubernetes.driver.volumes.
  • spark.kubernetes.executor.volumes.

Claim Name Placeholders

MountVolumesFeatureStep defines OnDemand and SPARK_EXECUTOR_ID as placeholders for runtime-replaceable parts of the claim name of a KubernetesPVCVolumeConf.

These placeholders allow for templating claim names to include parts to be replaced at deployment.

Follow Demo: PersistentVolumeClaims to see the feature in action.

OnDemand

MountVolumesFeatureStep replaces all OnDemands with the following:

Pod Replacement
driver [resourceNamePrefix]-exec-[executorId]-pvc-[volumeIndex]
executors [resourceNamePrefix]-driver-pvc-[volumeIndex]

SPARK_EXECUTOR_ID

MountVolumesFeatureStep replaces all SPARK_EXECUTOR_IDs with executor IDs.


Last update: 2021-03-16