Storage Volumes¶
Storage Volumes (Kubernetes) are configured using MountVolumesFeatureStep and KubernetesVolumeSpec.
Types of Volumes¶
Spark on Kubernetes supports the following types of volumes:
emptyDirhostPathnfspersistentVolumeClaim
Volume Configuration Prefixes¶
Spark on Kubernetes uses the prefixes for volume-related configuration properties for driver and executor pods:
spark.kubernetes.driver.volumes.spark.kubernetes.executor.volumes.
Claim Name Placeholders¶
MountVolumesFeatureStep defines OnDemand and SPARK_EXECUTOR_ID as placeholders for runtime-replaceable parts of the claim name of a KubernetesPVCVolumeConf.
These placeholders allow for templating claim names to include parts to be replaced at deployment.
Follow Demo: PersistentVolumeClaims to see the feature in action.
OnDemand¶
MountVolumesFeatureStep replaces all OnDemands with the following:
| Pod | Replacement |
|---|---|
| driver | [resourceNamePrefix]-exec-[executorId]-pvc-[volumeIndex] |
| executors | [resourceNamePrefix]-driver-pvc-[volumeIndex] |
SPARK_EXECUTOR_ID¶
MountVolumesFeatureStep replaces all SPARK_EXECUTOR_IDs with executor IDs.