Configuration Properties¶

spark.kubernetes.allocation.batch.delay¶

Time (in millis) to wait between each round of executor allocation

Default: 1s

Used when:

• ExecutorPodsAllocator is created

spark.kubernetes.appKillPodDeletionGracePeriod¶

Grace Period that is the time (in seconds) to wait for a graceful deletion of Spark pods when spark-submit --kill

Default: (undefined)

Used when:

• K8SSparkSubmitOperation is requested to kill

spark.kubernetes.allocation.batch.size¶

Maximum number of executor pods to allocate at once in each round of executor allocation

Default: 5

Used when:

• ExecutorPodsAllocator is requested to allocate executor pods

spark.kubernetes.allocation.executor.timeout¶

Time (in millis) to wait before a pending executor is considered timed out

Default: 600s

Used when:

• ExecutorPodsAllocator is requested to handle executor pods snapshots

spark.kubernetes.authenticate¶

FIXME

spark.kubernetes.authenticate.driver.mounted¶

FIXME

spark.kubernetes.authenticate.driver.serviceAccountName¶

Service account for a driver pod (for requesting executor pods from the API server)

Default: (undefined)

Used when:

• DriverKubernetesCredentialsFeatureStep is requested to configure a pod
• ExecutorKubernetesCredentialsFeatureStep is requested to configure a pod

spark.kubernetes.authenticate.executor.serviceAccountName¶

Service account for executor pods

Default: (undefined)

Used when:

• ExecutorKubernetesCredentialsFeatureStep is requested to configure a pod

spark.kubernetes.configMap.maxSize¶

Max size limit (long) for a config map. Configurable as per https://etcd.io/docs/v3.4.0/dev-guide/limit/ on k8s server end.

Default: 1572864 (1.5 MiB)

Used when:

• KubernetesClientUtils utility is used to loadSparkConfDirFiles

spark.kubernetes.container.image¶

Container image to use for Spark containers (unless spark.kubernetes.driver.container.image or spark.kubernetes.executor.container.image are defined)

Default: (undefined)

spark.kubernetes.container.image.pullPolicy¶

Kubernetes image pull policy:

• Always
• Never
• IfNotPresent

Default: IfNotPresent

Used when:

• KubernetesConf is requested for imagePullPolicy

spark.kubernetes.context¶

The desired context from your K8S config file used to configure the K8S client for interacting with the cluster. Useful if your config file has multiple clusters or user identities defined. The client library used locates the config file via the KUBECONFIG environment variable or by defaulting to .kube/config under your home directory. If not specified then your current context is used. You can always override specific aspects of the config file provided configuration using other Spark on K8S configuration options.

Default: (undefined)

Used when:

• SparkKubernetesClientFactory is requested to create a KubernetesClient

spark.kubernetes.driver.container.image¶

Container image for drivers

Default: spark.kubernetes.container.image

Used when:

• BasicDriverFeatureStep is requested for a driverContainerImage

spark.kubernetes.driver.master¶

The internal Kubernetes master (API server) address to be used for driver to request executors.

Default: https://kubernetes.default.svc

spark.kubernetes.driver.pod.name¶

Name of the driver pod

Default: (undefined)

Must be provided if a Spark application is deployed in cluster deploy mode

Used when:

• BasicDriverFeatureStep is requested for the driverPodName (and additional system properties of a driver pod)
• ExecutorPodsAllocator is requested for the kubernetesDriverPodName

spark.kubernetes.driver.podTemplateContainerName¶

Name of the driver container in a pod template

Default: (undefined)

Used when:

• KubernetesDriverBuilder is requested for a driver pod specification

spark.kubernetes.driver.podTemplateFile¶

Pod template file for drivers (in cluster deploy mode)

Default: (undefined)

Used when:

• KubernetesDriverBuilder is requested for a driver pod specification

spark.kubernetes.driver.request.cores¶

Specify the cpu request for the driver pod

Default: (undefined)

Used when:

• BasicDriverFeatureStep is requested to configure a pod

spark.kubernetes.executor.apiPollingInterval¶

Interval (in millis) between polls against the Kubernetes API server to inspect the state of executors.

Default: 30s

Used when:

• ExecutorPodsPollingSnapshotSource is requested to start

spark.kubernetes.executor.checkAllContainers¶

Controls whether or not to check the status of all containers in a running executor pod when reporting executor status

Default: false

Used when:

• KubernetesClusterManager is requested for a SchedulerBackend

spark.kubernetes.executor.container.image¶

Container image for executors

Default: spark.kubernetes.container.image

Used when:

• BasicExecutorFeatureStep is requested for a driverContainerImage

spark.kubernetes.executor.deleteOnTermination¶

Controls whether or not to delete executor pods after they have finished (successfully or not)

Default: true

Used when:

• ExecutorPodsAllocator is requested to handle executor pods snapshots
• ExecutorPodsLifecycleManager is requested to handle executor pods snapshots
• KubernetesClusterSchedulerBackend is requested to stop

spark.kubernetes.executor.eventProcessingInterval¶

Interval (in millis) between successive inspection of executor events sent from the Kubernetes API

Default: 1s

Used when:

• ExecutorPodsLifecycleManager is requested to start and register a new subscriber

spark.kubernetes.executor.missingPodDetectDelta¶

Time (in millis) to wait before an executor is removed due to the executor's pod being missed in the Kubernetes API server's polled list of pods

Default: 30s

Used when:

• ExecutorPodsLifecycleManager is requested to handle executor pods snapshots

spark.kubernetes.executor.podNamePrefix¶

Prefix of the name of executor pods

Default: (undefined)

Unless defined, it is set explicitly when KubernetesClusterManager is requested to create a SchedulerBackend

Used when:

• KubernetesExecutorConf is requested for the resourceNamePrefix

spark.kubernetes.executor.podTemplateContainerName¶

Name of the container for executors in a pod template

Default: (undefined)

Used when:

• KubernetesClusterManager is requested for a SchedulerBackend
• KubernetesExecutorBuilder is requested for a pod spec for executors

spark.kubernetes.executor.podTemplateFile¶

Pod template file for executors

Default: (undefined)

Used when:

• KubernetesClusterManager is requested to create a SchedulerBackend
• KubernetesExecutorBuilder is requested for a pod spec for executors
• PodTemplateConfigMapStep is created and requested to configurePod, getAdditionalPodSystemProperties, getAdditionalKubernetesResources

spark.kubernetes.executor.request.cores¶

Specifies the cpu quantity request for executor pods (to be more Kubernetes-oriented when requesting resources for executor pods than Spark scheduler's approach based on spark.executor.cores).

Default: (undefined)

Used when:

• BasicExecutorFeatureStep is requested to configure an executor pod

spark.kubernetes.executor.scheduler.name¶

Name of the scheduler for executor pods (a pod's spec.schedulerName)

Default: (undefined)

Used when:

• BasicExecutorFeatureStep is requested to configure an executor pod

spark.kubernetes.file.upload.path¶

Hadoop DFS-compatible file system path where files from the local file system will be uploded to in cluster deploy mode. The subdirectories (one per Spark application) with the local files are of the format spark-upload-[uuid].

Default: (undefined)

Used when:

• KubernetesUtils is requested to uploadFileUri

spark.kubernetes.local.dirs.tmpfs¶

If true, emptyDir volumes created to back SPARK_LOCAL_DIRS will have their medium set to Memory so that they will be created as tmpfs (i.e. RAM) backed volumes. This may improve performance but scratch space usage will count towards your pods memory limit so you may wish to request more memory.

Default: false

Used when:

• LocalDirsFeatureStep is requested to configure a pod

spark.kubernetes.memoryOverheadFactor¶

Memory Overhead Factor that will allocate memory to non-JVM jobs which in the case of JVM tasks will default to 0.10 and 0.40 for non-JVM jobs

Must be a double between (0, 1.0)

Default: 0.1

Used when:

• BasicDriverFeatureStep is requested to configure a pod
• BasicExecutorFeatureStep is requested to configure a pod

spark.kubernetes.namespace¶

Namespace for running the driver and executor pods

Default: default

Used when:

• KubernetesConf is requested for namespace
• KubernetesClusterManager is requested for a SchedulerBackend
• ExecutorPodsAllocator is created (and initializes namespace)

spark.kubernetes.report.interval¶

Interval between reports of the current app status in cluster mode

Default: 1s

Used when:

• LoggingPodStatusWatcherImpl is requested to watchOrStop

spark.kubernetes.submission.waitAppCompletion¶

In cluster deploy mode, whether to wait for the application to finish before exiting the launcher process.

Default: true

Used when:

• LoggingPodStatusWatcherImpl is requested to watchOrStop

spark.kubernetes.submitInDriver¶

(internal) Whether executing in cluster deploy mode

Default: false

spark.kubernetes.submitInDriver is true in BasicDriverFeatureStep.

Used when:

• BasicDriverFeatureStep is requested to getAdditionalPodSystemProperties
• KubernetesClusterManager is requested for a SchedulerBackend