Skip to content

Configuration Properties

spark.kubernetes.allocation.batch.delay

Time (in millis) to wait between each round of executor allocation

Default: 1s

Used when:

spark.kubernetes.allocation.batch.size

Minimum number of executor pods to allocate at once in each round of executor allocation

Default: 5

Used when:

spark.kubernetes.authenticate

FIXME

spark.kubernetes.authenticate.driver.mounted

FIXME

spark.kubernetes.container.image

Container image to use for Spark containers (unless spark.kubernetes.driver.container.image or spark.kubernetes.executor.container.image are defined)

Default: (undefined)

spark.kubernetes.container.image.pullPolicy

Kubernetes image pull policy:

  • Always
  • Never
  • IfNotPresent

Default: IfNotPresent

Used when:

spark.kubernetes.context

The desired context from your K8S config file used to configure the K8S client for interacting with the cluster. Useful if your config file has multiple clusters or user identities defined. The client library used locates the config file via the KUBECONFIG environment variable or by defaulting to .kube/config under your home directory. If not specified then your current context is used. You can always override specific aspects of the config file provided configuration using other Spark on K8S configuration options.

Default: (undefined)

Used when:

spark.kubernetes.driver.container.image

Container image for drivers

Default: spark.kubernetes.container.image

Used when:

spark.kubernetes.driver.master

The internal Kubernetes master (API server) address to be used for driver to request executors.

Default: https://kubernetes.default.svc

spark.kubernetes.driver.pod.name

Name of the driver pod

Default: (undefined)

Must be provided if a Spark application is deployed in cluster deploy mode

Used when:

spark.kubernetes.driver.podTemplateContainerName

Name of the driver container in a pod template

Default: (undefined)

Used when:

spark.kubernetes.driver.podTemplateFile

Pod template file for drivers (in cluster deploy mode)

Default: (undefined)

Used when:

spark.kubernetes.driver.request.cores

Specify the cpu request for the driver pod

Default: (undefined)

Used when:

spark.kubernetes.executor.apiPollingInterval

Interval (in millis) between polls against the Kubernetes API server to inspect the state of executors.

Default: 30s

Used when:

  • ExecutorPodsPollingSnapshotSource is requested to start

spark.kubernetes.executor.container.image

Container image for executors

Default: spark.kubernetes.container.image

Used when:

spark.kubernetes.executor.deleteOnTermination

When disabled (false), executor pods will not be deleted in case of failure or normal termination

Default: true

Used when:

spark.kubernetes.executor.eventProcessingInterval

Interval (in millis) between successive inspection of executor events sent from the Kubernetes API

Default: 1s

Used when:

spark.kubernetes.executor.podNamePrefix

(internal) Prefix of the executor pod names

Default: (undefined)

Unless defined, it is set explicitly when KubernetesClusterManager is requested to create a SchedulerBackend

Used when:

spark.kubernetes.executor.podTemplateContainerName

Container name to be used as a basis for executors in the given pod template

Default: (undefined)

Used when:

spark.kubernetes.executor.podTemplateFile

Pod template file for executors

Default: (undefined)

Used when:

spark.kubernetes.executor.request.cores

Specifies the cpu quantity request for executor pods (to be more Kubernetes-oriented when requesting resources for executor pods than Spark scheduler's approach based on spark.executor.cores).

Default: (undefined)

Used when:

spark.kubernetes.executor.scheduler.name

Name of the scheduler for executor pods (a pod's spec.schedulerName)

Default: (undefined)

Used when:

spark.kubernetes.file.upload.path

Hadoop DFS-compatible file system path where files from the local file system will be uploded to in cluster deploy mode. The subdirectories (one per Spark application) with the local files are of the format spark-upload-[uuid].

Default: (undefined)

Used when:

spark.kubernetes.local.dirs.tmpfs

If true, emptyDir volumes created to back SPARK_LOCAL_DIRS will have their medium set to Memory so that they will be created as tmpfs (i.e. RAM) backed volumes. This may improve performance but scratch space usage will count towards your pods memory limit so you may wish to request more memory.

Default: false

Used when:

spark.kubernetes.memoryOverheadFactor

Memory Overhead Factor that will allocate memory to non-JVM jobs which in the case of JVM tasks will default to 0.10 and 0.40 for non-JVM jobs

Must be a double between (0, 1.0)

Default: 0.1

Used when:

spark.kubernetes.namespace

Namespace for running the driver and executor pods

Default: default

Used when:

spark.kubernetes.report.interval

Interval between reports of the current app status in cluster mode

Default: 1s

Used when:

  • LoggingPodStatusWatcherImpl is requested to watchOrStop

spark.kubernetes.submission.waitAppCompletion

In cluster deploy mode, whether to wait for the application to finish before exiting the launcher process.

Default: true

Used when:

  • LoggingPodStatusWatcherImpl is requested to watchOrStop

spark.kubernetes.submitInDriver

(internal) Whether executing in cluster deploy mode

Default: false

spark.kubernetes.submitInDriver is true in BasicDriverFeatureStep.

Used when:


Last update: 2021-01-17