Skip to content

KubernetesConf

KubernetesConf is an abstraction of Kubernetes configuration metadata to build Spark pods (for the driver and executors).

Contract

annotations

annotations: Map[String, String]

Used when:

environment

environment: Map[String, String]

Used when:

labels

labels: Map[String, String]

Used when:

resourceNamePrefix

resourceNamePrefix: String

Prefix of resource names

secretEnvNamesToKeyRefs

secretEnvNamesToKeyRefs: Map[String, String]

Used when:

secretNamesToMountPaths

secretNamesToMountPaths: Map[String, String]

Used when:

Volumes

volumes: Seq[KubernetesVolumeSpec]

KubernetesVolumeSpecs

Used when:

Implementations

Creating Instance

KubernetesConf takes the following to be created:

  • SparkConf
Abstract Class

KubernetesConf is an abstract class and cannot be created directly. It is created indirectly for the concrete KubernetesConfs.

Namespace

namespace: String

namespace is the value of spark.kubernetes.namespace configuration property.

namespace is used when:

imagePullPolicy

imagePullPolicy: String

imagePullPolicy is the value of spark.kubernetes.container.image.pullPolicy configuration property.

imagePullPolicy is used when:

Creating KubernetesDriverConf

createDriverConf(
  sparkConf: SparkConf,
  appId: String,
  mainAppResource: MainAppResource,
  mainClass: String,
  appArgs: Array[String]): KubernetesDriverConf

Note

The goal of createDriverConf is to validate executor volumes before creating a KubernetesDriverConf.

createDriverConf parse volumes for executors (with spark.kubernetes.executor.volumes prefix).

Note

createDriverConf parses executor volumes in order to verify configuration before the driver pod is created.

In the end, createDriverConf creates a KubernetesDriverConf.

createDriverConf is used when:

  • KubernetesClientApplication is requested to start

Creating KubernetesExecutorConf

createExecutorConf(
  sparkConf: SparkConf,
  executorId: String,
  appId: String,
  driverPod: Option[Pod]): KubernetesExecutorConf

createExecutorConf (does nothing more but) creates a KubernetesExecutorConf for the given input arguments.

createExecutorConf is used when:

  • ExecutorPodsAllocator is requested to onNewSnapshots (and requests missing executors from Kubernetes)

AppName-Based Unique Resource Name Prefix

getResourceNamePrefix(
  appName: String): String

getResourceNamePrefix...FIXME

getResourceNamePrefix is used when:

Building Kubernetes Resource Name

buildKubernetesResourceName(
  vendorDomain: String,
  resourceName: String): String

buildKubernetesResourceName creates the following text (resource name):

[vendorDomain]/[resourceName]

buildKubernetesResourceName is used when:

Back to top