Skip to content

Client

Client submits a Spark application to run on Kubernetes (by creating the driver pod and starting a watcher that monitors and logs the application status).

Creating Instance

Client takes the following to be created:

Client is created when:

  • KubernetesClientApplication is requested to start

Running Driver Pod

run(): Unit

run requests the KubernetesDriverBuilder to build a KubernetesDriverSpec.

run requests the KubernetesDriverConf for the resourceNamePrefix to be used for the name of the driver's config map:

[resourceNamePrefix]-driver-conf-map

run builds a ConfigMap (with the name and the system properties of the KubernetesDriverSpec).

run creates a driver container (based on the KubernetesDriverSpec) and adds the following:

  • SPARK_CONF_DIR env var as /opt/spark/conf
  • spark-conf-volume volume mount as /opt/spark/conf

run creates a driver pod (based on the KubernetesDriverSpec) with the driver container and a new spark-conf-volume volume for the ConfigMap.

run requests the KubernetesClient to watch for the driver pod (using the LoggingPodStatusWatcher) and, when available, attaches the ConfigMap.

run is used when:

  • KubernetesClientApplication is requested to start

addDriverOwnerReference

addDriverOwnerReference(
  driverPod: Pod,
  resources: Seq[HasMetadata]): Unit

addDriverOwnerReference...FIXME

Building ConfigMap

buildConfigMap(
  configMapName: String,
  conf: Map[String, String]): ConfigMap

buildConfigMap builds a Kubernetes ConfigMap with the given conf key-value pairs stored as spark.properties data and the given configMapName name.

The stored data uses an extra comment:

Java properties built from Kubernetes config map with name: [configMapName]
Kubernetes Documentation

Learn more about ConfigMaps in the official Kubernetes Documentation.

Demo: Spark and Local Filesystem in minikube

Learn more about ConfigMaps and volumes in Demo: Spark and Local Filesystem in minikube.


Last update: 2021-01-16