Client¶

Client submits a Spark application to run on Kubernetes (by creating the driver pod and starting a watcher that monitors and logs the application status).

Creating Instance¶

Client takes the following to be created:

• KubernetesDriverConf
• KubernetesDriverBuilder
• Kubernetes' KubernetesClient
• LoggingPodStatusWatcher

Client is created when:

• KubernetesClientApplication is requested to start

Running Driver Pod¶

run(): Unit

run requests the KubernetesDriverBuilder to build a KubernetesDriverSpec.

run requests the KubernetesDriverConf for the resourceNamePrefix to be used for the name of the driver's config map:

[resourceNamePrefix]-driver-conf-map

run builds a ConfigMap (with the name and the system properties of the KubernetesDriverSpec).

run creates a driver container (based on the KubernetesDriverSpec) and adds the following:

• SPARK_CONF_DIR env var as /opt/spark/conf
• spark-conf-volume volume mount as /opt/spark/conf

run creates a driver pod (based on the KubernetesDriverSpec) with the driver container and a new spark-conf-volume volume for the ConfigMap.

run requests the KubernetesClient to watch for the driver pod (using the LoggingPodStatusWatcher) and, when available, attaches the ConfigMap.

run is used when:

• KubernetesClientApplication is requested to start

addDriverOwnerReference¶

addDriverOwnerReference(
  driverPod: Pod,
  resources: Seq[HasMetadata]): Unit

addDriverOwnerReference...FIXME

Building ConfigMap¶

buildConfigMap(
  configMapName: String,
  conf: Map[String, String]): ConfigMap

buildConfigMap builds a Kubernetes ConfigMap with the given conf key-value pairs stored as spark.properties data and the given configMapName name.

The stored data uses an extra comment:

Java properties built from Kubernetes config map with name: [configMapName]