Skip to content

DriverCommandFeatureStep

DriverCommandFeatureStep is a KubernetesFeatureConfigStep.

Creating Instance

DriverCommandFeatureStep takes the following to be created:

DriverCommandFeatureStep is created when:

Configuring Pod

configurePod(
  pod: SparkPod): SparkPod

configurePod is part of the KubernetesFeatureConfigStep abstraction.

configurePod branches off based on the MainAppResource (of the KubernetesDriverConf):

Configuring Pod for Java Application

configureForJava(
  pod: SparkPod,
  res: String): SparkPod

configureForJava builds the base driver container for the given SparkPod and the primary resource.

In the end, configureForJava creates another SparkPod (for the pod of the given SparkPod) and the driver container.

Configuring Pod for Python Application

configureForPython(
  pod: SparkPod,
  res: String): SparkPod

configureForPython...FIXME

Configuring Pod for R Application

configureForR(
  pod: SparkPod,
  res: String): SparkPod

configureForR...FIXME

Base Driver ContainerBuilder

baseDriverContainer(
  pod: SparkPod,
  resource: String): ContainerBuilder

baseDriverContainer renames the given primary resource if the MainAppResource is a JavaMainAppResource. Otherwise, baseDriverContainer leaves the primary resource as-is.

baseDriverContainer creates a ContainerBuilder (for the pod of the given SparkPod) and adds the following arguments (in that order):

  1. driver
  2. --properties-file with /opt/spark/conf/spark.properties
  3. --class with the mainClass of the KubernetesDriverConf
  4. the primary resource (possibly renamed if a MainAppResource)
  5. appArgs of the KubernetesDriverConf

Note

The arguments are then used by the default entrypoint.sh of the official Docker image of Apache Spark (in resource-managers/kubernetes/docker/src/main/dockerfiles/spark/).

Use the following kubectl command to see the arguments:

kubectl get po [driverPod] -o=jsonpath='{.spec.containers[0].args}'

Last update: 2021-01-10