DriverServiceFeatureStep¶

DriverServiceFeatureStep is a KubernetesFeatureConfigStep.

Creating Instance¶

DriverServiceFeatureStep takes the following to be created:

KubernetesDriverConf
Clock (default: SystemClock)

DriverServiceFeatureStep is created when:

KubernetesDriverBuilder is requested for a driver pod spec

Service Name¶

DriverServiceFeatureStep defines Service Name based on the Preferred Service Name (if fits the name limit) or generates one:

spark-[randomServiceId]-driver-svc

DriverServiceFeatureStep prints out the following WARN message when generating the service name:

Driver's hostname would preferably be [preferredServiceName], but this is too long (must be <= 63 characters). Falling back to use $shorterServiceName as the driver service's name.

The service name is used for the following:

Additional System Properties (as spark.driver.host)
Additional Kubernetes Resources

Preferred Service Name¶

DriverServiceFeatureStep uses the resourceNamePrefix (of the given KubernetesDriverConf) with -driver-svc postfix as the Preferred Service Name.

[resourceNamePrefix]-driver-svc

The preferred service name becomes the resolved service name only when shorter than 63 characters.

Additional Kubernetes Resources¶

getAdditionalKubernetesResources(): Seq[HasMetadata]

getAdditionalKubernetesResources is part of the KubernetesFeatureConfigStep abstraction.

getAdditionalKubernetesResources defines a Kubernetes service with the Service Name.

kubectl get services

Use k get services to list Kubernetes services.

Additional System Properties¶

getAdditionalPodSystemProperties(): Map[String, String]

getAdditionalPodSystemProperties is part of the KubernetesFeatureConfigStep abstraction.

getAdditionalPodSystemProperties sets the following additional Spark properties:

Name	Value
`spark.driver.host`	serviceName`.`namespace`.svc`
`spark.driver.port`	driverPort
`spark.driver.blockManager.port`	driverBlockManagerPort

Configuring Driver Pod¶

configurePod(
  pod: SparkPod): SparkPod

configurePod is part of the KubernetesFeatureConfigStep abstraction.

configurePod...FIXME

Logging¶

Enable ALL logging level for org.apache.spark.deploy.k8s.features.DriverServiceFeatureStep logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.deploy.k8s.features.DriverServiceFeatureStep=ALL

Refer to Logging.