Skip to content

DriverServiceFeatureStep

DriverServiceFeatureStep is a KubernetesFeatureConfigStep.

Creating Instance

DriverServiceFeatureStep takes the following to be created:

DriverServiceFeatureStep is created when:

Service Name

DriverServiceFeatureStep defines Service Name based on the Preferred Service Name (if fits the name limit) or generates one:

spark-[randomServiceId]-driver-svc

DriverServiceFeatureStep prints out the following WARN message when generating the service name:

Driver's hostname would preferably be [preferredServiceName], but this is too long (must be <= 63 characters). Falling back to use $shorterServiceName as the driver service's name.

The service name is used for the following:

Preferred Service Name

DriverServiceFeatureStep uses the resourceNamePrefix (of the given KubernetesDriverConf) with -driver-svc postfix as the Preferred Service Name.

[resourceNamePrefix]-driver-svc

The preferred service name becomes the resolved service name only when shorter than 63 characters.

Additional Kubernetes Resources

getAdditionalKubernetesResources(): Seq[HasMetadata]

getAdditionalKubernetesResources is part of the KubernetesFeatureConfigStep abstraction.

getAdditionalKubernetesResources defines a Kubernetes service with the Service Name.

kubectl get services

Use k get services to list Kubernetes services.

Additional System Properties

getAdditionalPodSystemProperties(): Map[String, String]

getAdditionalPodSystemProperties is part of the KubernetesFeatureConfigStep abstraction.

getAdditionalPodSystemProperties sets the following additional Spark properties:

Name Value
spark.driver.host serviceName.namespace.svc
spark.driver.port driverPort
spark.driver.blockManager.port         driverBlockManagerPort

Configuring Driver Pod

configurePod(
  pod: SparkPod): SparkPod

configurePod is part of the KubernetesFeatureConfigStep abstraction.

configurePod...FIXME

Logging

Enable ALL logging level for org.apache.spark.deploy.k8s.features.DriverServiceFeatureStep logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.deploy.k8s.features.DriverServiceFeatureStep=ALL

Refer to Logging.


Last update: 2021-01-16