K8SSparkSubmitOperation¶
K8SSparkSubmitOperation is an extension of spark-submit (Apache Spark) for Spark on Kubernetes to support the operations:
K8SSparkSubmitOperation is registered with Apache Spark using META-INF/services/org.apache.spark.deploy.SparkSubmitOperation service file.
Killing Submission¶
kill(
submissionId: String,
conf: SparkConf): Unit
kill is part of the SparkSubmitOperation (Apache Spark) abstraction.
kill prints out the following message to standard error:
Submitting a request to kill submission [submissionId] in [spark.master]. Grace period in secs: [[getGracePeriod] | not set].
kill creates a KillApplication to execute it (with the input submissionId and SparkConf).
Displaying Submission Status¶
printSubmissionStatus(
submissionId: String,
conf: SparkConf): Unit
printSubmissionStatus is part of the SparkSubmitOperation (Apache Spark) abstraction.
printSubmissionStatus prints out the following message to standard error:
Submitting a request for the status of submission [submissionId] in [spark.master].
printSubmissionStatus creates a ListStatus to execute it (with the input submissionId and SparkConf).
Checking Whether Master URL Supported¶
supports(
master: String): Boolean
supports is part of the SparkSubmitOperation (Apache Spark) abstraction.
supports is true when the input master starts with k8s:// prefix.
Executing Operation¶
execute(
submissionId: String,
sparkConf: SparkConf,
op: K8sSubmitOp): Unit
execute is used for kill and printSubmissionStatus.
execute parses the master URL (based on spark.master configuration property).
execute uses : (the colon) to split the given submissionId to at most two parts - an optional namespace and a driver pod name. The driver pod name can use a glob pattern as * (the star).
execute creates a KubernetesClient (with Submission client type).
If the pod name uses the glob pattern (with *), execute requests the KubernetesClient for the driver pods (pods with the spark-role label with driver value) that it then hands over to the given K8sSubmitOp to executeOnGlob (with the optional namespace).
Otherwise, execute requests the given K8sSubmitOp to executeOnPod with the pod and the optional namespace.
execute prints out the following message and exits when the given submissionId cannot be split on : to two parts:
Submission ID: {[submissionId]} is invalid.