LoggingPodStatusWatcherImpl¶

LoggingPodStatusWatcherImpl is a LoggingPodStatusWatcher that monitors and logs the application status.

Creating Instance¶

LoggingPodStatusWatcherImpl takes the following to be created:

KubernetesDriverConf

LoggingPodStatusWatcherImpl is created when:

KubernetesClientApplication is requested to start

watchOrStop¶

watchOrStop(
  sId: String): Unit

watchOrStop is part of the LoggingPodStatusWatcher abstraction.

watchOrStop uses spark.kubernetes.submission.waitAppCompletion configuration property to control whether to wait for the Spark application to complete (true) or merely print out the following INFO message to the logs:

Deployed Spark application [appName] with submission ID [sId] into Kubernetes

While waiting for the Spark application to complete, watchOrStop prints out the following INFO message to the logs:

Waiting for application [appName] with submission ID [sId] to finish...

Until podCompleted flag is true, watchOrStop waits spark.kubernetes.report.interval configuration property and prints out the following INFO message to the logs:

Application status for [appId] (phase: [phase])

Once podCompleted flag is true, watchOrStop prints out the following INFO messages to the logs:

Container final statuses:

[containersDescription]

Application [appName] with submission ID [sId] finished

When no pod is available, watchOrStop prints out the following INFO message to the logs:

No containers were found in the driver pod.

eventReceived¶

eventReceived(
  action: Action,
  pod: Pod): Unit

eventReceived is part of the Kubernetes' Watcher abstraction.

eventReceived brances off based on the given Action:

For DELETED or ERROR actions, eventReceived closeWatch
For any other actions, logLongStatus followed by closeWatch if hasCompleted.

logLongStatus¶

logLongStatus(): Unit

logLongStatus prints out the following INFO message to the logs:

State changed, new state: [formatPodState|unknown]

hasCompleted¶

hasCompleted(): Boolean

hasCompleted is true when the phase is Succeeded or Failed.

hasCompleted is used when:

LoggingPodStatusWatcherImpl is requested to eventReceived (when an action is neither DELETED nor ERROR)

podCompleted Flag¶

LoggingPodStatusWatcherImpl turns podCompleted off when created.

Until podCompleted is on, LoggingPodStatusWatcherImpl waits the spark.kubernetes.report.interval configuration property and prints out the following INFO message to the logs:

Application status for [appId] (phase: [phase])

podCompleted turns podCompleted on when closeWatch.

closeWatch¶

closeWatch(): Unit

closeWatch turns podCompleted on.

closeWatch is used when:

LoggingPodStatusWatcherImpl is requested to eventReceived and onClose

Logging¶

Enable ALL logging level for org.apache.spark.deploy.k8s.submit.LoggingPodStatusWatcherImpl logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.deploy.k8s.submit.LoggingPodStatusWatcherImpl=ALL

Refer to Logging.