ExecutionListenerManager¶

ExecutionListenerManager is the management interface for Spark developers to manage session-bound QueryExecutionListeners.

Management Interface¶

Removing All QueryExecutionListeners¶

clear(): Unit

Registering QueryExecutionListener¶

register(
  listener: QueryExecutionListener): Unit

De-registering QueryExecutionListener¶

unregister(
  listener: QueryExecutionListener): Unit

SparkSession¶

ExecutionListenerManager is available as SparkSession.listenerManager (and SessionState.listenerManager).

scala> :type spark.listenerManager
org.apache.spark.sql.util.ExecutionListenerManager

scala> :type spark.sessionState.listenerManager
org.apache.spark.sql.util.ExecutionListenerManager

loadExtensions Flag and spark.sql.queryExecutionListeners¶

ExecutionListenerManager is given loadExtensions flag when created.

When enabled, ExecutionListenerManager registers the QueryExecutionListeners that are configured using the spark.sql.queryExecutionListeners configuration property.

Creating Instance¶

ExecutionListenerManager takes the following to be created:

• SparkSession
• loadExtensions flag

ExecutionListenerManager is created when:

• BaseSessionStateBuilder is requested for the session-bound ExecutionListenerManager (while SessionState is built)

onSuccess¶

onSuccess(
  funcName: String,
  qe: QueryExecution,
  duration: Long): Unit

onSuccess...FIXME

onSuccess is used when:

• DataFrameWriter is requested to run a logical command (after it has finished with no exceptions)
• Dataset is requested to withAction

onFailure¶

onFailure(
  funcName: String,
  qe: QueryExecution,
  exception: Exception): Unit

onFailure...FIXME

onFailure is used when:

• DataFrameWriter is requested to run a logical command (after it has reported an exception)
• Dataset is requested to withAction

Logging¶

Enable ALL logging level for org.apache.spark.sql.util.ExecutionListenerManager logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.sql.util.ExecutionListenerManager=ALL

Refer to Logging.