Skip to content

ExecutionListenerManager

ExecutionListenerManager is the management interface for Spark developers to manage session-bound QueryExecutionListeners.

Management Interface

Removing All QueryExecutionListeners

clear(): Unit

Registering QueryExecutionListener

register(
  listener: QueryExecutionListener): Unit

De-registering QueryExecutionListener

unregister(
  listener: QueryExecutionListener): Unit

SparkSession

ExecutionListenerManager is available as SparkSession.listenerManager (and SessionState.listenerManager).

scala> :type spark.listenerManager
org.apache.spark.sql.util.ExecutionListenerManager
scala> :type spark.sessionState.listenerManager
org.apache.spark.sql.util.ExecutionListenerManager

loadExtensions Flag and spark.sql.queryExecutionListeners

ExecutionListenerManager is given loadExtensions flag when created.

When enabled, ExecutionListenerManager registers the QueryExecutionListeners that are configured using the spark.sql.queryExecutionListeners configuration property.

Creating Instance

ExecutionListenerManager takes the following to be created:

ExecutionListenerManager is created when:

onSuccess

onSuccess(
  funcName: String,
  qe: QueryExecution,
  duration: Long): Unit

onSuccess...FIXME

onSuccess is used when:

onFailure

onFailure(
  funcName: String,
  qe: QueryExecution,
  exception: Exception): Unit

onFailure...FIXME

onFailure is used when:

Logging

Enable ALL logging level for org.apache.spark.sql.util.ExecutionListenerManager logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.sql.util.ExecutionListenerManager=ALL

Refer to Logging.


Last update: 2021-05-09
Back to top