ExecutionListenerManager¶

ExecutionListenerManager is the <> for QueryExecutionListeners that listen for execution metrics:

• Name of the action (that triggered a query execution)

• QueryExecution

• Execution time of this query (in nanoseconds)

ExecutionListenerManager is available as SparkSession.md#listenerManager[listenerManager] property of SparkSession (and SessionState.md#listenerManager[listenerManager] property of SessionState).

[source, scala]¶

scala> :type spark.listenerManager org.apache.spark.sql.util.ExecutionListenerManager

scala> :type spark.sessionState.listenerManager org.apache.spark.sql.util.ExecutionListenerManager

[[conf]] [[creating-instance]] ExecutionListenerManager takes a single SparkConf when created

While <>, ExecutionListenerManager reads StaticSQLConf.md#spark.sql.queryExecutionListeners[spark.sql.queryExecutionListeners] configuration property with QueryExecutionListeners and <> them.

[[spark.sql.queryExecutionListeners]] ExecutionListenerManager uses StaticSQLConf.md#spark.sql.queryExecutionListeners[spark.sql.queryExecutionListeners] configuration property as the list of QueryExecutionListeners that should be automatically added to newly created sessions (and registers them while <>).

[[methods]] .ExecutionListenerManager's Public Methods [cols="1,2",options="header",width="100%"] |=== | Method | Description

| <> a|

[source, scala]¶

register(listener: QueryExecutionListener): Unit¶

| <> a|

[source, scala]¶

unregister(listener: QueryExecutionListener): Unit¶

| <> a|

[source, scala]¶

clear(): Unit¶

|===

ExecutionListenerManager is <> exclusively when BaseSessionStateBuilder is requested for BaseSessionStateBuilder.md#listenerManager[ExecutionListenerManager] (while SessionState is BaseSessionStateBuilder.md#build[built]).

[[listeners]] ExecutionListenerManager uses listeners internal registry for registered QueryExecutionListeners.

onSuccess Internal Method¶

onSuccess(
  funcName: String,
  qe: QueryExecution,
  duration: Long): Unit

onSuccess...FIXME

onSuccess is used when:

• DataFrameWriter is requested to run a logical command (after it has finished with no exceptions)

• Dataset is requested to withAction

onFailure Internal Method¶

onFailure(
  funcName: String,
  qe: QueryExecution,
  exception: Exception): Unit

onFailure...FIXME

onFailure is used when:

• DataFrameWriter is requested to run a logical command (after it has reported an exception)

• Dataset is requested to withAction

=== [[withErrorHandling]] withErrorHandling Internal Method

[source, scala]¶

withErrorHandling(f: QueryExecutionListener => Unit): Unit¶

withErrorHandling...FIXME

NOTE: withErrorHandling is used when ExecutionListenerManager is requested to <> and <>.

=== [[register]] Registering QueryExecutionListener -- register Method

[source, scala]¶

register(listener: QueryExecutionListener): Unit¶

Internally, register simply registers (adds) the input QueryExecutionListener to the listeners internal registry.