Skip to content

StreamingQueryListenerBus

StreamingQueryListenerBus is an event bus for dispatching streaming events (of active streaming queries) to StreamingQueryListeners.

Tip

Learn more about event buses in The Internals of Apache Spark online book.

Creating Instance

StreamingQueryListenerBus takes the following to be created:

When created, StreamingQueryListenerBus registers itself with the LiveListenerBus to streams event queue.

StreamingQueryListenerBus is created for StreamingQueryManager (once per SparkSession).

StreamingQueryListenerBus is Created Once In SparkSession

SparkListener

StreamingQueryListenerBus is an event listener (SparkListener) and registers itself with the LiveListenerBus to intercept QueryStartedEvents.

Tip

Learn more about SparkListener in The Internals of Apache Spark online book.

Run IDs of Active Streaming Queries

activeQueryRunIds: Set[UUID]

activeQueryRunIds is an internal registry of run IDs of active streaming queries in the SparkSession.

activeQueryRunIds is used internally to dispatch a streaming event to a StreamingQueryListener (so the events gets sent out to streaming queries in the SparkSession).

Posting Streaming Event to LiveListenerBus

post(
  event: StreamingQueryListener.Event): Unit

post simply posts the input event directly to the LiveListenerBus unless it is a QueryStartedEvent.

For a QueryStartedEvent, post adds the runId (of the streaming query that has been started) to the activeQueryRunIds internal registry first, posts the event to the LiveListenerBus and then postToAll.

post is used when StreamingQueryManager is requested to post a streaming event.

Notifying Listener about Event

doPostEvent(
  listener: StreamingQueryListener,
  event: StreamingQueryListener.Event): Unit

doPostEvent is part of the ListenerBus (Spark Core) abstraction.

doPostEvent branches per the type of StreamingQueryListener.Event:

For any other event, doPostEvent simply does nothing (swallows it).

Posting Event To All Listeners

postToAll(
  event: Event): Unit

postToAll is part of the ListenerBus (Spark Core) abstraction.

postToAll first requests the parent ListenerBus to post the event to all registered listeners.

For a QueryTerminatedEvent, postToAll simply removes the runId (of the streaming query that has been terminated) from the activeQueryRunIds internal registry.


Last update: 2020-11-28