StreamingQueryProgress¶

StreamingQueryProgress is information about a single micro-batch (progress) of a StreamingQuery:

• Unique identifier
• Unique identifier of a query execution
• Name
• Time when a trigger has started (in ISO8601 format)
• Unique ID of a micro-batch
• Batch Duration
• Durations of the internal phases (in ms)
• Statistics of the event time as seen in a batch
• StateOperatorProgress for every stateful operator
• SourceProgress for every streaming source
• SinkProgress
• Observed Metrics

StreamingQueryProgress is created when StreamExecution is requested to finish a trigger.

Last and Recent Progresses¶

Use lastProgress property of a StreamingQuery to access the most recent StreamingQueryProgress update.

val sq: StreamingQuery = ...
sq.lastProgress

Use recentProgress property of a StreamingQuery to access the most recent StreamingQueryProgress updates.

val sq: StreamingQuery = ...
sq.recentProgress

StreamingQueryListener¶

Use StreamingQueryListener to be notified about StreamingQueryProgress updates while a streaming query is executed.