StreamingQueryProgress¶
StreamingQueryProgress
is information about a single micro-batch (progress) of a StreamingQuery:
- Unique identifier
- Unique identifier of a query execution
- Name
- Time when a trigger has started (in ISO8601 format)
- Unique ID of a micro-batch
- Batch Duration
- Durations of the internal phases (in ms)
- Statistics of the event time as seen in a batch
- StateOperatorProgress for every stateful operator
- SourceProgress for every streaming source
- SinkProgress
- Observed Metrics
StreamingQueryProgress
is created when StreamExecution
is requested to finish a trigger.
Last and Recent Progresses¶
Use lastProgress property of a StreamingQuery
to access the most recent StreamingQueryProgress
update.
val sq: StreamingQuery = ...
sq.lastProgress
Use recentProgress property of a StreamingQuery
to access the most recent StreamingQueryProgress
updates.
val sq: StreamingQuery = ...
sq.recentProgress
StreamingQueryListener¶
Use StreamingQueryListener to be notified about StreamingQueryProgress
updates while a streaming query is executed.