StateStoreWriter Physical Operators¶
StateStoreWriter
is an extension of the StatefulOperator abstraction for stateful physical operators that write to a state store and collect the write metrics for execution progress reporting.
Implementations¶
- FlatMapGroupsWithStateExec
- StateStoreSaveExec
- StreamingDeduplicateExec
- StreamingGlobalLimitExec
- StreamingSymmetricHashJoinExec
Performance Metrics¶
ID | Name |
---|---|
numOutputRows | number of output rows |
numTotalStateRows | number of total state rows |
numUpdatedStateRows | number of updated state rows |
allUpdatesTimeMs | time to update |
allRemovalsTimeMs | time to remove |
commitTimeMs | time to commit changes |
stateMemory | memory used by state |
Setting StateStore-Specific Metrics for Stateful Physical Operator¶
setStoreMetrics(
store: StateStore): Unit
setStoreMetrics
requests the specified StateStore for the metrics and records the following metrics of a physical operator:
-
numTotalStateRows as the number of keys
setStoreMetrics
records the custom metrics.
setStoreMetrics
is used when the following physical operators are executed:
StateOperatorProgress¶
getProgress(): StateOperatorProgress
getProgress
...FIXME
getProgress
is used when ProgressReporter
is requested to extractStateOperatorMetrics (when MicroBatchExecution
is requested to run the activated streaming query).
Checking Out Whether Last Batch Execution Requires Another Non-Data Batch or Not¶
shouldRunAnotherBatch(
newMetadata: OffsetSeqMetadata): Boolean
shouldRunAnotherBatch
is negative (false
) by default (to indicate that another non-data batch is not required given the OffsetSeqMetadata with the event-time watermark and the batch timestamp).
shouldRunAnotherBatch
is used when IncrementalExecution
is requested to check out whether the last batch execution requires another batch (when MicroBatchExecution
is requested to run the activated streaming query).
stateStoreCustomMetrics Internal Method¶
stateStoreCustomMetrics: Map[String, SQLMetric]
stateStoreCustomMetrics
...FIXME
stateStoreCustomMetrics
is used when StateStoreWriter
is requested for the metrics and getProgress.