StateStoreWriter Physical Operators¶
StateStoreWriter is an extension of the StatefulOperator abstraction for stateful physical operators that write to a state store and collect the write metrics for execution progress reporting.
Implementations¶
- FlatMapGroupsWithStateExec
- StateStoreSaveExec
- StreamingDeduplicateExec
- StreamingGlobalLimitExec
- StreamingSymmetricHashJoinExec
Performance Metrics¶
| ID | Name |
|---|---|
| numOutputRows | number of output rows |
| numTotalStateRows | number of total state rows |
| numUpdatedStateRows | number of updated state rows |
| allUpdatesTimeMs | time to update |
| allRemovalsTimeMs | time to remove |
| commitTimeMs | time to commit changes |
| stateMemory | memory used by state |
Setting StateStore-Specific Metrics for Stateful Physical Operator¶
setStoreMetrics(
store: StateStore): Unit
setStoreMetrics requests the specified StateStore for the metrics and records the following metrics of a physical operator:
-
numTotalStateRows as the number of keys
setStoreMetrics records the custom metrics.
setStoreMetrics is used when the following physical operators are executed:
StateOperatorProgress¶
getProgress(): StateOperatorProgress
getProgress...FIXME
getProgress is used when ProgressReporter is requested to extractStateOperatorMetrics (when MicroBatchExecution is requested to run the activated streaming query).
Checking Out Whether Last Batch Execution Requires Another Non-Data Batch or Not¶
shouldRunAnotherBatch(
newMetadata: OffsetSeqMetadata): Boolean
shouldRunAnotherBatch is negative (false) by default (to indicate that another non-data batch is not required given the OffsetSeqMetadata with the event-time watermark and the batch timestamp).
shouldRunAnotherBatch is used when IncrementalExecution is requested to check out whether the last batch execution requires another batch (when MicroBatchExecution is requested to run the activated streaming query).
stateStoreCustomMetrics Internal Method¶
stateStoreCustomMetrics: Map[String, SQLMetric]
stateStoreCustomMetrics...FIXME
stateStoreCustomMetrics is used when StateStoreWriter is requested for the metrics and getProgress.