Skip to content

StateStoreWriter Physical Operators

StateStoreWriter is an extension of the StatefulOperator abstraction for stateful physical operators that write to a state store and collect the write metrics for execution progress reporting.

Implementations

Performance Metrics

ID Name
numOutputRows number of output rows
numTotalStateRows number of total state rows
numUpdatedStateRows number of updated state rows
allUpdatesTimeMs time to update
allRemovalsTimeMs time to remove
commitTimeMs time to commit changes
stateMemory memory used by state

Setting StateStore-Specific Metrics for Stateful Physical Operator

setStoreMetrics(
  store: StateStore): Unit

setStoreMetrics requests the specified StateStore for the metrics and records the following metrics of a physical operator:

setStoreMetrics records the custom metrics.

setStoreMetrics is used when the following physical operators are executed:

StateOperatorProgress

getProgress(): StateOperatorProgress

getProgress...FIXME

getProgress is used when ProgressReporter is requested to extractStateOperatorMetrics (when MicroBatchExecution is requested to run the activated streaming query).

Checking Out Whether Last Batch Execution Requires Another Non-Data Batch or Not

shouldRunAnotherBatch(
  newMetadata: OffsetSeqMetadata): Boolean

shouldRunAnotherBatch is negative (false) by default (to indicate that another non-data batch is not required given the OffsetSeqMetadata with the event-time watermark and the batch timestamp).

shouldRunAnotherBatch is used when IncrementalExecution is requested to check out whether the last batch execution requires another batch (when MicroBatchExecution is requested to run the activated streaming query).

stateStoreCustomMetrics Internal Method

stateStoreCustomMetrics: Map[String, SQLMetric]

stateStoreCustomMetrics...FIXME

stateStoreCustomMetrics is used when StateStoreWriter is requested for the metrics and getProgress.