StateStoreWriter Physical Operators¶

StateStoreWriter is an extension of the StatefulOperator abstraction for stateful physical operators that write to a state store and collect the write metrics for execution progress reporting.

Implementations¶

Performance Metrics¶

ID	Name
numOutputRows	number of output rows
numTotalStateRows	number of total state rows
numUpdatedStateRows	number of updated state rows
allUpdatesTimeMs	time to update
allRemovalsTimeMs	time to remove
commitTimeMs	time to commit changes
stateMemory	memory used by state

Setting StateStore-Specific Metrics for Stateful Physical Operator¶

setStoreMetrics(
  store: StateStore): Unit

setStoreMetrics requests the specified StateStore for the metrics and records the following metrics of a physical operator:

numTotalStateRows as the number of keys
stateMemory as the memory used (in bytes)

setStoreMetrics records the custom metrics.

setStoreMetrics is used when the following physical operators are executed:

StateOperatorProgress¶

getProgress(): StateOperatorProgress

getProgress...FIXME

getProgress is used when ProgressReporter is requested to extractStateOperatorMetrics (when MicroBatchExecution is requested to run the activated streaming query).

Checking Out Whether Last Batch Execution Requires Another Non-Data Batch or Not¶

shouldRunAnotherBatch(
  newMetadata: OffsetSeqMetadata): Boolean

shouldRunAnotherBatch is negative (false) by default (to indicate that another non-data batch is not required given the OffsetSeqMetadata with the event-time watermark and the batch timestamp).

shouldRunAnotherBatch is used when IncrementalExecution is requested to check out whether the last batch execution requires another batch (when MicroBatchExecution is requested to run the activated streaming query).

stateStoreCustomMetrics Internal Method¶

stateStoreCustomMetrics: Map[String, SQLMetric]

stateStoreCustomMetrics...FIXME

stateStoreCustomMetrics is used when StateStoreWriter is requested for the metrics and getProgress.