Skip to content

StreamingAggregationStateManagerImplV2

StreamingAggregationStateManagerImplV2 is the default state manager for streaming aggregations.

Note

The version of a state manager is controlled using spark.sql.streaming.aggregation.stateFormatVersion internal configuration property.

Creating Instance

StreamingAggregationStateManagerImplV2 (like the parent StreamingAggregationStateManagerBaseImpl) takes the following to be created:

  • [[keyExpressions]] Catalyst expressions for the keys (Seq[Attribute])
  • [[inputRowAttributes]] Catalyst expressions for the input rows (Seq[Attribute])

StreamingAggregationStateManagerImplV2 is created when StreamingAggregationStateManager is requested for a new StreamingAggregationStateManager.

Storing Row in State Store

put(
  store: StateStore,
  row: UnsafeRow): Unit

put is part of the StreamingAggregationStateManager abstraction.

put...FIXME

Getting Saved State for Non-Null Key from State Store

get(
  store: StateStore,
  key: UnsafeRow): UnsafeRow

get is part of the StreamingAggregationStateManager abstraction.

get requests the given StateStore for the current state value for the given key.

get returns null if the key could not be found in the state store. Otherwise, get restoreOriginalRow (for the key and the saved state).

=== [[restoreOriginalRow]] restoreOriginalRow Internal Method

[source, scala]

restoreOriginalRow(key: UnsafeRow, value: UnsafeRow): UnsafeRow restoreOriginalRow(rowPair: UnsafeRowPair): UnsafeRow


restoreOriginalRow...FIXME

NOTE: restoreOriginalRow is used when StreamingAggregationStateManagerImplV2 is requested to <>, <> and <>.

getStateValueSchema

getStateValueSchema: StructType

getStateValueSchema is part of the StreamingAggregationStateManager abstraction.

getStateValueSchema simply requests the valueExpressions for the schema.

iterator

iterator(
  store: StateStore): Iterator[UnsafeRowPair]

iterator is part of the StreamingAggregationStateManager abstraction.

iterator simply requests the input state store for the iterator that is mapped to an iterator of UnsafeRowPairs with the key (of the input UnsafeRowPair) and the value as a restored original row.

Note

scala.collection.Iterator is a data structure that allows to iterate over a sequence of elements that are usually fetched lazily (i.e. no elements are fetched from the underlying store until processed).

values

values(
  store: StateStore): Iterator[UnsafeRow]

values is part of the StreamingAggregationStateManager abstraction.

values...FIXME

Internal Properties

[cols="30m,70",options="header",width="100%"] |=== | Name | Description

| joiner | [[joiner]]

| keyValueJoinedExpressions a| [[keyValueJoinedExpressions]]

| needToProjectToRestoreValue a| [[needToProjectToRestoreValue]]

| restoreValueProjector a| [[restoreValueProjector]]

| valueExpressions a| [[valueExpressions]]

| valueProjector a| [[valueProjector]] |===


Last update: 2020-11-28