Skip to content

StateStoreProvider

StateStoreProvider is the <> of <> that manage <> in <> (e.g. for persisting running aggregates in Streaming Aggregation) in stateful streaming queries.

Note

StateStoreProvider utility uses spark.sql.streaming.stateStore.providerClass internal configuration property for the name of the class of the default <>.

[[implementations]] NOTE: HDFSBackedStateStoreProvider is the default and only known StateStoreProvider in Spark Structured Streaming.

[[contract]] .StateStoreProvider Contract [cols="30m,70",options="header",width="100%"] |=== | Method | Description

| close a| [[close]]

[source, scala]

close(): Unit

Closes the state store provider

Used exclusively when StateStore helper object is requested to unload a state store provider

| doMaintenance a| [[doMaintenance]]

[source, scala]

doMaintenance(): Unit = {}

Optional state maintenance

Used exclusively when StateStore utility is requested to perform maintenance of registered state store providers (on a separate MaintenanceTask daemon thread)

| getStore a| [[getStore]]

[source, scala]

getStore( version: Long): StateStore


Finds the StateStore for the specified version

Used exclusively when StateStore utility is requested to look up the StateStore by a given provider ID

| init a| [[init]]

[source, scala]

init( stateStoreId: StateStoreId, keySchema: StructType, valueSchema: StructType, keyIndexOrdinal: Option[Int], storeConfs: StateStoreConf, hadoopConf: Configuration): Unit


Initializes the state store provider

Used exclusively when StateStoreProvider helper object is requested to <> for a given StateStoreId (when StateStore helper object is requested to retrieve a StateStore by ID and version)

| stateStoreId a| [[stateStoreId]]

[source, scala]

stateStoreId: StateStoreId

StateStoreId associated with the provider (at <>)

Used when:

| supportedCustomMetrics a| [[supportedCustomMetrics]]

[source, scala]

supportedCustomMetrics: Seq[StateStoreCustomMetric]

<> of the state store provider

Used when:

|===

=== [[createAndInit]] Creating and Initializing StateStoreProvider -- createAndInit Object Method

[source, scala]

createAndInit( stateStoreId: StateStoreId, keySchema: StructType, valueSchema: StructType, indexOrdinal: Option[Int], storeConf: StateStoreConf, hadoopConf: Configuration): StateStoreProvider


createAndInit creates a new <> (per spark.sql.streaming.stateStore.providerClass internal configuration property).

createAndInit requests the StateStoreProvider to <>.

createAndInit is used when StateStore utility is requested for the StateStore by given provider ID and version.


Last update: 2020-11-28