Skip to content

The Internals of Spark Structured Streaming

StateStoreSaveExec with Update Output Mode

Initializing search

spark-structured-streaming-internals

Home
Internals
Streaming Operators
Data Sources
Monitoring
Web UI
Demos

The Internals of Spark Structured Streaming

spark-structured-streaming-internals

Home
Internals
Internals
- Overview
- StreamingQuery
- Configuration Properties
- Batch Processing Time
- Internals of Streaming Queries
- DataSource
- SparkDataStream
- SupportsAdmissionControl
- MicroBatchStream
- ContinuousStream
- Trigger
- Streaming Join
  Streaming Join
  - Streaming Join
  - StateStoreAwareZipPartitionsRDD
  - SymmetricHashJoinStateManager
    SymmetricHashJoinStateManager
    
    SymmetricHashJoinStateManager
    
    StateStoreHandler
    
    KeyToNumValuesStore
    
    KeyWithIndexToValueStore
  - OneSideHashJoiner
  - JoinStateWatermarkPredicates
  - JoinStateWatermarkPredicate
  - StateStoreAwareZipPartitionsHelper
    StateStoreAwareZipPartitionsHelper
    
    StateStoreAwareZipPartitionsHelper
    
    StreamingSymmetricHashJoinHelper
    
    StreamingJoinHelper
- Streaming Aggregation
  Streaming Aggregation
- Stateful Stream Processing
  Stateful Stream Processing
  - Stateful Stream Processing
  - Streaming Watermark
  - Streaming Deduplication
  - Streaming Limit
  - StateStore
    StateStore
    
    StateStore
    
    StateStoreId
    
    HDFSBackedStateStore
  - StateStoreProvider
    StateStoreProvider
    
    StateStoreProvider
    
    StateStoreProviderId
    
    HDFSBackedStateStoreProvider
  - StateStoreCoordinator
    StateStoreCoordinator
    
    StateStoreCoordinator
    
    StateStoreCoordinatorRef
  - WatermarkSupport
  - StatefulOperatorStateInfo
  - StateStoreMetrics
  - StateStoreCustomMetric
  - StateStoreUpdater
  - EventTimeStatsAccum
  - EventTimeStats
  - StateStoreConf
- Arbitrary Stateful Streaming Aggregation
  Arbitrary Stateful Streaming Aggregation
  - Arbitrary Stateful Streaming Aggregation
  - KeyValueGroupedDataset
    KeyValueGroupedDataset
    
    KeyValueGroupedDataset
    
    mapGroupsWithState Operator
    
    flatMapGroupsWithState Operator
  - GroupState
    GroupState
    
    GroupState
    
    GroupStateImpl
  - GroupStateTimeout
  - StateManager
    StateManager
    
    StateManager
    
    StateManagerImplV2
    
    StateManagerImplBase
    
    StateManagerImplV1
  - FlatMapGroupsWithStateExecHelper Helper Class
  - InputProcessor
- StreamingQueryManager
- StreamingQueryListenerBus
- Developer-Facing APIs
  Developer-Facing APIs
- Query Execution
  Query Execution
- StreamMetadata
- Logical Operators
  Logical Operators
- Physical Operators
  Physical Operators
- Execution Planning Strategies
  Execution Planning Strategies
- Offsets and Metadata Checkpointing
  Offsets and Metadata Checkpointing
  - Offsets and Metadata Checkpointing
  - MetadataLog
  - HDFSMetadataLog
  - CommitLog
    CommitLog
    
    CommitLog
    
    CommitMetadata
  - OffsetSeqLog
    OffsetSeqLog
    
    OffsetSeqLog
    
    OffsetSeq
  - OffsetSeqMetadata
  - CheckpointFileManager
    CheckpointFileManager
    
    CheckpointFileManager
    
    FileContextBasedCheckpointFileManager
    
    FileSystemBasedCheckpointFileManager
    
    Offset
    
    StreamProgress
- Micro-Batch Execution
  Micro-Batch Execution
- Continuous Execution
  Continuous Execution
  - ContinuousExecution
  - ContinuousReadSupport
  - ContinuousReader
  - EpochCoordinator
    EpochCoordinator
    
    EpochCoordinator RPC Endpoint
    
    EpochCoordinatorRef
    
    EpochTracker
  - ContinuousQueuedDataReader
    ContinuousQueuedDataReader
    
    ContinuousQueuedDataReader
    
    DataReaderThread
    
    EpochMarkerGenerator
  - PartitionOffset
  - ContinuousWriteRDD
  - ContinuousDataSourceRDD
- UnsupportedOperationChecker
- Extending Structured Streaming with New Data Sources
- SQLConf
Streaming Operators
Streaming Operators
Data Sources
Data Sources
- File
  File
- Kafka
  Kafka
  - KafkaSourceProvider
  - KafkaMicroBatchStream
  - KafkaTable
  - KafkaSource
  - KafkaRelation
  - KafkaSourceRDD
  - CachedKafkaConsumer
  - KafkaSourceOffset
  - KafkaOffsetReader
  - ConsumerStrategy
  - KafkaSink
  - KafkaBatch
  - KafkaScan
  - KafkaOffsetRangeLimit
  - KafkaDataConsumer
  - KafkaMicroBatchReader
    KafkaMicroBatchReader
    
    KafkaMicroBatchReader
    
    KafkaOffsetRangeCalculator
    
    KafkaMicroBatchInputPartition
    
    KafkaMicroBatchInputPartitionReader
    
    KafkaSourceInitialOffsetWriter
  - KafkaContinuousReader
    KafkaContinuousReader
    
    KafkaContinuousReader
    
    KafkaContinuousInputPartition
- Text Socket
  Text Socket
  - TextSocketSourceProvider
  - TextSocketSource
- Rate
  Rate
- Console Sink
  Console Sink
  - ConsoleSinkProvider
  - ConsoleWriter
- Foreach Sink
  Foreach Sink
- ForeachBatch Sink
  ForeachBatch Sink
  - ForeachBatchSink
- Memory
  Memory
  - MemoryStream
  - ContinuousMemoryStream
  - MemorySink
  - MemorySinkV2
    MemorySinkV2
    
    MemorySinkV2
    
    MemoryStreamWriter
  - MemoryStreamBase
  - MemorySinkBase
  - MemoryPlan
Monitoring
Monitoring
- StreamingQueryListener
- ProgressReporter
- StreamingQueryProgress
  StreamingQueryProgress
- StreamingQueryStatus
- MetricsReporter
- Logging
Web UI
Web UI
- Web UI
Demos
Demos

Demo: StateStoreSaveExec with Update Output Mode¶

CAUTION: FIXME Example of Update with StateStoreSaveExec (and optional watermark)

Last update: 2021-01-24

Previous StateStoreSaveExec with Complete Output Mode

Next Developing Custom Streaming Sink (and Monitoring SQL Queries in web UI)

Copyright © 2021 Jacek Laskowski

Made with Material for MkDocs Insiders