The Internals of Spark Structured Streaming
ExecutionStats
Initializing search
spark-structured-streaming-book
Home
Streaming Operators
Data Sources
Monitoring
Web UI
Demos
The Internals of Spark Structured Streaming
spark-structured-streaming-book
Home
Home
Welcome
Spark Structured Streaming and Streaming Queries
StreamingQuery
Configuration Properties
Batch Processing Time
Internals of Streaming Queries
DataSource
SparkDataStream
SupportsAdmissionControl
MicroBatchStream
ContinuousStream
Trigger
Streaming Join
Streaming Join
Streaming Join
StateStoreAwareZipPartitionsRDD
SymmetricHashJoinStateManager
SymmetricHashJoinStateManager
SymmetricHashJoinStateManager
StateStoreHandler
KeyToNumValuesStore
KeyWithIndexToValueStore
OneSideHashJoiner
JoinStateWatermarkPredicates
JoinStateWatermarkPredicate
StateStoreAwareZipPartitionsHelper
StateStoreAwareZipPartitionsHelper
StateStoreAwareZipPartitionsHelper
StreamingSymmetricHashJoinHelper
StreamingJoinHelper
Streaming Aggregation
Streaming Aggregation
Streaming Aggregation
StateStoreRDD
StateStoreOps
StreamingAggregationStateManager
StreamingAggregationStateManagerBaseImpl
StreamingAggregationStateManagerImplV1
StreamingAggregationStateManagerImplV2
Stateful Stream Processing
Stateful Stream Processing
Stateful Stream Processing
Streaming Watermark
Streaming Deduplication
Streaming Limit
StateStore
StateStore
StateStore
StateStoreId
HDFSBackedStateStore
StateStoreProvider
StateStoreProvider
StateStoreProvider
StateStoreProviderId
HDFSBackedStateStoreProvider
StateStoreCoordinator
StateStoreCoordinator
StateStoreCoordinator
StateStoreCoordinatorRef
WatermarkSupport
StatefulOperatorStateInfo
StateStoreMetrics
StateStoreCustomMetric
StateStoreUpdater
EventTimeStatsAccum
EventTimeStats
StateStoreConf
Arbitrary Stateful Streaming Aggregation
Arbitrary Stateful Streaming Aggregation
Arbitrary Stateful Streaming Aggregation
KeyValueGroupedDataset
KeyValueGroupedDataset
KeyValueGroupedDataset
mapGroupsWithState Operator
flatMapGroupsWithState Operator
GroupState
GroupState
GroupState
GroupStateImpl
GroupStateTimeout
StateManager
StateManager
StateManager
StateManagerImplV2
StateManagerImplBase
StateManagerImplV1
FlatMapGroupsWithStateExecHelper Helper Class
InputProcessor
StreamingQueryManager
StreamingQueryListenerBus
Developer-Facing APIs
Developer-Facing APIs
DataStreamReader
DataStreamWriter
OutputMode
window Function
Query Execution
Query Execution
StreamExecution
IncrementalExecution
TriggerExecutor
StreamingQueryWrapper
StreamMetadata
Logical Operators
Logical Operators
EventTimeWatermark
FlatMapGroupsWithState
Deduplicate
StreamingDataSourceV2Relation
StreamingRelation
StreamingRelationV2
StreamingExecutionRelation
Physical Operators
Physical Operators
MicroBatchScanExec
EventTimeWatermarkExec
FlatMapGroupsWithStateExec
StatefulOperator
StateStoreReader
StateStoreRestoreExec
StateStoreSaveExec
StateStoreWriter
StreamingDeduplicateExec
StreamingGlobalLimitExec
StreamingRelationExec
StreamingSymmetricHashJoinExec
Execution Planning Strategies
Execution Planning Strategies
FlatMapGroupsWithStateStrategy
StatefulAggregationStrategy
StreamingDeduplicationStrategy
StreamingGlobalLimitStrategy
StreamingJoinStrategy
StreamingRelationStrategy
Offsets and Metadata Checkpointing
Offsets and Metadata Checkpointing
Offsets and Metadata Checkpointing
MetadataLog
HDFSMetadataLog
CommitLog
CommitLog
CommitLog
CommitMetadata
OffsetSeqLog
OffsetSeqLog
OffsetSeqLog
OffsetSeq
OffsetSeqMetadata
CheckpointFileManager
CheckpointFileManager
CheckpointFileManager
FileContextBasedCheckpointFileManager
FileSystemBasedCheckpointFileManager
Offset
StreamProgress
Micro-Batch Stream Processing
Micro-Batch Stream Processing
Micro-Batch Stream Processing
MicroBatchExecution
MicroBatchExecution
MicroBatchExecution
MicroBatchWriter
MicroBatchReader
Source
Source
Source
StreamSourceProvider
Sink
StreamSinkProvider
WatermarkTracker
Continuous Stream Processing
Continuous Stream Processing
Continuous Stream Processing
ContinuousExecution
ContinuousReadSupport
ContinuousReader
RateStreamContinuousReader
EpochCoordinator
EpochCoordinator
EpochCoordinator RPC Endpoint
EpochCoordinatorRef
EpochTracker
ContinuousQueuedDataReader
ContinuousQueuedDataReader
ContinuousQueuedDataReader
DataReaderThread
EpochMarkerGenerator
PartitionOffset
ContinuousExecutionRelation
WriteToContinuousDataSource
WriteToContinuousDataSourceExec
WriteToContinuousDataSourceExec
WriteToContinuousDataSourceExec Unary Physical Operator
ContinuousWriteRDD
ContinuousDataSourceRDD
UnsupportedOperationChecker
Extending Structured Streaming with New Data Sources
SQLConf
Streaming Operators
Streaming Operators
Streaming Operators
crossJoin
dropDuplicates
explain
flatMapGroupsWithState
groupBy
groupByKey
join
joinWith
withWatermark
writeStream
Data Sources
Data Sources
Data Sources
File
File
File Data Source
FileStreamSource
FileStreamSink
CompactibleFileStreamLog
SinkFileStatus
ManifestFileCommitProtocol
MetadataLogFileIndex
FileStreamSourceCleaner
FileStreamSourceLog
FileStreamSinkLog
Kafka
Kafka
Kafka Data Source
KafkaSourceProvider
KafkaMicroBatchStream
KafkaTable
KafkaSource
KafkaRelation
KafkaSourceRDD
CachedKafkaConsumer
KafkaSourceOffset
KafkaOffsetReader
ConsumerStrategy
KafkaSink
KafkaBatch
KafkaScan
KafkaOffsetRangeLimit
KafkaDataConsumer
KafkaMicroBatchReader
KafkaMicroBatchReader
KafkaMicroBatchReader
KafkaOffsetRangeCalculator
KafkaMicroBatchInputPartition
KafkaMicroBatchInputPartitionReader
KafkaSourceInitialOffsetWriter
KafkaContinuousReader
KafkaContinuousReader
KafkaContinuousReader
KafkaContinuousInputPartition
Socket
Socket
TextSocketSourceProvider
TextSocketSource
Rate
Rate
RateSourceProvider
RateStreamProvider
RateStreamSource
RateStreamMicroBatchReader
Console Sink
Console Sink
ConsoleSinkProvider
ConsoleWriter
Foreach Sink
Foreach Sink
ForeachWriterProvider
ForeachWriter
ForeachSink
ForeachBatch Sink
ForeachBatch Sink
ForeachBatchSink
Memory
Memory
Memory Data Source
MemoryStream
ContinuousMemoryStream
MemorySink
MemorySinkV2
MemorySinkV2
MemorySinkV2
MemoryStreamWriter
MemoryStreamBase
MemorySinkBase
MemoryPlan
Monitoring
Monitoring
StreamingQueryListener
ProgressReporter
StreamingQueryProgress
StreamingQueryProgress
StreamingQueryProgress
StateOperatorProgress
ExecutionStats
SourceProgress
SinkProgress
StreamingQueryStatus
MetricsReporter
Logging
Web UI
Web UI
Web UI
Demos
Demos
Internals of FlatMapGroupsWithStateExec Physical Operator
Arbitrary Stateful Streaming Aggregation with KeyValueGroupedDataset.flatMapGroupsWithState Operator
Exploring Checkpointed State
Streaming Watermark with Aggregation in Append Output Mode
Streaming Query for Running Counts (Socket Source and Complete Output Mode)
Streaming Aggregation with Kafka Data Source
groupByKey Streaming Aggregation in Update Mode
StateStoreSaveExec with Complete Output Mode
StateStoreSaveExec with Update Output Mode
Developing Custom Streaming Sink (and Monitoring SQL Queries in web UI)
current_timestamp Function For Processing Time in Streaming Queries
Using StreamingQueryManager for Query Termination Management
Using File Streaming Source
Deep Dive into FileStreamSink
ExecutionStats
== [[ExecutionStats]] ExecutionStats
ExecutionStats
is...FIXME