MicroBatchStream¶
MicroBatchStream
is an extension of the SparkDataStream abstraction for streaming sources for Micro-Batch Stream Processing.
Contract¶
Creating PartitionReaderFactory¶
PartitionReaderFactory createReaderFactory()
Used when MicroBatchScanExec
physical operator is requested for a PartitionReaderFactory
Latest Offset¶
Offset latestOffset()
Used when MicroBatchExecution
is requested to constructNextBatch
Input Partitions¶
InputPartition[] planInputPartitions(
Offset start,
Offset end)
Used when MicroBatchScanExec
physical operator is requested for partitions
Implementations¶
- KafkaMicroBatchStream
- MemoryStream
- RateStreamMicroBatchStream
- TextSocketMicroBatchStream