MicroBatchStream¶

MicroBatchStream is an extension of the SparkDataStream abstraction for streaming sources for Micro-Batch Stream Processing.

Contract¶

Creating PartitionReaderFactory¶

PartitionReaderFactory createReaderFactory()

Used when MicroBatchScanExec physical operator is requested for a PartitionReaderFactory

Latest Offset¶

Offset latestOffset()

Used when MicroBatchExecution is requested to constructNextBatch

Input Partitions¶

InputPartition[] planInputPartitions(
  Offset start,
  Offset end)

Used when MicroBatchScanExec physical operator is requested for partitions

Implementations¶

• KafkaMicroBatchStream
• MemoryStream
• RateStreamMicroBatchStream
• TextSocketMicroBatchStream