Skip to content

MicroBatchStream

MicroBatchStream is an extension of the SparkDataStream abstraction for streaming sources for Micro-Batch Stream Processing.

Contract

Creating PartitionReaderFactory

PartitionReaderFactory createReaderFactory()

Used when MicroBatchScanExec physical operator is requested for a PartitionReaderFactory

Latest Offset

Offset latestOffset()

Used when MicroBatchExecution is requested to constructNextBatch

Input Partitions

InputPartition[] planInputPartitions(
  Offset start,
  Offset end)

Used when MicroBatchScanExec physical operator is requested for partitions

Implementations

  • KafkaMicroBatchStream
  • MemoryStream
  • RateStreamMicroBatchStream
  • TextSocketMicroBatchStream