KafkaSourceOffset¶
KafkaSourceOffset
is an Offset for kafka data source.
KafkaSourceOffset
is <
-
KafkaContinuousReader
is requested to setStartOffset, deserializeOffset, and mergeOffsets -
KafkaMicroBatchReader
is requested to getStartOffset, getEndOffset, deserializeOffset, and getOrCreateInitialPartitionOffsets -
KafkaOffsetReader
is requested to fetchSpecificOffsets -
KafkaSource
is requested for the initial partition offsets (of 0th batch) and getOffset -
KafkaSourceInitialOffsetWriter
is requested to deserialize a KafkaSourceOffset (from an InputStream) -
KafkaSourceOffset
is requested for <>
[[creating-instance]][[partitionToOffsets]] KafkaSourceOffset
takes a collection of Kafka TopicPartitions
with offsets to be created.
=== [[getPartitionOffsets]] Partition Offsets -- getPartitionOffsets
Method
[source, scala]¶
getPartitionOffsets( offset: Offset): Map[TopicPartition, Long]
getPartitionOffsets
takes <offset
.
If offset
is KafkaSourceOffset
, getPartitionOffsets
takes the partitions and offsets straight from it.
If however offset
is SerializedOffset
, getPartitionOffsets
deserializes the offsets from JSON.
getPartitionOffsets
reports an IllegalArgumentException
when offset
is neither KafkaSourceOffset
or SerializedOffset
.
Invalid conversion from offset of [class] to KafkaSourceOffset
getPartitionOffsets
is used when:
-
KafkaContinuousReader
is requested to planInputPartitions -
KafkaSource
is requested to generate a streaming DataFrame with records from Kafka for a streaming micro-batch
=== [[json]] JSON-Encoded Offset -- json
Method
[source, scala]¶
json: String¶
json
is part of the Offset abstraction.
json
...FIXME
=== [[apply]] Creating KafkaSourceOffset Instance -- apply
Utility Method
[source, scala]¶
apply( offsetTuples: (String, Int, Long)*): KafkaSourceOffset // <1> apply( offset: SerializedOffset): KafkaSourceOffset
<1> Used in tests only
apply
...FIXME
apply
is used when:
-
KafkaSourceInitialOffsetWriter
is requested to deserialize a KafkaSourceOffset (from an InputStream) -
KafkaSource
is requested for the initial partition offsets (of 0th batch) -
KafkaSourceOffset
is requested to getPartitionOffsets