KafkaSourceOffset¶

KafkaSourceOffset is an Offset for kafka data source.

KafkaSourceOffset is <> (directly or indirectly using <>) when:

• KafkaContinuousReader is requested to setStartOffset, deserializeOffset, and mergeOffsets

• KafkaMicroBatchReader is requested to getStartOffset, getEndOffset, deserializeOffset, and getOrCreateInitialPartitionOffsets

• KafkaOffsetReader is requested to fetchSpecificOffsets

• KafkaSource is requested for the initial partition offsets (of 0^th batch) and getOffset

• KafkaSourceInitialOffsetWriter is requested to deserialize a KafkaSourceOffset (from an InputStream)

• KafkaSourceOffset is requested for <>

[[creating-instance]][[partitionToOffsets]] KafkaSourceOffset takes a collection of Kafka TopicPartitions with offsets to be created.

=== [[getPartitionOffsets]] Partition Offsets -- getPartitionOffsets Method

[source, scala]¶

getPartitionOffsets( offset: Offset): Map[TopicPartition, Long]

getPartitionOffsets takes <> from offset.

If offset is KafkaSourceOffset, getPartitionOffsets takes the partitions and offsets straight from it.

If however offset is SerializedOffset, getPartitionOffsets deserializes the offsets from JSON.

getPartitionOffsets reports an IllegalArgumentException when offset is neither KafkaSourceOffset or SerializedOffset.

Invalid conversion from offset of [class] to KafkaSourceOffset

getPartitionOffsets is used when:

• KafkaContinuousReader is requested to planInputPartitions

• KafkaSource is requested to generate a streaming DataFrame with records from Kafka for a streaming micro-batch

=== [[json]] JSON-Encoded Offset -- json Method

[source, scala]¶

json: String¶

json is part of the Offset abstraction.

json...FIXME

=== [[apply]] Creating KafkaSourceOffset Instance -- apply Utility Method

[source, scala]¶

apply( offsetTuples: (String, Int, Long)*): KafkaSourceOffset // <1> apply( offset: SerializedOffset): KafkaSourceOffset

<1> Used in tests only

apply...FIXME

apply is used when:

• KafkaSourceInitialOffsetWriter is requested to deserialize a KafkaSourceOffset (from an InputStream)

• KafkaSource is requested for the initial partition offsets (of 0^th batch)

• KafkaSourceOffset is requested to getPartitionOffsets