KafkaSourceInitialOffsetWriter¶

KafkaSourceInitialOffsetWriter is a Hadoop DFS-based metadata storage for KafkaSourceOffsets.

KafkaSourceInitialOffsetWriter is <> exclusively when KafkaMicroBatchReader is requested to getOrCreateInitialPartitionOffsets.

[[VERSION]] KafkaSourceInitialOffsetWriter uses 1 for the version.

Creating Instance¶

KafkaSourceInitialOffsetWriter takes the following to be created:

• [[sparkSession]] SparkSession
• [[metadataPath]] Path of the metadata log directory

=== [[deserialize]] Deserializing Metadata (Reading Metadata from Serialized Format) -- deserialize Method

[source, scala]¶

deserialize( in: InputStream): KafkaSourceOffset

deserialize...FIXME

deserialize is part of the HDFSMetadataLog abstraction.