Skip to content

KafkaSourceInitialOffsetWriter

KafkaSourceInitialOffsetWriter is a Hadoop DFS-based metadata storage for KafkaSourceOffsets.

KafkaSourceInitialOffsetWriter is <> exclusively when KafkaMicroBatchReader is requested to getOrCreateInitialPartitionOffsets.

[[VERSION]] KafkaSourceInitialOffsetWriter uses 1 for the version.

Creating Instance

KafkaSourceInitialOffsetWriter takes the following to be created:

  • [[sparkSession]] SparkSession
  • [[metadataPath]] Path of the metadata log directory

=== [[deserialize]] Deserializing Metadata (Reading Metadata from Serialized Format) -- deserialize Method

[source, scala]

deserialize( in: InputStream): KafkaSourceOffset


deserialize...FIXME

deserialize is part of the HDFSMetadataLog abstraction.