OffsetSeq is the metadata managed by Hadoop DFS-based metadata storage.
OffsetSeq is <
OffsetSeqLogis requested to deserialize metadata (retrieve metadata from a persistent storage)
StreamProgressis requested to convert itself to OffsetSeq (most importantly when
MicroBatchExecutionstream execution engine is requested to construct the next streaming micro-batch to commit available offsets for a batch to the write-ahead log)
ContinuousExecutionstream execution engine is requested to <
> and < >
OffsetSeq takes the following when created:
- [[offsets]] Collection of optional Offsets (with
- [[metadata]] Optional OffsetSeqMetadata (default:
=== [[toStreamProgress]] Converting to StreamProgress --
toStreamProgress( sources: Seq[BaseStreamingSource]): StreamProgress
toStreamProgress throws an
AssertionError if the number of the input
sources does not match the <
There are [[offsets.size]] sources in the checkpoint offsets and now there are [[sources.size]] sources requested by the query. Cannot continue.
toStreamProgress is used when:
MicroBatchExecutionis requested to <
> and < >
ContinuousExecutionis requested for <
=== [[toString]] Textual Representation --
toString is part of the ++https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html#toString--++[java.lang.Object] contract for the string representation of the object.
toString simply converts the <
- (a dash if an offset is not available for a streaming source at that position).
=== [[fill]] Creating OffsetSeq Instance --
fill Factory Methods
fill( offsets: Offset*): OffsetSeq // <1> fill( metadata: Option[String], offsets: Offset*): OffsetSeq
<1> Uses no metadata (
fill is used when:
OffsetSeqLogis requested to deserialize metadata