OffsetSeq¶
OffsetSeq
is the metadata managed by Hadoop DFS-based metadata storage.
OffsetSeq
is <
-
OffsetSeqLog
is requested to deserialize metadata (retrieve metadata from a persistent storage) -
StreamProgress
is requested to convert itself to OffsetSeq (most importantly whenMicroBatchExecution
stream execution engine is requested to construct the next streaming micro-batch to commit available offsets for a batch to the write-ahead log) -
ContinuousExecution
stream execution engine is requested to <> and < >
Creating Instance¶
OffsetSeq
takes the following when created:
- [[offsets]] Collection of optional Offsets (with
None
for <>) - [[metadata]] Optional OffsetSeqMetadata (default:
None
)
=== [[toStreamProgress]] Converting to StreamProgress -- toStreamProgress
Method
[source, scala]¶
toStreamProgress( sources: Seq[BaseStreamingSource]): StreamProgress
toStreamProgress
creates a new StreamProgress and adds the streaming sources for which there are new offsets available.
NOTE: <
toStreamProgress
throws an AssertionError
if the number of the input sources
does not match the <
There are [[offsets.size]] sources in the checkpoint offsets and now there are [[sources.size]] sources requested by the query. Cannot continue.
toStreamProgress
is used when:
-
MicroBatchExecution
is requested to <> and < > -
ContinuousExecution
is requested for <>
=== [[toString]] Textual Representation -- toString
Method
[source, scala]¶
toString: String¶
NOTE: toString
is part of the ++https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html#toString--++[java.lang.Object] contract for the string representation of the object.
toString
simply converts the <-
(a dash if an offset is not available for a streaming source at that position).
=== [[fill]] Creating OffsetSeq Instance -- fill
Factory Methods
[source, scala]¶
fill( offsets: Offset*): OffsetSeq // <1> fill( metadata: Option[String], offsets: Offset*): OffsetSeq
<1> Uses no metadata (None
)
fill
simply creates an <
fill
is used when:
-
OffsetSeqLog
is requested to deserialize metadata -
ContinuousExecution
stream execution engine is requested to get start offsets and addOffset