StreamProgress — Collection of Offsets per Streaming Source¶
StreamProgress
is a collection of Offsets per streaming source.
StreamProgress
is <
-
StreamExecution is created (and creates committed and available offsets)
-
OffsetSeq
is requested to convert to StreamProgress
StreamProgress
is an extension of Scala's scala.collection.immutable.Map with streaming sources as keys and their Offsets as values.
Creating Instance¶
StreamProgress
takes the following to be created:
- [[baseMap]] Offsets per streaming source (
Map[BaseStreamingSource, Offset]
) (default: empty)
=== [[get]] Looking Up Offset by Streaming Source -- get
Method
[source, scala]¶
get(key: BaseStreamingSource): Option[Offset]¶
NOTE: get
is part of the Scala's scala.collection.MapLike
to...FIXME.
get
simply looks up an Offsets for the given streaming source in the <
=== [[plusplus]] ++
Method
[source, scala]¶
++( updates: GenTraversableOnce[(BaseStreamingSource, Offset)]): StreamProgress
++
simply creates a new <
++
is used exclusively when OffsetSeq
is requested to convert to StreamProgress.
=== [[toOffsetSeq]] Converting to OffsetSeq -- toOffsetSeq
Method
[source, scala]¶
toOffsetSeq( sources: Seq[BaseStreamingSource], metadata: OffsetSeqMetadata): OffsetSeq
toOffsetSeq
creates a OffsetSeq with offsets that are <
toOffsetSeq
is used when:
-
MicroBatchExecution
stream execution engine is requested to construct the next streaming micro-batch (to commit available offsets for a batch to the write-ahead log) -
StreamExecution
is requested to run stream processing (that failed with a Throwable)