StreamProgress — Collection of Offsets per Streaming Source¶

StreamProgress is a collection of Offsets per streaming source.

StreamProgress is <> when:

• StreamExecution is created (and creates committed and available offsets)

• OffsetSeq is requested to convert to StreamProgress

StreamProgress is an extension of Scala's scala.collection.immutable.Map with streaming sources as keys and their Offsets as values.

Creating Instance¶

StreamProgress takes the following to be created:

• [[baseMap]] Offsets per streaming source (Map[BaseStreamingSource, Offset]) (default: empty)

=== [[get]] Looking Up Offset by Streaming Source -- get Method

[source, scala]¶

get(key: BaseStreamingSource): Option[Offset]¶

NOTE: get is part of the Scala's scala.collection.MapLike to...FIXME.

get simply looks up an Offsets for the given streaming source in the <>.

=== [[plusplus]] ++ Method

[source, scala]¶

++( updates: GenTraversableOnce[(BaseStreamingSource, Offset)]): StreamProgress

++ simply creates a new <> with the <> and the given updates.

++ is used exclusively when OffsetSeq is requested to convert to StreamProgress.

=== [[toOffsetSeq]] Converting to OffsetSeq -- toOffsetSeq Method

[source, scala]¶

toOffsetSeq( sources: Seq[BaseStreamingSource], metadata: OffsetSeqMetadata): OffsetSeq

toOffsetSeq creates a OffsetSeq with offsets that are <> for every streaming source.

toOffsetSeq is used when:

• MicroBatchExecution stream execution engine is requested to construct the next streaming micro-batch (to commit available offsets for a batch to the write-ahead log)

• StreamExecution is requested to run stream processing (that failed with a Throwable)