Skip to content

StreamingSymmetricHashJoinHelper Utility

StreamingSymmetricHashJoinHelper is a Scala object with the following utility methods:

=== [[getStateWatermarkPredicates]] Creating JoinStateWatermarkPredicates -- getStateWatermarkPredicates Object Method

[source, scala]

getStateWatermarkPredicates( leftAttributes: Seq[Attribute], rightAttributes: Seq[Attribute], leftKeys: Seq[Expression], rightKeys: Seq[Expression], condition: Option[Expression], eventTimeWatermark: Option[Long]): JoinStateWatermarkPredicates

[[getStateWatermarkPredicates-joinKeyOrdinalForWatermark]] getStateWatermarkPredicates tries to find the index of the watermark attribute among the left keys first, and if not found, the right keys.

getStateWatermarkPredicates <> for the left side of a join (for the given leftAttributes, the leftKeys and the rightAttributes).

getStateWatermarkPredicates <> for the right side of a join (for the given rightAttributes, the rightKeys and the leftAttributes).

In the end, getStateWatermarkPredicates creates a JoinStateWatermarkPredicates with the left- and right-side state watermark predicates.

NOTE: getStateWatermarkPredicates is used exclusively when IncrementalExecution is requested to apply the state preparation rule for batch-specific configuration (while optimizing query plans with StreamingSymmetricHashJoinExec physical operators).

==== [[getOneSideStateWatermarkPredicate]] Join State Watermark Predicate (for One Side of Join) -- getOneSideStateWatermarkPredicate Internal Method

[source, scala]

getOneSideStateWatermarkPredicate( oneSideInputAttributes: Seq[Attribute], oneSideJoinKeys: Seq[Expression], otherSideInputAttributes: Seq[Attribute]): Option[JoinStateWatermarkPredicate]

getOneSideStateWatermarkPredicate finds what attributes were used to define the watermark attribute (the oneSideInputAttributes attributes, the <>) and creates a <> as follows:

  • <> if the watermark was defined on a join key (with the watermark expression for the index of the join key expression)

  • <> if the watermark was defined among the oneSideInputAttributes (with the state value watermark based on the given oneSideInputAttributes and otherSideInputAttributes)

NOTE: getOneSideStateWatermarkPredicate creates no <> (None) for no watermark found.

NOTE: getStateWatermarkPredicates is used exclusively to <>.