StreamingSymmetricHashJoinHelper Utility¶
StreamingSymmetricHashJoinHelper
is a Scala object with the following utility methods:
=== [[getStateWatermarkPredicates]] Creating JoinStateWatermarkPredicates -- getStateWatermarkPredicates
Object Method
[source, scala]¶
getStateWatermarkPredicates( leftAttributes: Seq[Attribute], rightAttributes: Seq[Attribute], leftKeys: Seq[Expression], rightKeys: Seq[Expression], condition: Option[Expression], eventTimeWatermark: Option[Long]): JoinStateWatermarkPredicates
[[getStateWatermarkPredicates-joinKeyOrdinalForWatermark]] getStateWatermarkPredicates
tries to find the index of the watermark attribute among the left keys first, and if not found, the right keys.
getStateWatermarkPredicates
<leftAttributes
, the leftKeys
and the rightAttributes
).
getStateWatermarkPredicates
<rightAttributes
, the rightKeys
and the leftAttributes
).
In the end, getStateWatermarkPredicates
creates a JoinStateWatermarkPredicates with the left- and right-side state watermark predicates.
NOTE: getStateWatermarkPredicates
is used exclusively when IncrementalExecution
is requested to apply the state preparation rule for batch-specific configuration (while optimizing query plans with StreamingSymmetricHashJoinExec physical operators).
==== [[getOneSideStateWatermarkPredicate]] Join State Watermark Predicate (for One Side of Join) -- getOneSideStateWatermarkPredicate
Internal Method
[source, scala]¶
getOneSideStateWatermarkPredicate( oneSideInputAttributes: Seq[Attribute], oneSideJoinKeys: Seq[Expression], otherSideInputAttributes: Seq[Attribute]): Option[JoinStateWatermarkPredicate]
getOneSideStateWatermarkPredicate
finds what attributes were used to define the watermark attribute (the oneSideInputAttributes
attributes, the <
-
<
> if the watermark was defined on a join key (with the watermark expression for the index of the join key expression) -
<
> if the watermark was defined among the oneSideInputAttributes
(with the state value watermark based on the givenoneSideInputAttributes
andotherSideInputAttributes
)
NOTE: getOneSideStateWatermarkPredicate
creates no <None
) for no watermark found.
NOTE: getStateWatermarkPredicates
is used exclusively to <