StreamingJoinHelper Utility¶
StreamingJoinHelper
is a Scala object with the following utility methods:
- <
>
[[logging]] [TIP] ==== Enable ALL
logging level for org.apache.spark.sql.catalyst.analysis.StreamingJoinHelper
to see what happens inside.
Add the following line to conf/log4j.properties
:
log4j.logger.org.apache.spark.sql.catalyst.analysis.StreamingJoinHelper=ALL
Refer to <>.¶
State Value Watermark¶
getStateValueWatermark(
attributesToFindStateWatermarkFor: AttributeSet,
attributesWithEventWatermark: AttributeSet,
joinCondition: Option[Expression],
eventWatermark: Option[Long]): Option[Long]
getStateValueWatermark
...FIXME
getStateValueWatermark
is used when:
-
UnsupportedOperationChecker
utility is used to checkForStreaming -
StreamingSymmetricHashJoinHelper
utility is used to create a JoinStateWatermarkPredicates
Last update: 2020-11-28