StreamingJoinHelper Utility¶

StreamingJoinHelper is a Scala object with the following utility methods:

• <>

[[logging]] [TIP] ==== Enable ALL logging level for org.apache.spark.sql.catalyst.analysis.StreamingJoinHelper to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.sql.catalyst.analysis.StreamingJoinHelper=ALL

Refer to <>.¶

State Value Watermark¶

getStateValueWatermark(
  attributesToFindStateWatermarkFor: AttributeSet,
  attributesWithEventWatermark: AttributeSet,
  joinCondition: Option[Expression],
  eventWatermark: Option[Long]): Option[Long]

getStateValueWatermark...FIXME

getStateValueWatermark is used when:

• UnsupportedOperationChecker utility is used to checkForStreaming

• StreamingSymmetricHashJoinHelper utility is used to create a JoinStateWatermarkPredicates