EventTimeStatsAccum Accumulator¶
EventTimeStatsAccum is an AccumulatorV2 (Spark Core) that accumulates Long values and produces an EventTimeStats.
AccumulatorV2[Long, EventTimeStats]
Creating Instance¶
EventTimeStatsAccum takes the following to be created:
- EventTimeStats (default: EventTimeStats.zero)
EventTimeStatsAccum is created when:
- EventTimeWatermarkExec unary physical operator is created (and initializes eventTimeStats)
EventTimeWatermarkExec Physical Operator
When EventTimeWatermarkExec physical operator is requested to execute, every task simply adds the values of the event-time watermark column expression to the EventTimeStatsAccum accumulator.
As per design of Spark accumulators in Apache Spark, accumulator updates are automatically sent out (propagated) from tasks to the driver every heartbeat and then they are accumulated together.
EventTimeStats¶
EventTimeStatsAccum is given an EventTimeStats when created.
Every time AccumulatorV2 methods are called, EventTimeStatsAccum simply relays them to the EventTimeStats (that is responsible for event-time statistics, i.e. max, min, avg, count).
Adding Value¶
add(
v: Long): Unit
add is part of the AccumulatorV2 (Spark Core) abstraction.
add simply requests the EventTimeStats to add the given v value.
add is used when EventTimeWatermarkExec physical operator is executed.