EventTimeStatsAccum Accumulator¶
EventTimeStatsAccum
is an AccumulatorV2
(Spark Core) that accumulates Long
values and produces an EventTimeStats.
Creating Instance¶
EventTimeStatsAccum
takes the following to be created:
- EventTimeStats (default: EventTimeStats.zero)
EventTimeStatsAccum
is created when EventTimeWatermarkExec unary physical operator is created (and initializes eventTimeStats).
Note
When EventTimeWatermarkExec
physical operator is requested to execute, every task simply adds the values of the event-time watermark column to the EventTimeStatsAccum accumulator.
As per design of Spark accumulators in Apache Spark, accumulator updates are automatically sent out (propagated) from tasks to the driver every heartbeat and then they are accumulated together.
Adding Value¶
add(
v: Long): Unit
add
is part of the AccumulatorV2
(Spark Core) abstraction.
add
simply requests the EventTimeStats to add the given v
value.
add
is used when EventTimeWatermarkExec physical operator is executed.