Skip to content

EventTimeStatsAccum Accumulator

EventTimeStatsAccum is an AccumulatorV2 (Spark Core) that accumulates Long values and produces an EventTimeStats.

Creating Instance

EventTimeStatsAccum takes the following to be created:

EventTimeStatsAccum is created when EventTimeWatermarkExec unary physical operator is created (and initializes eventTimeStats).

Note

When EventTimeWatermarkExec physical operator is requested to execute, every task simply adds the values of the event-time watermark column to the EventTimeStatsAccum accumulator.

As per design of Spark accumulators in Apache Spark, accumulator updates are automatically sent out (propagated) from tasks to the driver every heartbeat and then they are accumulated together.

Adding Value

add(
  v: Long): Unit

add is part of the AccumulatorV2 (Spark Core) abstraction.

add simply requests the EventTimeStats to add the given v value.

add is used when EventTimeWatermarkExec physical operator is executed.