Skip to content

CollectMetrics Logical Operator

CollectMetrics is a unary logical operator that represents Dataset.observe operator (in the logical query plan).

Creating Instance

CollectMetrics takes the following to be created:

CollectMetrics is created when Dataset.observe operator is used.

Execution Planning

CollectMetrics is planned as CollectMetricsExec physical operator.

Demo

// TIP: Use AggregateExpressions
val q = spark
  .range(0, 5, 1, numPartitions = 1)
  .observe(name = "myMetric", expr = count('id))
val plan = q.queryExecution.logical

scala> println(plan.numberedTreeString)
00 'CollectMetrics myMetric, [count('id) AS count(id)#46]
01 +- Range (0, 5, step=1, splits=Some(1))
Back to top