QueryPlanningTracker¶

QueryPlanningTracker is used to track structured query execution phases:

• parsing (when SparkSession is requested to execute a SQL query)
• analysis (when QueryExecution is requested for an analyzed query plan)
• optimization (when QueryExecution is requested for an optimized query plan)
• planning (when QueryExecution is requested for an physical and executed query plans)

Accessing QueryPlanningTracker¶

QueryPlanningTracker of a structured query is available using QueryExecution.

val df_ops = spark.range(1000).selectExpr("count(*)")
val tracker = df_ops.queryExecution.tracker

// There are three execution phases tracked for structured queries using Dataset API
assert(tracker.phases.keySet == Set("analysis", "optimization", "planning"))

val df_sql = sql("SELECT * FROM range(1000)")
val tracker = df_sql.queryExecution.tracker

// There are four execution phases tracked for structured queries using SQL
assert(tracker.phases.keySet == Set("parsing", "analysis", "optimization", "planning"))

Creating Instance¶

QueryPlanningTracker takes no arguments to be created.

QueryPlanningTracker is created when:

• SparkSession is requested to execute a SQL query

• QueryExecution is created

Getting QueryPlanningTracker¶

get: Option[QueryPlanningTracker]

get utility allows to access the QueryPlanningTracker bound to the current thread (using a thread local variable facility).

import org.apache.spark.sql.catalyst.QueryPlanningTracker

scala> :type QueryPlanningTracker.get
Option[org.apache.spark.sql.catalyst.QueryPlanningTracker]

get is used when RuleExecutor is requested to execute rules on a query plan

Measuring Execution Phase¶

measurePhase[T](
  phase: String)(f: => T): T

measurePhase...FIXME

measurePhase is used when:

• SparkSession is requested to execute a SQL query

• QueryExecution is requested to executePhase

Execution Phases Summaries¶

phases: Map[String, PhaseSummary]

phases gives phasesMap.

phasesMap Internal Registry¶

phasesMap: HashMap[String, PhaseSummary]

phasesMap is used as a registry of execution phase summaries when QueryPlanningTracker is requested to measure phase.

phasesMap is available using phases method.