QueryPlanningTracker¶
QueryPlanningTracker
is used to track structured query execution phases:
- parsing (when
SparkSession
is requested to execute a SQL query) - analysis (when
QueryExecution
is requested for an analyzed query plan) - optimization (when
QueryExecution
is requested for an optimized query plan) - planning (when
QueryExecution
is requested for an physical and executed query plans)
Accessing QueryPlanningTracker¶
QueryPlanningTracker
of a structured query is available using QueryExecution.
val df_ops = spark.range(1000).selectExpr("count(*)")
val tracker = df_ops.queryExecution.tracker
// There are three execution phases tracked for structured queries using Dataset API
assert(tracker.phases.keySet == Set("analysis", "optimization", "planning"))
val df_sql = sql("SELECT * FROM range(1000)")
val tracker = df_sql.queryExecution.tracker
// There are four execution phases tracked for structured queries using SQL
assert(tracker.phases.keySet == Set("parsing", "analysis", "optimization", "planning"))
Creating Instance¶
QueryPlanningTracker
takes no arguments to be created.
QueryPlanningTracker
is created when:
-
SparkSession
is requested to execute a SQL query -
QueryExecution
is created
Getting QueryPlanningTracker¶
get: Option[QueryPlanningTracker]
get
utility allows to access the QueryPlanningTracker
bound to the current thread (using a thread local variable facility).
import org.apache.spark.sql.catalyst.QueryPlanningTracker
scala> :type QueryPlanningTracker.get
Option[org.apache.spark.sql.catalyst.QueryPlanningTracker]
get
is used when RuleExecutor
is requested to execute rules on a query plan
Measuring Execution Phase¶
measurePhase[T](
phase: String)(f: => T): T
measurePhase
...FIXME
measurePhase
is used when:
-
SparkSession
is requested to execute a SQL query -
QueryExecution
is requested to executePhase
Execution Phases Summaries¶
phases: Map[String, PhaseSummary]
phases
gives phasesMap.
phasesMap Internal Registry¶
phasesMap: HashMap[String, PhaseSummary]
phasesMap
is used as a registry of execution phase summaries when QueryPlanningTracker
is requested to measure phase.
phasesMap
is available using phases method.