Skip to content

QueryPlan — Structured Query Plan

QueryPlan is an extension of the TreeNode abstraction for query plans in Catalyst Framework.

QueryPlan is used to build a tree of relational operators of a structured query. QueryPlan is a tree of (logical or physical) operators that have a tree of expressions.

QueryPlan has an output attributes, expressions and a schema.

QueryPlan has statePrefix that is used when displaying a plan with ! to indicate an invalid plan, and ' to indicate an unresolved plan.

A QueryPlan is invalid if there are missing input attributes and children subnodes are non-empty.

A QueryPlan is unresolved if the column names have not been verified and column types have not been looked up in the Catalog.

Contract

Output Attributes

output: Seq[Attribute]

Output attribute expressions

val q = spark.range(3)

scala> q.queryExecution.analyzed.output
res0: Seq[org.apache.spark.sql.catalyst.expressions.Attribute] = List(id#0L)

scala> q.queryExecution.withCachedData.output
res1: Seq[org.apache.spark.sql.catalyst.expressions.Attribute] = List(id#0L)

scala> q.queryExecution.optimizedPlan.output
res2: Seq[org.apache.spark.sql.catalyst.expressions.Attribute] = List(id#0L)

scala> q.queryExecution.sparkPlan.output
res3: Seq[org.apache.spark.sql.catalyst.expressions.Attribute] = List(id#0L)

scala> q.queryExecution.executedPlan.output
res4: Seq[org.apache.spark.sql.catalyst.expressions.Attribute] = List(id#0L)

Tip

You can build a StructType from output collection of attributes using toStructType method (that is available through the implicit class AttributeSeq).

scala> q.queryExecution.analyzed.output.toStructType
res5: org.apache.spark.sql.types.StructType = StructType(StructField(id,LongType,false))

Implementations

Expressions

expressions: Seq[Expression]

expressions is all of the expressions present in this query plan operator.

references AttributeSet

references: AttributeSet

references is a AttributeSet of all attributes that appear in expressions of this operator.

Transforming Expressions

transformExpressions(
  rule: PartialFunction[Expression, Expression]): this.type

transformExpressions executes transformExpressionsDown with the input rule.

transformExpressions is used when...FIXME

Transforming Expressions (Down The Tree)

transformExpressionsDown(
  rule: PartialFunction[Expression, Expression]): this.type

transformExpressionsDown applies the given rule to each expression in the query operator.

transformExpressionsDown is used when...FIXME

Output Schema Attribute Set

outputSet: AttributeSet

outputSet simply returns an AttributeSet for the output attributes.

outputSet is used when...FIXME

Missing Input Attributes

missingInput: AttributeSet

missingInput are attributes that are referenced in expressions but not provided by this node's children (as inputSet) and are not produced by this node (as producedAttributes).

Output Schema

You can request the schema of a QueryPlan using schema that builds StructType from the output attributes.

// the query
val dataset = spark.range(3)

scala> dataset.queryExecution.analyzed.schema
res6: org.apache.spark.sql.types.StructType = StructType(StructField(id,LongType,false))

Simple (Basic) Description with State Prefix

simpleString: String

simpleString adds a state prefix to the node's simple text description.

simpleString is part of the TreeNode abstraction.

State Prefix

statePrefix: String

Internally, statePrefix gives ! (exclamation mark) when the node is invalid, i.e. missingInput is not empty, and the node is a parent node. Otherwise, statePrefix gives an empty string.

statePrefix is used when QueryPlan is requested for the simple text node description.

Simple (Basic) Description with State Prefix

verboseString: String

verboseString simply returns the simple (basic) description with state prefix.

verboseString is part of the TreeNode abstraction.

innerChildren

innerChildren: Seq[QueryPlan[_]]

innerChildren simply returns the subqueries.

innerChildren is part of the TreeNode abstraction.

subqueries

subqueries: Seq[PlanType]

subqueries...FIXME

subqueries is used when...FIXME

simpleStringWithNodeId

simpleStringWithNodeId(): String

simpleStringWithNodeId is part of the TreeNode abstraction.

simpleStringWithNodeId finds the operatorId tag or defaults to unknown.

simpleStringWithNodeId uses the nodeName to return the following text:

[nodeName] ([operatorId])

append

append[T <: QueryPlan[T]](
  plan: => QueryPlan[T],
  append: String => Unit,
  verbose: Boolean,
  addSuffix: Boolean,
  maxFields: Int = SQLConf.get.maxToStringFields,
  printOperatorId: Boolean = false): Unit

append...FIXME

append is used when:

Detailed Description (with Operator Id)

verboseStringWithOperatorId(): String

verboseStringWithOperatorId returns the following text (with spark.sql.debug.maxToStringFields configuration property for the number of arguments to this node, if there are any, and the formatted node name):

[formattedNodeName]
Arguments: [argumentString]

verboseStringWithOperatorId is used when:

  • QueryExecution is requested for simple description (and ExplainUtils utility is requested to processPlanSkippingSubqueries)

Formatted Node Name

formattedNodeName: String

formattedNodeName...FIXME

formattedNodeName is used when:

transformAllExpressionsWithPruning

transformAllExpressionsWithPruning(
  cond: TreePatternBits => Boolean,
  ruleId: RuleId = UnknownRuleId)(
  rule: PartialFunction[Expression, Expression]): this.type

transformAllExpressionsWithPruning...FIXME

transformAllExpressionsWithPruning is used when:

  • QueryPlan is requested for transformAllExpressions and normalizeExpressions
  • AnalysisHelper is requested to transformAllExpressionsWithPruning
  • PlanSubqueries physical optimization is executed
  • PlanDynamicPruningFilters physical optimization is executed
  • PlanAdaptiveDynamicPruningFilters physical optimization is executed
  • PlanAdaptiveSubqueries physical optimization is executed
  • ReuseAdaptiveSubquery physical optimization is executed
Back to top