Skip to content

QueryPlan — Structured Query Plan

QueryPlan is an extension of the TreeNode abstraction for query plans in Catalyst Framework.

QueryPlan is used to build a tree of relational operators of a structured query. QueryPlan is a tree of (logical or physical) operators that have a tree of expressions.

QueryPlan has an output attributes, expressions and a schema.

QueryPlan has statePrefix that is used when displaying a plan with ! to indicate an invalid plan, and ' to indicate an unresolved plan.

A QueryPlan is invalid if there are missing input attributes and children subnodes are non-empty.

A QueryPlan is unresolved if the column names have not been verified and column types have not been looked up in the Catalog.

Contract

Output Attributes

output: Seq[Attribute]

Output attribute expressions

val q = spark.range(3)

scala> q.queryExecution.analyzed.output
res0: Seq[org.apache.spark.sql.catalyst.expressions.Attribute] = List(id#0L)

scala> q.queryExecution.withCachedData.output
res1: Seq[org.apache.spark.sql.catalyst.expressions.Attribute] = List(id#0L)

scala> q.queryExecution.optimizedPlan.output
res2: Seq[org.apache.spark.sql.catalyst.expressions.Attribute] = List(id#0L)

scala> q.queryExecution.sparkPlan.output
res3: Seq[org.apache.spark.sql.catalyst.expressions.Attribute] = List(id#0L)

scala> q.queryExecution.executedPlan.output
res4: Seq[org.apache.spark.sql.catalyst.expressions.Attribute] = List(id#0L)

Tip

You can build a StructType from output collection of attributes using toStructType method (that is available through the implicit class AttributeSeq).

scala> q.queryExecution.analyzed.output.toStructType
res5: org.apache.spark.sql.types.StructType = StructType(StructField(id,LongType,false))

Implementations

Expressions

expressions: Seq[Expression]

expressions is all of the expressions present in this query plan operator.

references AttributeSet

references: AttributeSet

references is a AttributeSet of all attributes that appear in expressions of this operator.

Transforming Expressions

transformExpressions(
  rule: PartialFunction[Expression, Expression]): this.type

transformExpressions executes transformExpressionsDown with the input rule.

transformExpressions is used when...FIXME

Transforming Expressions (Down The Tree)

transformExpressionsDown(
  rule: PartialFunction[Expression, Expression]): this.type

transformExpressionsDown applies the given rule to each expression in the query operator.

transformExpressionsDown is used when...FIXME

Output Schema Attribute Set

outputSet: AttributeSet

outputSet simply returns an AttributeSet for the output attributes.

outputSet is used when...FIXME

Missing Input Attributes

missingInput: AttributeSet

missingInput are attributes that are referenced in expressions but not provided by this node's children (as inputSet) and are not produced by this node (as producedAttributes).

Output Schema

You can request the schema of a QueryPlan using schema that builds StructType from the output attributes.

// the query
val dataset = spark.range(3)

scala> dataset.queryExecution.analyzed.schema
res6: org.apache.spark.sql.types.StructType = StructType(StructField(id,LongType,false))

Simple (Basic) Description with State Prefix

simpleString: String

simpleString adds a state prefix to the node's simple text description.

simpleString is part of the TreeNode abstraction.

State Prefix

statePrefix: String

Internally, statePrefix gives ! (exclamation mark) when the node is invalid, i.e. missingInput is not empty, and the node is a parent node. Otherwise, statePrefix gives an empty string.

statePrefix is used when QueryPlan is requested for the simple text node description.

Simple (Basic) Description with State Prefix

verboseString: String

verboseString simply returns the simple (basic) description with state prefix.

verboseString is part of the TreeNode abstraction.

innerChildren

innerChildren: Seq[QueryPlan[_]]

innerChildren simply returns the subqueries.

innerChildren is part of the TreeNode abstraction.

subqueries

subqueries: Seq[PlanType]

subqueries...FIXME

subqueries is used when...FIXME

simpleStringWithNodeId

simpleStringWithNodeId(): String

simpleStringWithNodeId finds the operatorId tag or defaults to unknown.

simpleStringWithNodeId uses the nodeName to return the following text:

[nodeName] ([operatorId])

simpleStringWithNodeId is part of the TreeNode abstraction.

append

append[T <: QueryPlan[T]](
  plan: => QueryPlan[T],
  append: String => Unit,
  verbose: Boolean,
  addSuffix: Boolean,
  maxFields: Int = SQLConf.get.maxToStringFields,
  printOperatorId: Boolean = false): Unit

append...FIXME

append is used when:

verboseStringWithOperatorId

verboseStringWithOperatorId(): String

verboseStringWithOperatorId uses spark.sql.debug.maxToStringFields configuration property for the number of arguments to this node and the formattedNodeName to build a text of the following format:

[formattedNodeName]
Arguments: [argumentString]

verboseStringWithOperatorId is used when QueryExecution is requested for simpleString (and ExplainUtils utility is requested to processPlanSkippingSubqueries).

formattedNodeName

formattedNodeName: String

formattedNodeName...FIXME

formattedNodeName is used when QueryPlan (in general) and ProjectExec, FilterExec, DataSourceScanExec, FileSourceScanExec, LeafExecNode, UnaryExecNode, BinaryExecNode, BaseAggregateExec, ReusedExchangeExec, CartesianProductExec, HashJoin, SortMergeJoinExec physical operators (in particular) are requested for verboseStringWithOperatorId


Last update: 2020-11-07