Skip to content

QueryStageExec Leaf Physical Operators

QueryStageExec is an extension of the LeafExecNode abstraction for query stage operators.

Contract

cancel

cancel(): Unit

Cancels the stage materialization if in progress; otherwise does nothing.

Used when AdaptiveSparkPlanExec physical operator is requested to cleanUpAndThrowException

doMaterialize

doMaterialize(): Future[Any]

Used when QueryStageExec is requested to materialize

id

id: Int

Unique ID

Used when CoalesceShufflePartitions adaptive physical optimization is executed

newReuseInstance

newReuseInstance(
  newStageId: Int,
  newOutput: Seq[Attribute]): QueryStageExec

Used when AdaptiveSparkPlanExec physical operator is requested to reuseQueryStage

plan

plan: SparkPlan

"Child" physical operator (but QueryStageExec is a LeafExecNode and has got no children)

Used when...FIXME

Implementations

generateTreeString

generateTreeString(
  depth: Int,
  lastChildren: Seq[Boolean],
  append: String => Unit,
  verbose: Boolean,
  prefix: String = "",
  addSuffix: Boolean = false,
  maxFields: Int,
  printNodeId: Boolean): Unit

generateTreeString...FIXME

generateTreeString is part of the TreeNode abstraction.

resultOption Internal Registry

resultOption: Option[Any] = None

resultOption is the result of materializing the QueryStageExec operator:

resultOption is None by default.

resultOption is set a value when AdaptiveSparkPlanExec physical operator is requested for the final physical query plan

resultOption is used when:

Computing Statistics

computeStats(): Option[Statistics]

computeStats uses the resultOption to access the underlying ReusedExchangeExec or Exchange physical operators.

If available, computeStats creates Statistics with the sizeInBytes as the dataSize performance metric of the physical operator.

Otherwise, computeStats returns no statistics.

computeStats is used when LogicalQueryStage logical operator is requested for the stats.

Materializing Query Stage

materialize(): Future[Any]

materialize prepares the query stage operator for execution followed by doMaterialize.

Note

materialize is a final method and cannot be overriden.

materialize is used when AdaptiveSparkPlanExec physical operator is requested to getFinalPhysicalPlan.


Last update: 2020-09-07