ShuffleQueryStageExec Leaf Physical Operator¶

ShuffleQueryStageExec is a QueryStageExec with ShuffleExchangeExec or ReusedExchangeExec child operators.

Creating Instance¶

ShuffleQueryStageExec takes the following to be created:

• ID
• Physical Operator (ShuffleExchangeExec or ReusedExchangeExec)

ShuffleQueryStageExec is created when:

• AdaptiveSparkPlanExec physical operator is requested to newQueryStage (for a ShuffleExchangeExec)

• ShuffleQueryStageExec physical operator is requested to newReuseInstance

newReuseInstance¶

newReuseInstance(
  newStageId: Int,
  newOutput: Seq[Attribute]): QueryStageExec

newReuseInstance is...FIXME

newReuseInstance is part of the QueryStageExec abstraction.

MapOutputStatistics¶

mapStats: Option[MapOutputStatistics]

mapStats takes a MapOutputStatistics from the resultOption.

mapStats throws an AssertionError when the resultOption is not available:

assertion failed: ShuffleQueryStageExec should already be ready

mapStats is used when:

• DemoteBroadcastHashJoin logical optimization is executed
• CoalesceShufflePartitions and OptimizeSkewedJoin adaptive physical optimizations are executed