ShuffleQueryStageExec Leaf Physical Operator¶
ShuffleQueryStageExec
is a QueryStageExec with ShuffleExchangeExec or ReusedExchangeExec child operators.
Creating Instance¶
ShuffleQueryStageExec
takes the following to be created:
ShuffleQueryStageExec
is created when:
-
AdaptiveSparkPlanExec physical operator is requested to newQueryStage (for a ShuffleExchangeExec)
-
ShuffleQueryStageExec
physical operator is requested to newReuseInstance
newReuseInstance¶
newReuseInstance(
newStageId: Int,
newOutput: Seq[Attribute]): QueryStageExec
newReuseInstance
is...FIXME
newReuseInstance
is part of the QueryStageExec abstraction.
MapOutputStatistics¶
mapStats: Option[MapOutputStatistics]
mapStats
takes a MapOutputStatistics
from the resultOption.
mapStats
throws an AssertionError
when the resultOption is not available:
assertion failed: ShuffleQueryStageExec should already be ready
mapStats
is used when:
- DemoteBroadcastHashJoin logical optimization is executed
- CoalesceShufflePartitions and OptimizeSkewedJoin adaptive physical optimizations are executed
Last update: 2020-08-23