ShuffleQueryStageExec Adaptive Leaf Physical Operator¶
ShuffleQueryStageExec takes the following to be created:
ShuffleQueryStageExec is created when:
ShuffleQueryStageExecphysical operator is requested to newReuseInstance
ShuffleQueryStageExec initializes the
shuffle internal registry when created.
ShuffleQueryStageExec throws an
wrong plan for shuffle stage: [tree]
shuffle is used when:
AQEShuffleReadExecunary physical operator is requested for the shuffleRDD
- CoalesceShufflePartitions physical optimization is executed
- OptimizeShuffleWithLocalRead physical optimization is executed
- OptimizeSkewedJoin physical optimization is executed
- OptimizeSkewInRebalancePartitions physical optimization is executed
ShuffleQueryStageExecleaf physical operator is requested for the shuffle MapOutputStatistics, newReuseInstance and getRuntimeStatistics
Shuffle MapOutputStatistics Future¶
shuffleFuture is a Scala lazy value to guarantee that the code to initialize it is executed once only (when accessed for the first time) and the computed value never changes afterwards.
Learn more in the Scala Language Specification.
shuffleFuture is used when:
doMaterialize returns the Shuffle MapOutputStatistics Future.
doMaterialize is part of the QueryStageExec abstraction.
cancel cancels the Shuffle MapOutputStatistics Future (unless already completed).
cancel is part of the QueryStageExec abstraction.
newReuseInstance( newStageId: Int, newOutput: Seq[Attribute]): QueryStageExec
newReuseInstance is part of the QueryStageExec abstraction.
mapStats assumes that the resultOption (with the
MapOutputStatistics) is already available or throws an
assertion failed: ShuffleQueryStageExec should already be ready
mapStats takes a
MapOutputStatistics from the resultOption.
mapStats is used when:
AQEShuffleReadExecunary physical operator is requested for the partitionDataSizes
- DynamicJoinSelection adaptive optimization is executed (and selectJoinStrategy)
- OptimizeShuffleWithLocalRead adaptive physical optimization is executed (and canUseLocalShuffleRead)
- CoalesceShufflePartitions, OptimizeSkewedJoin and OptimizeSkewInRebalancePartitions adaptive physical optimizations are executed