CoalesceShufflePartitions Physical Optimization¶
CoalesceShufflePartitions
is a physical query plan optimization for Adaptive Query Execution.
CoalesceShufflePartitions
uses spark.sql.adaptive.coalescePartitions.enabled configuration property.
CoalesceShufflePartitions
is a Catalyst rule for transforming physical plans (Rule[SparkPlan]
).
Creating Instance¶
CoalesceShufflePartitions
takes the following to be created:
CoalesceShufflePartitions
is created when AdaptiveSparkPlanExec
physical operator is requested for the QueryStage Optimizer Rules.
Executing Rule¶
apply(
plan: SparkPlan): SparkPlan
spark.sql.adaptive.coalescePartitions.enabled Configuration Property
With spark.sql.adaptive.coalescePartitions.enabled configuration property disabled (false
), apply
does nothing and simply gives the input SparkPlan back unmodified.
apply
makes sure that one of the following holds or does nothing (and simply gives the input SparkPlan back unmodified):
- All the leaves in the query plan are QueryStageExec physical operators
- No CustomShuffleReaderExec physical operator in the given query plan
apply
collects all ShuffleQueryStageExec leaf physical operators in the input physical query plan.
apply
makes sure that the following holds or does nothing (and simply gives the input SparkPlan back unmodified):
- All ShuffleQueryStageExec leaf physical operators use ShuffleExchangeExec unary physical operators that have canChangeNumPartitions flag enabled
apply
...FIXME
apply
is part of the Rule abstraction.