CoalesceShufflePartitions Physical Optimization¶
CoalesceShufflePartitions is a physical query plan optimization for Adaptive Query Execution.
CoalesceShufflePartitions uses spark.sql.adaptive.coalescePartitions.enabled configuration property.
CoalesceShufflePartitions takes the following to be created:
CoalesceShufflePartitions is created when
AdaptiveSparkPlanExec physical operator is requested for the QueryStage Optimizer Rules.
apply( plan: SparkPlan): SparkPlan
spark.sql.adaptive.coalescePartitions.enabled Configuration Property
apply makes sure that one of the following holds or does nothing (and simply gives the input SparkPlan back unmodified):
- All the leaves in the query plan are QueryStageExec physical operators
- No CustomShuffleReaderExec physical operator in the given query plan
apply makes sure that the following holds or does nothing (and simply gives the input SparkPlan back unmodified):
- All ShuffleQueryStageExec leaf physical operators use ShuffleExchangeExec unary physical operators that have canChangeNumPartitions flag enabled
apply is part of the Rule abstraction.