CoalesceShufflePartitions Physical Optimization¶

CoalesceShufflePartitions is a physical query plan optimization for Adaptive Query Execution.

CoalesceShufflePartitions uses spark.sql.adaptive.coalescePartitions.enabled configuration property.

CoalesceShufflePartitions is a Catalyst rule for transforming physical plans (Rule[SparkPlan]).

Creating Instance¶

CoalesceShufflePartitions takes the following to be created:

• SparkSession

CoalesceShufflePartitions is created when AdaptiveSparkPlanExec physical operator is requested for the QueryStage Optimizer Rules.

Executing Rule¶

apply(
  plan: SparkPlan): SparkPlan

apply makes sure that one of the following holds or does nothing (and simply gives the input SparkPlan back unmodified):

1. All the leaves in the query plan are QueryStageExec physical operators
2. No CustomShuffleReaderExec physical operator in the given query plan

apply collects all ShuffleQueryStageExec leaf physical operators in the input physical query plan.

apply makes sure that the following holds or does nothing (and simply gives the input SparkPlan back unmodified):

1. All ShuffleQueryStageExec leaf physical operators use ShuffleExchangeExec unary physical operators that have canChangeNumPartitions flag enabled

apply...FIXME

apply is part of the Rule abstraction.