Skip to content

CoalesceShufflePartitions Physical Optimization

CoalesceShufflePartitions is a physical query plan optimization for Adaptive Query Execution.

CoalesceShufflePartitions uses spark.sql.adaptive.coalescePartitions.enabled configuration property.

CoalesceShufflePartitions is a Catalyst rule for transforming physical plans (Rule[SparkPlan]).

Creating Instance

CoalesceShufflePartitions takes the following to be created:

CoalesceShufflePartitions is created when AdaptiveSparkPlanExec physical operator is requested for the QueryStage Optimizer Rules.

Executing Rule

apply(
  plan: SparkPlan): SparkPlan

spark.sql.adaptive.coalescePartitions.enabled Configuration Property

With spark.sql.adaptive.coalescePartitions.enabled configuration property disabled (false), apply does nothing and simply gives the input SparkPlan back unmodified.

apply makes sure that one of the following holds or does nothing (and simply gives the input SparkPlan back unmodified):

  1. All the leaves in the query plan are QueryStageExec physical operators
  2. No CustomShuffleReaderExec physical operator in the given query plan

apply collects all ShuffleQueryStageExec leaf physical operators in the input physical query plan.

apply makes sure that the following holds or does nothing (and simply gives the input SparkPlan back unmodified):

  1. All ShuffleQueryStageExec leaf physical operators use ShuffleExchangeExec unary physical operators that have canChangeNumPartitions flag enabled

apply...FIXME

apply is part of the Rule abstraction.


Last update: 2021-05-17