CoalesceShufflePartitions Adaptive Physical Optimization¶
CoalesceShufflePartitions can be turned on and off using spark.sql.adaptive.coalescePartitions.enabled configuration property.
CoalesceShufflePartitions takes the following to be created:
CoalesceShufflePartitions is created when:
AdaptiveSparkPlanExecphysical operator is requested for the QueryStage Optimizer Rules
apply( plan: SparkPlan): SparkPlan
spark.sql.adaptive.coalescePartitions.enabled Configuration Property
apply makes sure that one of the following holds or does nothing (and simply gives the input SparkPlan back unmodified):
- All the leaves in the query plan are QueryStageExec physical operators
- No CustomShuffleReaderExec physical operator in the given query plan
apply makes sure that the following holds or does nothing (and simply gives the input SparkPlan back unmodified):
- All ShuffleQueryStageExec leaf physical operators use ShuffleExchangeExec unary physical operators that have canChangeNumPartitions flag enabled
apply is part of the Rule abstraction.
updateShuffleReads( plan: SparkPlan, specsMap: Map[Int, Seq[ShufflePartitionSpec]]): SparkPlan
updateShuffleReads is used when: