Skip to content

OptimizeSkewInRebalancePartitions Adaptive Physical Optimization

OptimizeSkewInRebalancePartitions is a physical optimization in Adaptive Query Execution.

OptimizeSkewInRebalancePartitions can be turned on and off using spark.sql.adaptive.optimizeSkewsInRebalancePartitions.enabled configuration property.

Executing Rule

apply(
  plan: SparkPlan): SparkPlan

apply works with ShuffleQueryStageExec leaf physical operators only (with the ShuffleExchangeLikes that are supported).

apply tryOptimizeSkewedPartitions of the ShuffleQueryStageExec.

apply is part of the Rule abstraction.

tryOptimizeSkewedPartitions

tryOptimizeSkewedPartitions(
  shuffle: ShuffleQueryStageExec): SparkPlan

tryOptimizeSkewedPartitions...FIXME

Back to top