Skip to content

Adaptive Query Execution (AQE)

Adaptive Query Execution (aka Adaptive Query Optimisation, Adaptive Optimisation, or AQE in short) is an optimisation of a physical query execution plan in the middle of query execution for alternative execution plans at runtime.

Adaptive Query Execution can only be used for queries with exchanges or sub-queries.

Adaptive Query Execution re-optimizes the query plan based on runtime statistics.

Quoting the description of a talk by the authors of Adaptive Query Execution:

At runtime, the adaptive execution mode can change shuffle join to broadcast join if it finds the size of one table is less than the broadcast threshold. It can also handle skewed input data for join and change the partition number of the next stage to better fit the data scale. In general, adaptive execution decreases the effort involved in tuning SQL query parameters and improves the execution performance by choosing a better execution plan and parallelism at runtime.

spark.sql.adaptive.enabled

Adaptive Query Execution is disabled by default and can be enabled using spark.sql.adaptive.enabled configuration property.

InsertAdaptiveSparkPlan Physical Optimization

Adaptive Query Execution is applied to a physical query plan using the InsertAdaptiveSparkPlan physical optimization.

AdaptiveSparkPlanExec Physical Operator

Adaptive Query Execution is based on AdaptiveSparkPlanExec physical operator (and the adaptive optimizations).

CacheManager

Adaptive Query Execution is disabled for CacheManager to cacheQuery and recacheByCondition

Structured Streaming Not Supported

Adaptive Query Execution is not supported for streaming queries (Spark Structured Streaming).

SparkListenerSQLAdaptiveExecutionUpdates

Adaptive Query Execution notifies Spark listeners about a physical plan change using SparkListenerSQLAdaptiveExecutionUpdate and SparkListenerSQLAdaptiveSQLMetricUpdates events.

Logging

Adaptive Query Execution uses logOnLevel to print out diagnostic messages to the log.

References

Articles

Videos


Last update: 2021-04-28
Back to top