Skip to content

CleanupDynamicPruningFilters Logical Optimization

CleanupDynamicPruningFilters is a logical optimization for Dynamic Partition Pruning.

CleanupDynamicPruningFilters removes DynamicPruning predicate expressions in Filter logical operators.

CleanupDynamicPruningFilters is a Rule[LogicalPlan] (a rule for logical operators).

CleanupDynamicPruningFilters is part of the Cleanup filters that cannot be pushed down batch of the SparkOptimizer.

Executing Rule

  plan: LogicalPlan): LogicalPlan

apply does nothing when the spark.sql.optimizer.dynamicPartitionPruning.enabled configuration property is disabled (false).

apply transforms the given logical plan as follows:

  • LogicalRelation logical operators with HadoopFsRelation are left unmodified (pass through)

  • DynamicPruning predicate expressions in Filter logical operators are replaced with true literals (cleaned up)

apply is part of the Rule abstraction.

Last update: 2020-11-07