Skip to content

PruneFileSourcePartitions Logical Optimization

PruneFileSourcePartitions is the only logical optimization rule in the Prune File Source Table Partitions batch of the SparkOptimizer.

PruneFileSourcePartitions <> into a[Project] operator with a Filter logical operator over a "pruned" LogicalRelation with the HadoopFsRelation of a Hive partitioned table (with a[PrunedInMemoryFileIndex]).

=== [[apply]] Executing Rule -- apply Method

[source, scala]

apply( plan: LogicalPlan): LogicalPlan

apply[destructures the input logical plan] into a tuple of projection and filter expressions together with a leaf logical operator.

apply transforms[LogicalRelations] with a HadoopFsRelation and a CatalogFileIndex (i.e. for Hive tables) when there are filter expressions defined and the Hive table is partitioned.

apply resolves partition column references (by requesting the logical operator to[resolve partition column attributes to concrete references in the query plan]) and excludes[subquery expressions].

If there are no predicates (filter expressions) left for partition pruning, apply simply does nothing more and returns the input logical query untouched.

With predicates left for partition pruning, apply requests the[CatalogFileIndex] for the[partitions by the predicate expressions] (that gives a[PrunedInMemoryFileIndex] for a partitioned table).

apply replaces the FileIndex in the HadoopFsRelation with the PrunedInMemoryFileIndex and the[total size] statistic to the[PrunedInMemoryFileIndex's].

In the end, apply creates a Filter logical operator (with the "pruned" LogicalRelation as a child operator and all the filter predicate expressions combined together with And expression) and makes it a child operator of a[Project] operator.

apply is part of the Rule abstraction.

Last update: 2020-11-13