Columnar Processing¶
New in 3.0.0
Columnar Processing (Columnar Execution) introduces the following main abstractions:
- ColumnarRule
- ApplyColumnarRulesAndInsertTransitions physical optimization
Whole-Stage Java Code Generation
Columnar Processing is similar and a kind of "opposite" at the same time to Whole-Stage Java Code Generation (which is row-based). It is assumed that if a plan supports columnar execution, it can't support whole-stage-codegen at the same time (see the comment in the source code).
Columnar Processing was introduced to Apache Spark 3.0.0 as SPARK-27396.