ProjectExec Unary Physical Operator¶
ProjectExec
is a unary physical operator that...FIXME
ProjectExec
supports Java code generation (aka codegen).
ProjectExec
is <
-
InMemoryScans and HiveTableScans execution planning strategies are executed (and request
SparkPlanner
to pruneFilterProject) -
BasicOperators execution planning strategy is executed
-
DataSourceStrategy
execution planning strategy is requested to creates a RowDataSourceScanExec -
FileSourceStrategy
execution planning strategy is requested to plan a LogicalRelation with a HadoopFsRelation -
ExtractPythonUDFs physical optimization is executed
Note
The following is the order of applying the above execution planning strategies to logical query plans when SparkPlanner
or hive/HiveSessionStateBuilder.md#planner[Hive-specific SparkPlanner] are requested to catalyst/QueryPlanner.md#plan[plan a logical query plan into one or more physical query plans]:
=== [[doExecute]] Executing Physical Operator (Generating RDD[InternalRow]) -- doExecute
Method
[source, scala]¶
doExecute(): RDD[InternalRow]¶
doExecute
is part of the SparkPlan abstraction.
doExecute
requests the input <RDD.mapPartitionsWithIndexInternal
).
.RDD.mapPartitionsWithIndexInternal [source, scala]
mapPartitionsWithIndexInternalU
==== [[doExecute-mapPartitionsWithIndexInternal]] Inside doExecute
(RDD.mapPartitionsWithIndexInternal
)
Inside the function (that is part of RDD.mapPartitionsWithIndexInternal
), doExecute
creates an UnsafeProjection with the following:
. <
. catalyst/QueryPlan.md#output[Output] of the <
. SparkPlan.md#subexpressionEliminationEnabled[subexpressionEliminationEnabled] flag
doExecute
requests the UnsafeProjection
to Projection.md#initialize[initialize] and maps over the internal rows (of a partition) using the projection.
=== [[creating-instance]] Creating ProjectExec Instance
ProjectExec
takes the following when created:
- [[projectList]] expressions/NamedExpression.md[NamedExpressions] for the projection
- [[child]] Child SparkPlan.md[physical operator]
=== [[doConsume]] Generating Java Source Code for Consume Path in Whole-Stage Code Generation -- doConsume
Method
[source, scala]¶
doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String¶
doConsume
...FIXME
doConsume
is part of the CodegenSupport abstraction.