ProjectExec Unary Physical Operator¶

ProjectExec is a unary physical operator that...FIXME

ProjectExec supports Java code generation (aka codegen).

ProjectExec is <> when:

• InMemoryScans and HiveTableScans execution planning strategies are executed (and request SparkPlanner to pruneFilterProject)

• BasicOperators execution planning strategy is executed

• DataSourceStrategy execution planning strategy is requested to creates a RowDataSourceScanExec

• FileSourceStrategy execution planning strategy is requested to plan a LogicalRelation with a HadoopFsRelation

• ExtractPythonUDFs physical optimization is executed

=== [[doExecute]] Executing Physical Operator (Generating RDD[InternalRow]) -- doExecute Method

[source, scala]¶

doExecute(): RDD[InternalRow]¶

doExecute is part of the SparkPlan abstraction.

doExecute requests the input <> to SparkPlan.md#execute[produce an RDD of internal rows] and applies a <> (using RDD.mapPartitionsWithIndexInternal).

.RDD.mapPartitionsWithIndexInternal [source, scala]

mapPartitionsWithIndexInternalU

==== [[doExecute-mapPartitionsWithIndexInternal]] Inside doExecute (RDD.mapPartitionsWithIndexInternal)

Inside the function (that is part of RDD.mapPartitionsWithIndexInternal), doExecute creates an UnsafeProjection with the following:

. <>

. catalyst/QueryPlan.md#output[Output] of the <> physical operator as the input schema

. SparkPlan.md#subexpressionEliminationEnabled[subexpressionEliminationEnabled] flag

doExecute requests the UnsafeProjection to Projection.md#initialize[initialize] and maps over the internal rows (of a partition) using the projection.

=== [[creating-instance]] Creating ProjectExec Instance

ProjectExec takes the following when created:

• [[projectList]] expressions/NamedExpression.md[NamedExpressions] for the projection
• [[child]] Child SparkPlan.md[physical operator]

=== [[doConsume]] Generating Java Source Code for Consume Path in Whole-Stage Code Generation -- doConsume Method

[source, scala]¶

doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String¶

doConsume...FIXME

doConsume is part of the CodegenSupport abstraction.