Skip to content

EvalPythonExec Physical Operators

EvalPythonExec is an extension of the UnaryExecNode abstraction for unary physical operators that can evaluate PythonUDFs.

Contract

Evaluating PythonUDFs

evaluate(
  funcs: Seq[ChainedPythonFunctions],
  argOffsets: Array[Array[Int]],
  iter: Iterator[InternalRow],
  schema: StructType,
  context: TaskContext): Iterator[InternalRow]

Evaluates PythonUDFs (and produces internal binary rows)

Used when EvalPythonExec physical operator is executed

Implementations

  • ArrowEvalPythonExec
  • BatchEvalPythonExec

Creating Instance

EvalPythonExec takes the following to be created:

Abstract Class

EvalPythonExec is an abstract class and cannot be created directly. It is created indirectly for the concrete EvalPythonExecs.

Executing Physical Operator

doExecute(): RDD[InternalRow]

doExecute...FIXME

doExecute is part of the SparkPlan abstraction.


Last update: 2021-02-19