Skip to content

MapPartitions Unary Logical Operator

MapPartitions is a unary logical operator to represent Dataset.mapPartitions operator.

MapPartitions is an ObjectConsumer and an ObjectProducer.

Creating Instance

MapPartitions takes the following to be created:

MapPartitions is created (indirectly using apply utility) when:

Execution Planning

MapPartitions is planned as MapPartitionsExec physical operator when BasicOperators execution planning strategy is executed.

Creating MapPartitions

apply[T : Encoder, U : Encoder](
  func: Iterator[T] => Iterator[U],
  child: LogicalPlan): LogicalPlan

apply creates a deserializer (for the type T of the input objects) and uses it to create a MapPartitions.

In the end, apply creates a serializer (for the type U of the output objects) with the MapPartitions as the child.

apply is used when:

Back to top