Skip to content

StateStoreAwareZipPartitionsHelper

StateStoreAwareZipPartitionsHelper is a Scala implicit class of a data RDD (of type RDD[T]) to create a StateStoreAwareZipPartitionsRDD for StreamingSymmetricHashJoinExec physical operator.

Note

Implicit Classes are a language feature in Scala for implicit conversions with extension methods for existing types.

Creating StateStoreAwareZipPartitionsRDD

stateStoreAwareZipPartitions[U: ClassTag, V: ClassTag](
  dataRDD2: RDD[U],
  stateInfo: StatefulOperatorStateInfo,
  storeNames: Seq[String],
  storeCoordinator: StateStoreCoordinatorRef)(
  f: (Iterator[T], Iterator[U]) => Iterator[V]): RDD[V]

stateStoreAwareZipPartitions simply creates a new StateStoreAwareZipPartitionsRDD.

stateStoreAwareZipPartitions is used when StreamingSymmetricHashJoinExec physical operator is executed.


Last update: 2020-11-28