StateStoreAwareZipPartitionsHelper¶
StateStoreAwareZipPartitionsHelper
is a Scala implicit class of a data RDD (of type RDD[T]
) to create a StateStoreAwareZipPartitionsRDD for StreamingSymmetricHashJoinExec physical operator.
Note
Implicit Classes are a language feature in Scala for implicit conversions with extension methods for existing types.
Creating StateStoreAwareZipPartitionsRDD¶
stateStoreAwareZipPartitions[U: ClassTag, V: ClassTag](
dataRDD2: RDD[U],
stateInfo: StatefulOperatorStateInfo,
storeNames: Seq[String],
storeCoordinator: StateStoreCoordinatorRef)(
f: (Iterator[T], Iterator[U]) => Iterator[V]): RDD[V]
stateStoreAwareZipPartitions
simply creates a new StateStoreAwareZipPartitionsRDD.
stateStoreAwareZipPartitions
is used when StreamingSymmetricHashJoinExec physical operator is executed.
Last update: 2020-11-28