Skip to content

StateStoreAwareZipPartitionsRDD

StateStoreAwareZipPartitionsRDD is a ZippedPartitionsRDD2 with the <> and <> parent RDDs.

StateStoreAwareZipPartitionsRDD is <> exclusively when StreamingSymmetricHashJoinExec physical operator is requested to <> (and requests <> for one).

Creating Instance

StateStoreAwareZipPartitionsRDD takes the following to be created:

=== [[getPreferredLocations]] Placement Preferences of Partition (Preferred Locations) -- getPreferredLocations Method

[source, scala]

getPreferredLocations(partition: Partition): Seq[String]

NOTE: getPreferredLocations is a part of the RDD Contract to specify placement preferences (aka preferred task locations), i.e. where tasks should be executed to be as close to the data as possible.

getPreferredLocations simply requests the <> for the location of every <> (with the <> and the partition ID) and returns unique executor IDs (so that processing a partition happens on the executor with the proper state store for the operator and the partition).


Last update: 2020-11-28