Review SPARK-28939 to learn when and why
SQLExecutionRDD would be used outside a tracked SQL operation (and with no
SQLExecutionRDD takes the following to be created:
While being created,
SQLExecutionRDD initializes a sqlConfigs internal registry.
SQLExecutionRDD is created when:
QueryExecutionis requested to toRdd
SQLExecutionRDD is given an
RDD[InternalRow] when created.
SQLExecutionRDD requests the given SQLConf for all the configuration properties that have been set when created.
sqlConfigs is a Scala lazy value to guarantee that the code to initialize it is executed once only (when accessed for the first time) and the computed value never changes afterwards.
Learn more in the Scala Language Specification.
compute( split: Partition, context: TaskContext): Iterator[InternalRow]
If not defined (
compute sets the sqlConfigs as thread-local properties to requests the sqlRDD for
iterator (execute the
sqlRDD). Otherwise, if in the context of a tracked SQL operation (and the
spark.sql.execution.id is defined),
compute simply requests the parent sqlRDD for
compute is part of the
RDD (Apache Spark) abstraction.