Skip to content

DataSourceV2ScanExecBase

DataSourceV2ScanExecBase is an extension of LeafExecNode abstraction for leaf physical operators that track number of output rows when executed (with or without support for columnar reads).

Contract

inputRDD

inputRDD: RDD[InternalRow]

Used when...FIXME

partitions

partitions: Seq[InputPartition]

Used when:

  • BatchScanExec physical operator is requested for an input RDD

  • ContinuousScanExec and MicroBatchScanExec physical operators (from Spark Structured Streaming) are requested for an inputRDD

  • DataSourceV2ScanExecBase physical operator is requested to outputPartitioning or supportsColumnar

readerFactory

readerFactory: PartitionReaderFactory

PartitionReaderFactory for partition readers

Used when:

  • BatchScanExec physical operator is requested for an input RDD

  • ContinuousScanExec and MicroBatchScanExec physical operators (from Spark Structured Streaming) are requested for an inputRDD

  • DataSourceV2ScanExecBase physical operator is requested to outputPartitioning or supportsColumnar

scan

scan: Scan

Used when...FIXME

Implementations

Executing Physical Operator

doExecute(): RDD[InternalRow]

doExecute...FIXME

doExecute is part of the SparkPlan abstraction.

doExecuteColumnar

doExecuteColumnar(): RDD[ColumnarBatch]

doExecuteColumnar...FIXME

doExecuteColumnar is part of the SparkPlan abstraction.

inputRDDs

inputRDDs(): Seq[RDD[InternalRow]]

inputRDDs...FIXME

inputRDDs is used when...FIXME

metrics

metrics: Map[String, SQLMetric]

metrics...FIXME

metrics is part of the SparkPlan abstraction.

outputPartitioning

outputPartitioning: physical.Partitioning

outputPartitioning...FIXME

outputPartitioning is part of the SparkPlan abstraction.

simpleString

simpleString(
    maxFields: Int): String

simpleString...FIXME

simpleString is part of the TreeNode abstraction.

supportsColumnar

supportsColumnar: Boolean

supportsColumnar...FIXME

supportsColumnar is part of the SparkPlan abstraction.


Last update: 2020-09-03