Skip to content

FileScan

FileScan is an extension of the Scan abstraction for scans in Batch queries.

FileScan is with SupportsReportStatistics.

Contract

dataFilters

dataFilters: Seq[Expression]

Used when...FIXME

fileIndex

fileIndex: PartitioningAwareFileIndex

Used when...FIXME

getFileUnSplittableReason

getFileUnSplittableReason(
    path: Path): String

Used when...FIXME

isSplitable

isSplitable(
    path: Path): Boolean

Used when...FIXME

partitionFilters

partitionFilters: Seq[Expression]

Used when...FIXME

readDataSchema

readDataSchema: StructType

Used when...FIXME

readPartitionSchema

readPartitionSchema: StructType

Used when...FIXME

seqToString

seqToString(
    seq: Seq[Any]): String

Used when...FIXME

sparkSession

sparkSession: SparkSession

Used when...FIXME

withFilters

withFilters(
    partitionFilters: Seq[Expression],
    dataFilters: Seq[Expression]): FileScan

Used when...FIXME

Implementations

  • AvroScan
  • OrcScan
  • ParquetScan
  • TextBasedFileScan

description

description(): String

description...FIXME

description is part of the Scan abstraction.

partitions

partitions: Seq[FilePartition]

partitions...FIXME

partitions is used when FileScan is requested to planInputPartitions.

planInputPartitions

planInputPartitions(): Array[InputPartition]

planInputPartitions...FIXME

planInputPartitions is part of the Batch abstraction.

estimateStatistics

estimateStatistics(): Statistics

estimateStatistics...FIXME

estimateStatistics is part of the SupportsReportStatistics abstraction.

toBatch

toBatch: Batch

toBatch is enabled (true) by default.

toBatch is part of the Scan abstraction.

readSchema

readSchema(): StructType

readSchema...FIXME

readSchema is part of the Scan abstraction.

Back to top