FileScan¶

FileScan is an extension of the Scan abstraction for scans in Batch queries.

FileScan is with SupportsReportStatistics.

Contract¶

dataFilters¶

dataFilters: Seq[Expression]

Used when...FIXME

fileIndex¶

fileIndex: PartitioningAwareFileIndex

Used when...FIXME

getFileUnSplittableReason¶

getFileUnSplittableReason(
    path: Path): String

Used when...FIXME

isSplitable¶

isSplitable(
    path: Path): Boolean

Used when...FIXME

partitionFilters¶

partitionFilters: Seq[Expression]

Used when...FIXME

readDataSchema¶

readDataSchema: StructType

Used when...FIXME

readPartitionSchema¶

readPartitionSchema: StructType

Used when...FIXME

seqToString¶

seqToString(
    seq: Seq[Any]): String

Used when...FIXME

sparkSession¶

sparkSession: SparkSession

Used when...FIXME

withFilters¶

withFilters(
    partitionFilters: Seq[Expression],
    dataFilters: Seq[Expression]): FileScan

Used when...FIXME

Implementations¶

• AvroScan
• OrcScan
• ParquetScan
• TextBasedFileScan

description¶

description(): String

description...FIXME

description is part of the Scan abstraction.

partitions¶

partitions: Seq[FilePartition]

partitions...FIXME

partitions is used when FileScan is requested to planInputPartitions.

planInputPartitions¶

planInputPartitions(): Array[InputPartition]

planInputPartitions...FIXME

planInputPartitions is part of the Batch abstraction.

estimateStatistics¶

estimateStatistics(): Statistics

estimateStatistics...FIXME

estimateStatistics is part of the SupportsReportStatistics abstraction.

toBatch¶

toBatch: Batch

toBatch is enabled (true) by default.

toBatch is part of the Scan abstraction.

readSchema¶

readSchema(): StructType

readSchema...FIXME

readSchema is part of the Scan abstraction.