FileScan¶
FileScan
is an extension of the Scan abstraction for scans in Batch
queries.
FileScan
is with SupportsReportStatistics
.
Contract¶
dataFilters¶
dataFilters: Seq[Expression]
Used when...FIXME
fileIndex¶
fileIndex: PartitioningAwareFileIndex
Used when...FIXME
getFileUnSplittableReason¶
getFileUnSplittableReason(
path: Path): String
Used when...FIXME
isSplitable¶
isSplitable(
path: Path): Boolean
Used when...FIXME
partitionFilters¶
partitionFilters: Seq[Expression]
Used when...FIXME
readDataSchema¶
readDataSchema: StructType
Used when...FIXME
readPartitionSchema¶
readPartitionSchema: StructType
Used when...FIXME
seqToString¶
seqToString(
seq: Seq[Any]): String
Used when...FIXME
sparkSession¶
sparkSession: SparkSession
Used when...FIXME
withFilters¶
withFilters(
partitionFilters: Seq[Expression],
dataFilters: Seq[Expression]): FileScan
Used when...FIXME
Implementations¶
AvroScan
OrcScan
ParquetScan
TextBasedFileScan
description¶
description(): String
description
...FIXME
description
is part of the Scan abstraction.
partitions¶
partitions: Seq[FilePartition]
partitions
...FIXME
partitions
is used when FileScan
is requested to planInputPartitions.
planInputPartitions¶
planInputPartitions(): Array[InputPartition]
planInputPartitions
...FIXME
planInputPartitions
is part of the Batch abstraction.
estimateStatistics¶
estimateStatistics(): Statistics
estimateStatistics
...FIXME
estimateStatistics
is part of the SupportsReportStatistics abstraction.
toBatch¶
toBatch: Batch
toBatch
is enabled (true
) by default.
toBatch
is part of the Scan abstraction.
readSchema¶
readSchema(): StructType
readSchema
...FIXME
readSchema
is part of the Scan abstraction.