Skip to content

FileTable

FileTable is an extension of the Table abstraction for file-backed tables with support for read and write.

Contract

fallbackFileFormat

fallbackFileFormat: Class[_ <: FileFormat]

Fallback V1 FileFormat

Used when FallBackFileSourceV2 extended resolution rule is executed (to resolve an InsertIntoStatement with a DataSourceV2Relation with a FileTable)

formatName

formatName: String

Name of the file table (format)

inferSchema

inferSchema(
    files: Seq[FileStatus]): Option[StructType]

Infers schema of the given files (as Hadoop FileStatuses)

Used when FileTable is requested for a data schema

supportsDataType

supportsDataType(
    dataType: DataType): Boolean = true

supportsDataType indicates whether a given DataType is supported in read/write path or not.

Default: All DataTypes are supported by default

Implementations

  • AvroTable
  • CSVTable
  • JsonTable
  • OrcTable
  • ParquetTable
  • TextTable

Creating Instance

FileTable takes the following to be created:

FileTable is an abstract class and cannot be created directly. It is created indirectly for the concrete FileTables.

Table Capabilities

capabilities: java.util.Set[TableCapability]

capabilities are the following TableCapabilities:

capabilities is part of the Table abstraction.

dataSchema

dataSchema: StructType

dataSchema is a schema of the data of the file-backed table

Lazy Value

dataSchema is a Scala lazy value to guarantee that the code to initialize it is executed once only (when accessed for the first time) and cached afterwards.

dataSchema is used when:

fileIndex

fileIndex: PartitioningAwareFileIndex

fileIndex...FIXME

fileIndex is used when...FIXME

partitioning

partitioning: Array[Transform]

partitioning...FIXME

partitioning is part of the Table abstraction.

properties

properties: util.Map[String, String]

properties is simply the options.

properties is part of the Table abstraction.

schema

schema: StructType

schema...FIXME

schema is part of the Table abstraction.


Last update: 2021-05-30