Table¶

Table is an abstraction of logical structured data set:

• a directory or files on a file system
• a topic of Apache Kafka
• a table in a catalog
• others

Contract¶

Table Capabilities¶

Set<TableCapability> capabilities()

TableCapabilities of the table

Used when Table is asked whether or not it supports a given capability

Name¶

String name()

Name of the table

Partitioning¶

Transform[] partitioning()

Partitions of the table (as Transforms)

Default: (empty)

Used when:

• ResolveInsertInto logical analysis rule is executed
• DataFrameWriter is requested to insertInto and save
• DescribeTableExec physical operator is executed

Properties¶

Map<String, String> properties()

Table properties

Default: (empty)

Used when:

• DescribeTableExec and ShowTablePropertiesExec physical operators are executed

Schema¶

StructType schema()

StructType of the table

Used when:

• DataSourceV2Relation utility is used to create a DataSourceV2Relation logical operator
• SimpleTableProvider is requested to inferSchema
• DataSourceV2Strategy execution planning strategy is executed
• DescribeTableExec physical operator is executed
• FileDataSourceV2 is requested to inferSchema
• (Spark Structured Streaming) TextSocketTable is requested for a ScanBuilder with a read schema
• (Spark Structured Streaming) DataStreamReader is requested to load data

Implementations¶

• ConsoleTable (Spark Structured Streaming)
• FileTable
• ForeachWriterTable (Spark Structured Streaming)
• KafkaTable
• MemorySink (Spark Structured Streaming)
• MemoryStreamTable (Spark Structured Streaming)
• NoopTable
• RateStreamTable (Spark Structured Streaming)
• Sink (Spark Structured Streaming)
• StagedTable
• SupportsRead
• SupportsWrite
• TextSocketTable (Spark Structured Streaming)
• V1Table