Skip to content

SimpleTableProvider

SimpleTableProvider is an extension of the TableProvider abstraction for table providers that do not support custom table schema and partitioning.

Contract

Creating Table

getTable(
  options: CaseInsensitiveStringMap): Table

Creates a Table

Used for getOrLoadTable

Implementations

  • ConsoleSinkProvider (Spark Structured Streaming)
  • KafkaSourceProvider
  • MemoryStreamTableProvider (Spark Structured Streaming)
  • NoopDataSource
  • RateStreamProvider (Spark Structured Streaming)
  • TextSocketSourceProvider (Spark Structured Streaming)

Inferring Schema

inferSchema(
  options: CaseInsensitiveStringMap): StructType

inferSchema creates a table and requests it for the schema.

inferSchema is part of the TableProvider abstraction.

Creating Table

getTable(
  schema: StructType,
  partitioning: Array[Transform],
  properties: util.Map[String, String]): Table

getTable creates a table (with the given properties).

getTable asserts that there are no Transforms in given partitioning.

getTable is part of the TableProvider abstraction.

Looking Up Table Once

getOrLoadTable(
  options: CaseInsensitiveStringMap): Table

getOrLoadTable creates a table and caches it for later use (in an internal variable).

Back to top