DataSourceV2Utils Utility¶

DataSourceV2Utils is an utility to extractSessionConfigs and getTableFromProvider for batch and streaming reads and writes.

extractSessionConfigs¶

extractSessionConfigs(
  source: TableProvider,
  conf: SQLConf): Map[String, String]

extractSessionConfigs requests the SessionConfigSupport data source for the custom key prefix for configuration options that is used to find all configuration options with the keys in the format of spark.datasource.[keyPrefix] in the given SQLConf.

extractSessionConfigs returns the matching keys with the spark.datasource.[keyPrefix] prefix removed (i.e. spark.datasource.keyPrefix.k1 becomes k1).

extractSessionConfigs is used when:

• DataFrameReader is requested to load data
• DataFrameWriter is requested to save data
• (Spark Structured Streaming) DataStreamReader is requested to load data from a streaming data source
• (Spark Structured Streaming) DataStreamWriter is requested to start a streaming query

getTableFromProvider¶

getTableFromProvider(
  provider: TableProvider,
  options: CaseInsensitiveStringMap,
  userSpecifiedSchema: Option[StructType]): Table

getTableFromProvider creates a Table for the given TableProvider, options and user-defined schema.

getTableFromProvider is used when:

• DataFrameReader is requested to load data
• DataFrameWriter is requested to save data
• (Spark Structured Streaming) DataStreamReader is requested to load data from a streaming data source
• (Spark Structured Streaming) DataStreamWriter is requested to start a streaming query