DataSourceV2Utils Utility¶

DataSourceV2Utils is an utility to extractSessionConfigs and getTableFromProvider for batch and streaming reads and writes.

extractSessionConfigs¶

extractSessionConfigs(
  source: TableProvider,
  conf: SQLConf): Map[String, String]

Note

extractSessionConfigs supports data sources with SessionConfigSupport only.

extractSessionConfigs requests the SessionConfigSupport data source for the custom key prefix for configuration options that is used to find all configuration options with the keys in the format of spark.datasource.[keyPrefix] in the given SQLConf.

extractSessionConfigs returns the matching keys with the spark.datasource.[keyPrefix] prefix removed (i.e. spark.datasource.keyPrefix.k1 becomes k1).

extractSessionConfigs is used when:

DataFrameReader is requested to load data
DataFrameWriter is requested to save data
(Spark Structured Streaming) DataStreamReader is requested to load data from a streaming data source
(Spark Structured Streaming) DataStreamWriter is requested to start a streaming query

getTableFromProvider¶

getTableFromProvider(
  provider: TableProvider,
  options: CaseInsensitiveStringMap,
  userSpecifiedSchema: Option[StructType]): Table

getTableFromProvider creates a Table for the given TableProvider, options and user-defined schema.

getTableFromProvider is used when:

DataFrameReader is requested to load data
DataFrameWriter is requested to save data
(Spark Structured Streaming) DataStreamReader is requested to load data from a streaming data source
(Spark Structured Streaming) DataStreamWriter is requested to start a streaming query