DataSourceV2Utils Utility¶
DataSourceV2Utils
is an utility to extractSessionConfigs and getTableFromProvider for batch and streaming reads and writes.
extractSessionConfigs¶
extractSessionConfigs(
source: TableProvider,
conf: SQLConf): Map[String, String]
Note
extractSessionConfigs
supports data sources with SessionConfigSupport only.
extractSessionConfigs
requests the SessionConfigSupport
data source for the custom key prefix for configuration options that is used to find all configuration options with the keys in the format of spark.datasource.[keyPrefix] in the given SQLConf.
extractSessionConfigs
returns the matching keys with the spark.datasource.[keyPrefix] prefix removed (i.e. spark.datasource.keyPrefix.k1
becomes k1
).
extractSessionConfigs
is used when:
DataFrameReader
is requested to load dataDataFrameWriter
is requested to save data- (Spark Structured Streaming)
DataStreamReader
is requested to load data from a streaming data source - (Spark Structured Streaming)
DataStreamWriter
is requested to start a streaming query
getTableFromProvider¶
getTableFromProvider(
provider: TableProvider,
options: CaseInsensitiveStringMap,
userSpecifiedSchema: Option[StructType]): Table
getTableFromProvider
creates a Table for the given TableProvider, options and user-defined schema.
getTableFromProvider
is used when:
DataFrameReader
is requested to load dataDataFrameWriter
is requested to save data- (Spark Structured Streaming)
DataStreamReader
is requested to load data from a streaming data source - (Spark Structured Streaming)
DataStreamWriter
is requested to start a streaming query