JdbcRelationProvider¶
[[shortName]] JdbcRelationProvider
is a DataSourceRegister and registers itself to handle jdbc data source format.
Note
JdbcRelationProvider
uses META-INF/services/org.apache.spark.sql.sources.DataSourceRegister
file for the registration which is available in the source code of Apache Spark.
JdbcRelationProvider
is a RelationProvider and a CreatableRelationProvider.
JdbcRelationProvider
is used when DataFrameReader
is requested to load data from jdbc data source.
val table = spark.read.jdbc(...)
// or in a more verbose way
val table = spark.read.format("jdbc").load(...)
Loading Data from Table Using JDBC¶
createRelation(
sqlContext: SQLContext,
parameters: Map[String, String]): BaseRelation
createRelation
is part of the RelationProvider abstraction.
createRelation
creates a JDBCPartitioningInfo
(using JDBCOptions and the input parameters
that correspond to the Options for JDBC Data Source).
NOTE: createRelation
uses partitionColumn, lowerBound, upperBound and numPartitions.
In the end, createRelation
creates a datasources/jdbc/JDBCRelation.md#creating-instance[JDBCRelation] with datasources/jdbc/JDBCRelation.md#columnPartition[column partitions] (and JDBCOptions).
Writing Rows of Structured Query (DataFrame) to Table Using JDBC¶
createRelation(
sqlContext: SQLContext,
mode: SaveMode,
parameters: Map[String, String],
df: DataFrame): BaseRelation
createRelation
is part of the CreatableRelationProvider abstraction.
Internally, createRelation
creates a JDBCOptions (from the input parameters
).
createRelation
reads caseSensitiveAnalysis (using the input sqlContext
).
createRelation
checks whether the table (given dbtable
and url
options in the input parameters
) exists.
NOTE: createRelation
uses a database-specific JdbcDialect
to check whether a table exists.
createRelation
branches off per whether the table already exists in the database or not.
If the table does not exist, createRelation
creates the table (by executing CREATE TABLE
with createTableColumnTypes and createTableOptions options from the input parameters
) and writes the rows to the database in a single transaction.
If however the table does exist, createRelation
branches off per SaveMode (see the following createRelation and SaveMode).
[[createRelation-CreatableRelationProvider-SaveMode]] .createRelation and SaveMode [cols="1,2",options="header",width="100%"] |=== | Name | Description
| Append | Saves the records to the table.
| ErrorIfExists a| Throws a AnalysisException
with the message:
Table or view '[table]' already exists. SaveMode: ErrorIfExists.
| Ignore | Does nothing.
| Overwrite a| Truncates or drops the table
NOTE: createRelation
truncates the table only when truncate JDBC option is enabled and JdbcDialect.md#isCascadingTruncateTable[isCascadingTruncateTable] is disabled. |===
In the end, createRelation
closes the JDBC connection to the database and creates a JDBCRelation.