Developing Custom Data Source
Apache Spark 2.4.4 / Spark SQL
@jaceklaskowski
/
StackOverflow
/
GitHub
The "Internals" Books:
Apache Spark
/
Spark SQL
/
Spark Structured Streaming
## Agenda 1. [Introduction](#/introduction) 1. ["Final Product"](#/final-product) 1. [RelationProvider](#/RelationProvider) 1. [DataSourceRegister](#/DataSourceRegister) 1. [BaseRelation](#/BaseRelation) 1. [PrunedFilteredScan](#/PrunedFilteredScan) 1. [RDD, Partitions and compute](#/rdd-partitions-compute) 1. [Demo](#/demo)
## Introduction 1. **Data Source** - a Spark SQL component that is used to load or save datasets from external data storages or systems 1. **Data Source API** - a set of interfaces (contracts) in Spark SQL to implement a custom data source 1. **DataSource** - a class in Spark SQL that is responsible for **data source resolution** 1. Data Source is referenced in a Spark SQL application using **DataFrameReader.format** or **DataFrameWriter.format** 1. Switch to [The Internals of Spark SQL](https://bit.ly/spark-sql-internals) * [DataSource — Pluggable Data Provider Framework](https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-DataSource.html)
## "Final Product" / Load Side (1 of 2) <pre style="margin-left: 0px;"><code style="width: 900px;" class="lang-scala hljs">spark .read .format("XXX") // <-- your custom data source / format .option("header", true) // <-- the data source supports options .load("data.xxx") .select("id", "name") // column pruning .where($"id" > 5) // filter pushdown .show </code></pre>
## "Final Product" / Save Side (2 of 2) <pre style="margin-left: 0px;"><code style="width: 900px;" class="lang-scala hljs">dataframe .write .format("XXX") // <-- your custom data source / format .option("header", true) // <-- the data source supports options .mode("overwrite") // modes .save("data.xxx") </code></pre>
## BaseRelation 1. **BaseRelation** - the contract of relations with a known schema * Schema is the metadata of a data * Schema describes the shape of a data 1. Does not define data itself * Use the data-specific contracts <small><i>(in the following slides)</i></small> <pre><code class="lang-scala hljs"> abstract class BaseRelation { def sqlContext: SQLContext def schema: StructType def sizeInBytes: Long = sqlContext.conf.defaultSizeInBytes def needConversion: Boolean = true def unhandledFilters(filters: Array[Filter]): Array[Filter] = filters } </code></pre> 1. Switch to [The Internals of Spark SQL](https://bit.ly/spark-sql-internals) * [BaseRelation — Collection of Tuples with Schema](https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-BaseRelation.html)
## PrunedScan 1. **PrunedScan** - the contract of relations that support: * **Column Pruning** - eliminating unneeded columns <pre><code class="lang-scala hljs"> trait PrunedScan { def buildScan(requiredColumns: Array[String]): RDD[Row] } </code></pre> 1. Switch to [The Internals of Spark SQL](https://bit.ly/spark-sql-internals) * [PrunedScan Contract — Relations with Column Pruning](https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-PrunedScan.html)
## PrunedFilteredScan 1. **PrunedFilteredScan** - the contract of relations that support: * **Column Pruning** - eliminating unneeded columns * **Filter Pushdown** - filtering using selected predicates <pre><code class="lang-scala hljs"> trait PrunedFilteredScan { def buildScan(requiredColumns: Array[String], filters: Array[Filter]): RDD[Row] } </code></pre> 1. Switch to [The Internals of Spark SQL](https://bit.ly/spark-sql-internals) * [PrunedFilteredScan Contract — Relations with Column Pruning and Filter Pushdown](https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-PrunedFilteredScan.html)
## TableScan 1. **TableScan** - the contract of relations that support: * **Column Pruning** - eliminating unneeded columns <pre><code class="lang-scala hljs"> trait TableScan { def buildScan(): RDD[Row] } </code></pre> 1. Switch to [The Internals of Spark SQL](https://bit.ly/spark-sql-internals) * [TableScan Contract — Relations with Column Pruning](https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-TableScan.html)
## RelationProvider 1. **RelationProvider** - the contract of `BaseRelation` providers with schema inference (discovery) 1. Switch to [The Internals of Spark SQL](https://bit.ly/spark-sql-internals) * [RelationProvider Contract — Relation Providers With Schema Inference](https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-RelationProvider.html)
## DataSourceRegister 1. **DataSourceRegister** is...FIXME 1. **META-INF/services/org.apache.spark.sql.sources.DataSourceRegister** * DataSource uses Java's ServiceLoader to load DataSourceRegisters 1. Switch to [The Internals of Spark SQL](https://bit.ly/spark-sql-internals) * FIXME
## RDD, Partitions and compute 1. **RDD** is...FIXME 1. **Partition** is...FIXME 1. **compute** is...FIXME 1. Switch to [The Internals of Apache Spark](https://bit.ly/apache-spark-internals) * FIXME
## Demo
## Recap 1. [DataSource](#/DataSource) 1. ["Final Product"](#/final-product) 1. [RelationProvider](#/RelationProvider) 1. [DataSourceRegister](#/DataSourceRegister) 1. [BaseRelation](#/BaseRelation) 1. [PrunedFilteredScan](#/PrunedFilteredScan) 1. [RDD, Partitions and compute](#/rdd-partitions-compute) 1. [Demo](#/demo)
# Questions? * Read [The Internals of Apache Spark](https://bit.ly/apache-spark-internals) * Read [The Internals of Spark SQL](https://bit.ly/spark-sql-internals) * Read [The Internals of Spark Structured Streaming](https://bit.ly/spark-structured-streaming) * Follow [@jaceklaskowski](https://twitter.com/jaceklaskowski) on twitter * Upvote [my questions and answers on StackOverflow](http://stackoverflow.com/users/1305344/jacek-laskowski)
