Schema — Structure of Data¶
A schema is the description of the structure of your data (which together create a Dataset in Spark SQL). It can be implicit (and <
A schema is described using StructType which is a collection of spark-sql-StructField.md[StructField] objects (that in turn are tuples of names, types, and nullability
classifier).
StructType
and StructField
belong to the org.apache.spark.sql.types
package.
[source, scala]¶
import org.apache.spark.sql.types.StructType val schemaUntyped = new StructType() .add("a", "int") .add("b", "string")
// alternatively using Schema DSL val schemaUntyped_2 = new StructType() .add("a".int) .add("b".string)
You can use the canonical string representation of SQL types to describe the types in a schema (that is inherently untyped at compile type) or use type-safe types from the org.apache.spark.sql.types
package.
[source, scala]¶
// it is equivalent to the above expressions import org.apache.spark.sql.types.{IntegerType, StringType} val schemaTyped = new StructType() .add("a", IntegerType) .add("b", StringType)
TIP: Read up on spark-sql-CatalystSqlParser.md[CatalystSqlParser] that is responsible for parsing data types.
It is however recommended to use the singleton DataTypes class with static methods to create schema types.
import org.apache.spark.sql.types.DataTypes._
val schemaWithMap = StructType(
StructField("map", createMapType(LongType, StringType), false) :: Nil)
StructType offers printTreeString that makes presenting the schema more user-friendly.
scala> schemaTyped.printTreeString
root
|-- a: integer (nullable = true)
|-- b: string (nullable = true)
scala> schemaWithMap.printTreeString
root
|-- map: map (nullable = false)
| |-- key: long
| |-- value: string (valueContainsNull = true)
// You can use prettyJson method on any DataType
scala> println(schema1.prettyJson)
{
"type" : "struct",
"fields" : [ {
"name" : "a",
"type" : "integer",
"nullable" : true,
"metadata" : { }
}, {
"name" : "b",
"type" : "string",
"nullable" : true,
"metadata" : { }
} ]
}
As of Spark 2.0, you can describe the schema of your strongly-typed datasets using encoders.
[source, scala]¶
import org.apache.spark.sql.Encoders
scala> Encoders.INT.schema.printTreeString root |-- value: integer (nullable = true)
scala> Encoders.product[(String, java.sql.Timestamp)].schema.printTreeString root |-- _1: string (nullable = true) |-- _2: timestamp (nullable = true)
case class Person(id: Long, name: String) scala> Encoders.product[Person].schema.printTreeString root |-- id: long (nullable = false) |-- name: string (nullable = true)
=== [[implicit-schema]] Implicit Schema
[source, scala]¶
val df = Seq((0, s"""hello\tworld"""), (1, "two spaces inside")).toDF("label", "sentence")
scala> df.printSchema root |-- label: integer (nullable = false) |-- sentence: string (nullable = true)
scala> df.schema res0: org.apache.spark.sql.types.StructType = StructType(StructField(label,IntegerType,false), StructField(sentence,StringType,true))
scala> df.schema("label").dataType res1: org.apache.spark.sql.types.DataType = IntegerType