ParserInterface — SQL Parsers¶

ParserInterface is the abstraction of SQL parsers that can convert (parse) textual representation of SQL statements (SQL text) into Spark SQL's relational entities (e.g. Catalyst expressions, logical operators, table and function identifiers, table schema, and data types).

Accessing ParserInterface¶

ParserInterface is available as SessionState.sqlParser.

scala> :type spark
org.apache.spark.sql.SparkSession

scala> :type spark.sessionState.sqlParser
org.apache.spark.sql.catalyst.parser.ParserInterface

Contract¶

parseDataType¶

parseDataType(
  sqlText: String): DataType

Parses a SQL text to a DataType

Used when:

• DataType utility is requested to convert a DDL into a DataType (DataType.fromDDL)

• StructType is requested to add a field

• Column.cast

• HiveClientImpl utility is requested to getSparkSQLDataType

• OrcFileOperator is requested to readSchema

• PythonSQLUtils is requested to parseDataType

• SQLUtils is requested to createStructField

• OrcUtils is requested to readSchema

parseExpression¶

parseExpression(
  sqlText: String): Expression

Parses a SQL text to an Expression

Used in the following:

• Dataset operators: <>, <> and <>

• <> standard function

parseFunctionIdentifier¶

parseFunctionIdentifier(
  sqlText: String): FunctionIdentifier

Parses a SQL text to a FunctionIdentifier

Used when:

• SessionCatalog is requested to listFunctions

• CatalogImpl is requested to getFunction and functionExists

parseMultipartIdentifier¶

parseMultipartIdentifier(
  sqlText: String): Seq[String]

Parses a SQL text to a multi-part identifier

Used when:

• CatalogV2Implicits utility is requested to parseColumnPath

• LogicalExpressions utility is requested to parseReference

• DataFrameWriter is requested to insertInto and saveAsTable

• DataFrameWriterV2 is created (and requested for tableName)

• SparkSession is requested to table

parsePlan¶

parsePlan(
  sqlText: String): LogicalPlan

Parses a SQL text to a LogicalPlan

Used when:

• SessionCatalog is requested to look up a relation (table or view) in catalogs

• SparkSession is requested to <>

parseRawDataType¶

parseRawDataType(
  sqlText: String): DataType

Used when...FIXME

parseTableIdentifier¶

parseTableIdentifier(
  sqlText: String): TableIdentifier

Parses a SQL text to a TableIdentifier

Used when:

• DataFrameWriter is requested to insertInto and saveAsTable

• Dataset is requested to createTempViewCommand

• SparkSession is requested to table

• CatalogImpl is requested to listColumns, getTable, tableExists, createTable, recoverPartitions, uncacheTable, and refreshTable

• SessionState is requested to <>

parseTableSchema¶

parseTableSchema(
  sqlText: String): StructType

Parses a SQL text to a StructType

Used when:

• DataType utility is requested to convert a DDL into a DataType (DataType.fromDDL)

• StructType utility is requested to create a StructType for a given DDL-formatted string (StructType.fromDDL)

• JdbcUtils utility is requested to parseUserSpecifiedCreateTableColumnTypes and getCustomSchema

Extensions¶

AbstractSqlParser is the base extension of the ParserInterface abstraction.