ParserInterface — SQL Parsers¶
ParserInterface
is the abstraction of SQL parsers that can convert (parse) textual representation of SQL statements (SQL text) into Spark SQL's relational entities (e.g. Catalyst expressions, logical operators, table and function identifiers, table schema, and data types).
Accessing ParserInterface¶
ParserInterface
is available as SessionState.sqlParser.
scala> :type spark
org.apache.spark.sql.SparkSession
scala> :type spark.sessionState.sqlParser
org.apache.spark.sql.catalyst.parser.ParserInterface
Contract¶
parseDataType¶
parseDataType(
sqlText: String): DataType
Parses a SQL text to a DataType
Used when:
-
DataType
utility is requested to convert a DDL into a DataType (DataType.fromDDL) -
StructType
is requested to add a field -
HiveClientImpl
utility is requested to getSparkSQLDataType -
OrcFileOperator
is requested toreadSchema
-
PythonSQLUtils
is requested toparseDataType
-
SQLUtils
is requested tocreateStructField
-
OrcUtils
is requested toreadSchema
parseExpression¶
parseExpression(
sqlText: String): Expression
Parses a SQL text to an Expression
Used in the following:
-
Dataset operators: <
>, < > and < > -
<
> standard function
parseFunctionIdentifier¶
parseFunctionIdentifier(
sqlText: String): FunctionIdentifier
Parses a SQL text to a FunctionIdentifier
Used when:
-
SessionCatalog
is requested to listFunctions -
CatalogImpl
is requested to getFunction and functionExists
parseMultipartIdentifier¶
parseMultipartIdentifier(
sqlText: String): Seq[String]
Parses a SQL text to a multi-part identifier
Used when:
-
CatalogV2Implicits
utility is requested to parseColumnPath -
LogicalExpressions
utility is requested to parseReference -
DataFrameWriter
is requested to insertInto and saveAsTable -
DataFrameWriterV2 is created (and requested for tableName)
-
SparkSession
is requested to table
parsePlan¶
parsePlan(
sqlText: String): LogicalPlan
Parses a SQL text to a LogicalPlan
Used when:
-
SessionCatalog
is requested to look up a relation (table or view) in catalogs -
SparkSession
is requested to <>
parseRawDataType¶
parseRawDataType(
sqlText: String): DataType
Used when...FIXME
parseTableIdentifier¶
parseTableIdentifier(
sqlText: String): TableIdentifier
Parses a SQL text to a TableIdentifier
Used when:
-
DataFrameWriter
is requested to insertInto and saveAsTable -
Dataset
is requested to createTempViewCommand -
SparkSession
is requested to table -
CatalogImpl
is requested to listColumns, getTable, tableExists, createTable, recoverPartitions, uncacheTable, and refreshTable -
SessionState
is requested to <>
parseTableSchema¶
parseTableSchema(
sqlText: String): StructType
Parses a SQL text to a StructType
Used when:
-
DataType
utility is requested to convert a DDL into a DataType (DataType.fromDDL) -
StructType
utility is requested to create a StructType for a given DDL-formatted string (StructType.fromDDL) -
JdbcUtils
utility is requested to parseUserSpecifiedCreateTableColumnTypes and getCustomSchema
Extensions¶
AbstractSqlParser is the base extension of the ParserInterface
abstraction.