RewriteCorrelatedScalarSubquery Logical Optimization¶

RewriteCorrelatedScalarSubquery is a base logical optimization that <> with the following operators:

. FIXME

RewriteCorrelatedScalarSubquery is part of the Operator Optimization before Inferring Filters fixed-point batch in the standard batches of the Logical Optimizer.

RewriteCorrelatedScalarSubquery is simply a <> for transforming <>, i.e. Rule[LogicalPlan].

[source, scala]¶

import org.apache.spark.sql.catalyst.optimizer.RewriteCorrelatedScalarSubquery

// FIXME // Demo: Filter + Aggregate // Demo: Filter + UnaryNode

val plan = ??? val optimizedPlan = RewriteCorrelatedScalarSubquery(plan)

=== [[evalExpr]] evalExpr Internal Method

[source, scala]¶

evalExpr(expr: Expression, bindings: Map[ExprId, Option[Any]]) : Option[Any]¶

evalExpr...FIXME

NOTE: evalExpr is used exclusively when RewriteCorrelatedScalarSubquery is...FIXME

=== [[evalAggOnZeroTups]] evalAggOnZeroTups Internal Method

[source, scala]¶

evalAggOnZeroTups(expr: Expression) : Option[Any]¶

evalAggOnZeroTups...FIXME

NOTE: evalAggOnZeroTups is used exclusively when RewriteCorrelatedScalarSubquery is...FIXME

=== [[evalSubqueryOnZeroTups]] evalSubqueryOnZeroTups Internal Method

[source, scala]¶

evalSubqueryOnZeroTups(plan: LogicalPlan) : Option[Any]¶

evalSubqueryOnZeroTups...FIXME

NOTE: evalSubqueryOnZeroTups is used exclusively when RewriteCorrelatedScalarSubquery is requsted to <>.

=== [[constructLeftJoins]] constructLeftJoins Internal Method

[source, scala]¶

constructLeftJoins( child: LogicalPlan, subqueries: ArrayBuffer[ScalarSubquery]): LogicalPlan

constructLeftJoins...FIXME

NOTE: constructLeftJoins is used exclusively when RewriteCorrelatedScalarSubquery logical optimization is <> (i.e. applied to <>, <> or <> logical operators with correlated scalar subqueries)

=== [[apply]] Executing Rule -- apply Method

[source, scala]¶

apply(plan: LogicalPlan): LogicalPlan¶

apply transforms the input spark-sql-LogicalPlan.md[logical plan] as follows:

. For Aggregate.md[Aggregate] operators, apply...FIXME

. For Project.md[Project] operators, apply...FIXME

. For Filter.md[Filter] operators, apply...FIXME

apply is part of the Rule abstraction.

=== [[extractCorrelatedScalarSubqueries]] Extracting ScalarSubquery Expressions with Children -- extractCorrelatedScalarSubqueries Internal Method

[source, scala]¶

extractCorrelatedScalarSubqueriesE <: Expression: E

extractCorrelatedScalarSubqueries finds all spark-sql-Expression-ExecSubqueryExpression-ScalarSubquery.md[ScalarSubquery] expressions with at least one spark-sql-Expression-ExecSubqueryExpression-ScalarSubquery.md#children[child] in the input expression and adds them to the input subqueries collection.

extractCorrelatedScalarSubqueries traverses the input expression down (the expression tree) and, every time a ScalarSubquery with at least one child is found, returns the head of the output attributes of the spark-sql-Expression-ExecSubqueryExpression-ScalarSubquery.md#plan[subquery plan].

In the end, extractCorrelatedScalarSubqueries returns the rewritten expression.

NOTE: extractCorrelatedScalarSubqueries uses https://docs.scala-lang.org/overviews/collections/concrete-mutable-collection-classes.html[scala.collection.mutable.ArrayBuffer] and mutates an instance inside (i.e. adds ScalarSubquery expressions) that makes for two output values, i.e. the rewritten expression and the ScalarSubquery expressions.

NOTE: extractCorrelatedScalarSubqueries is used exclusively when RewriteCorrelatedScalarSubquery is <> (i.e. applied to a spark-sql-LogicalPlan.md[logical plan]).