WindowExpression Unevaluable Expression¶

WindowExpression is an unevaluable expression that represents a window function (over some WindowSpecDefinition).

WindowExpression is created when:

• WindowSpec is requested to withAggregate (when Column.over operator is used)

• WindowsSubstitution logical evaluation rule is executed (with WithWindowDefinition logical operators with UnresolvedWindowExpression expressions)

• AstBuilder is requested to parse a function call in a SQL statement

WindowExpression can only be with AggregateExpression, AggregateWindowFunction or OffsetWindowFunction expressions which is enforced at analysis.

// Using Catalyst DSL
val wf = 'count.function(star())
val windowSpec = ???

WindowExpression is resolved in ExtractWindowExpressions, ResolveWindowFrame and ResolveWindowOrder logical rules.

import org.apache.spark.sql.catalyst.expressions.WindowExpression
// relation - Dataset as a table to query
val table = spark.emptyDataset[Int]

scala> val windowExpr = table
  .selectExpr("count() OVER (PARTITION BY value) AS count")
  .queryExecution
  .logical      // <1>
  .expressions
  .toList(0)
  .children(0)
  .asInstanceOf[WindowExpression]
windowExpr: org.apache.spark.sql.catalyst.expressions.WindowExpression = 'count() windowspecdefinition('value, UnspecifiedFrame)

scala> windowExpr.sql
res2: String = count() OVER (PARTITION BY `value` UnspecifiedFrame)

[[properties]] .WindowExpression's Properties [width="100%",cols="1,2",options="header"] |=== | Name | Description

| children | Collection of two Expression.md[expressions], i.e. <> and <>, for which WindowExpression was created.

| dataType | DataType of windowFunction

| foldable | Whether or not <> is foldable.

| nullable | Whether or not <> is nullable.

| sql | "[windowFunction].sql OVER [windowSpec].sql"

| toString | "[windowFunction] [windowSpec]" |===

NOTE: WindowExpression is subject to <> and <> logical optimizations.

NOTE: Distinct window functions are not supported which is enforced at <>.

NOTE: An offset window function can only be evaluated in an ordered row-based window frame with a single offset which is enforced at <>.

=== [[catalyst-dsl]][[windowExpr]] Catalyst DSL -- windowExpr Operator

[source, scala]¶

windowExpr( windowFunc: Expression, windowSpec: WindowSpecDefinition): WindowExpression

windowExpr operator in Catalyst DSL creates a <> expression, e.g. for testing or Spark SQL internals exploration.

[source, scala]¶

// FIXME: DEMO¶

Creating Instance¶

WindowExpression takes the following when created:

• [[windowFunction]] Window function expression
• [[windowSpec]] WindowSpecDefinition expression