Skip to content

WindowExpression Unevaluable Expression

WindowExpression is an unevaluable expression that represents a window function (over some WindowSpecDefinition).

WindowExpression is created when:

WindowExpression can only be with AggregateExpression, AggregateWindowFunction or OffsetWindowFunction expressions which is enforced at analysis.

// Using Catalyst DSL
val wf = 'count.function(star())
val windowSpec = ???

WindowExpression is resolved in ExtractWindowExpressions, ResolveWindowFrame and ResolveWindowOrder logical rules.

import org.apache.spark.sql.catalyst.expressions.WindowExpression
// relation - Dataset as a table to query
val table = spark.emptyDataset[Int]

scala> val windowExpr = table
  .selectExpr("count() OVER (PARTITION BY value) AS count")
  .logical      // <1>
windowExpr: org.apache.spark.sql.catalyst.expressions.WindowExpression = 'count() windowspecdefinition('value, UnspecifiedFrame)

scala> windowExpr.sql
res2: String = count() OVER (PARTITION BY `value` UnspecifiedFrame)
<1> Use sqlParser directly as in[WithWindowDefinition Example]

[[properties]] .WindowExpression's Properties [width="100%",cols="1,2",options="header"] |=== | Name | Description

| children | Collection of two[expressions], i.e. <> and <>, for which WindowExpression was created.

| dataType | DataType of windowFunction

| foldable | Whether or not <> is foldable.

| nullable | Whether or not <> is nullable.

| sql | "[windowFunction].sql OVER [windowSpec].sql"

| toString | "[windowFunction] [windowSpec]" |===

NOTE: WindowExpression is subject to <> and <> logical optimizations.

NOTE: Distinct window functions are not supported which is enforced at <>.

NOTE: An offset window function can only be evaluated in an ordered row-based window frame with a single offset which is enforced at <>.

=== [[catalyst-dsl]][[windowExpr]] Catalyst DSL -- windowExpr Operator

[source, scala]

windowExpr( windowFunc: Expression, windowSpec: WindowSpecDefinition): WindowExpression

windowExpr operator in Catalyst DSL creates a <> expression, e.g. for testing or Spark SQL internals exploration.

[source, scala]


Creating Instance

WindowExpression takes the following when created:

Last update: 2021-02-18