Skip to content

Nondeterministic Expression Contract

Nondeterministic is a <> for Catalyst expressions that are non-deterministic and non-foldable.

Nondeterministic expressions require explicit <> (with the current partition index) before <>.

[[contract]] [source, scala]

package org.apache.spark.sql.catalyst.expressions

trait Nondeterministic extends Expression { // only required methods that have no implementation protected def initializeInternal(partitionIndex: Int): Unit protected def evalInternal(input: InternalRow): Any }

.Nondeterministic Contract [cols="1,2",options="header",width="100%"] |=== | Method | Description

| initializeInternal | [[initializeInternal]] Initializing the Nondeterministic expression

Used exclusively when Nondeterministic expression is requested to <>

| evalInternal | [[evalInternal]] Evaluating the Nondeterministic expression

Used exclusively when Nondeterministic expression is requested to <> |===

NOTE: Nondeterministic expressions are the target of PullOutNondeterministic logical plan rule.

[[implementations]] .Nondeterministic Expressions [cols="1,2",options="header",width="100%"] |=== | Expression | Description

| CurrentBatchTimestamp | [[CurrentBatchTimestamp]]

| InputFileBlockLength | [[InputFileBlockLength]]

| InputFileBlockStart | [[InputFileBlockStart]]

| InputFileName | [[InputFileName]]

| expressions/MonotonicallyIncreasingID.md[MonotonicallyIncreasingID] | [[MonotonicallyIncreasingID]]

| NondeterministicExpression | [[NondeterministicExpression]]

| Rand | [[Rand]]

| Randn | [[Randn]]

| RDG | [[RDG]]

| SparkPartitionID | [[SparkPartitionID]] |===

[[internal-registries]] .Nondeterministic's Internal Properties (e.g. Registries, Counters and Flags) [cols="1,2",options="header",width="100%"] |=== | Name | Description

| [[deterministic]] Expression.md#deterministic[deterministic] | Always turned off (i.e. false)

| [[foldable]] Expression.md#foldable[foldable] | Always turned off (i.e. false)

| [[initialized]] initialized | Controls whether a Nondeterministic expression has been <> before <>.

Turned off by default. |===

=== [[initialize]] Initializing Expression -- initialize Method

[source, scala]

initialize(partitionIndex: Int): Unit

Internally, initialize <> itself (with the input partition index) and turns the internal <> flag on.

initialize is used when InterpretedProjection and InterpretedMutableProjection are requested to initialize themselves.

=== [[eval]] Evaluating Expression -- eval Method

[source, scala]

eval(input: InternalRow): Any

eval is part of the Expression abstraction.

eval is just a wrapper of <> that makes sure that <> has already been executed (and so the expression is initialized).

Internally, eval makes sure that the expression was <> and calls <>.

eval reports a IllegalArgumentException exception when the internal <> flag is off, i.e. <> has not yet been executed.

requirement failed: Nondeterministic expression [name] should be initialized before eval.
Back to top