MonotonicallyIncreasingID Nondeterministic Leaf Expression¶

MonotonicallyIncreasingID is a non-deterministic leaf expression that is the internal representation of the monotonically_increasing_id standard and SQL functions.

As a Nondeterministic expression, MonotonicallyIncreasingID requires explicit <> (with the current partition index) before <>.

MonotonicallyIncreasingID <>.

import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
val monotonicallyIncreasingID = MonotonicallyIncreasingID()

import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, ExprCode}
val ctx = new CodegenContext
// doGenCode is used when Expression.genCode is executed
val ExprCode(code, _, _) = monotonicallyIncreasingID.genCode(ctx)

// Helper methods
def trim(code: String): String = {
  code.trim.split("\n").map(_.trim).filter(line => line.nonEmpty).mkString("\n")
}
def prettyPrint(code: String) = println(trim(code))
// END: Helper methods

scala> println(trim(code))
final long value_0 = partitionMask + count_0;
count_0++;

[[dataType]] MonotonicallyIncreasingID uses LongType as the <> of the result of evaluating itself.

[[nullable]] MonotonicallyIncreasingID is never nullable.

[[prettyName]] MonotonicallyIncreasingID uses monotonically_increasing_id for the <>.

[[sql]] MonotonicallyIncreasingID uses monotonically_increasing_id() for the <>.

MonotonicallyIncreasingID is <> when <> standard function is used in a structured query.

MonotonicallyIncreasingID is <> as monotonically_increasing_id SQL function.

[[creating-instance]] MonotonicallyIncreasingID takes no input parameters when created.

[[internal-registries]] .MonotonicallyIncreasingID's Internal Properties (e.g. Registries, Counters and Flags) [cols="1,2",options="header",width="100%"] |=== | Name | Description

| count | [[count]] Number of <> calls, i.e. the number of rows for which MonotonicallyIncreasingID was evaluated

Initialized when MonotonicallyIncreasingID is requested to <> and used to <>.

| partitionMask | [[partitionMask]] Current partition index shifted 33 bits left

Initialized when MonotonicallyIncreasingID is requested to <> and used to <>. |===

=== [[doGenCode]] Generating Java Source Code (ExprCode) For Code-Generated Expression Evaluation -- doGenCode Method

[source, scala]¶

doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode¶

NOTE: doGenCode is part of <> to generate a Java source code (ExprCode) for code-generated expression evaluation.

doGenCode requests the CodegenContext to add a mutable state as count name and long Java type.

doGenCode requests the CodegenContext to add an immutable state (unless exists already) as partitionMask name and long Java type.

doGenCode requests the CodegenContext to addPartitionInitializationStatement with [countTerm] = 0L; statement.

doGenCode requests the CodegenContext to addPartitionInitializationStatement with [partitionMaskTerm] = ((long) partitionIndex) << 33; statement.

In the end, doGenCode returns the input ExprCode with the code as follows and isNull property disabled (false):

final [dataType] [value] = [partitionMaskTerm] + [countTerm];
      [countTerm]++;

=== [[initializeInternal]] Initializing Nondeterministic Expression -- initializeInternal Method

[source, scala]¶

initializeInternal(input: InternalRow): Long¶

NOTE: initializeInternal is part of <> to initialize a Nondeterministic expression.

initializeInternal simply sets the <> to 0 and the <> to partitionIndex.toLong << 33.

[source, scala]¶

val partitionIndex = 1 val partitionMask = partitionIndex.toLong << 33 scala> println(partitionMask.toBinaryString) 1000000000000000000000000000000000

=== [[evalInternal]] Evaluating Nondeterministic Expression -- evalInternal Method

[source, scala]¶

evalInternal(input: InternalRow): Long¶

NOTE: evalInternal is part of <> to evaluate the value of a Nondeterministic expression.

evalInternal remembers the current value of the <> and increments it.

In the end, evalInternal returns the sum of the current value of the <> and the remembered value of the <>.