MonotonicallyIncreasingID Nondeterministic Leaf Expression¶
MonotonicallyIncreasingID
is a non-deterministic leaf expression that is the internal representation of the monotonically_increasing_id
standard and SQL functions.
As a Nondeterministic
expression, MonotonicallyIncreasingID
requires explicit <
MonotonicallyIncreasingID
<
import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
val monotonicallyIncreasingID = MonotonicallyIncreasingID()
import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, ExprCode}
val ctx = new CodegenContext
// doGenCode is used when Expression.genCode is executed
val ExprCode(code, _, _) = monotonicallyIncreasingID.genCode(ctx)
// Helper methods
def trim(code: String): String = {
code.trim.split("\n").map(_.trim).filter(line => line.nonEmpty).mkString("\n")
}
def prettyPrint(code: String) = println(trim(code))
// END: Helper methods
scala> println(trim(code))
final long value_0 = partitionMask + count_0;
count_0++;
[[dataType]] MonotonicallyIncreasingID
uses LongType as the <
[[nullable]] MonotonicallyIncreasingID
is never nullable.
[[prettyName]] MonotonicallyIncreasingID
uses monotonically_increasing_id for the <
[[sql]] MonotonicallyIncreasingID
uses monotonically_increasing_id() for the <
MonotonicallyIncreasingID
is <
MonotonicallyIncreasingID
is <monotonically_increasing_id
SQL function.
[[creating-instance]] MonotonicallyIncreasingID
takes no input parameters when created.
[[internal-registries]] .MonotonicallyIncreasingID's Internal Properties (e.g. Registries, Counters and Flags) [cols="1,2",options="header",width="100%"] |=== | Name | Description
| count
| [[count]] Number of <MonotonicallyIncreasingID
was evaluated
Initialized when MonotonicallyIncreasingID
is requested to <
| partitionMask
| [[partitionMask]] Current partition index shifted 33 bits left
Initialized when MonotonicallyIncreasingID
is requested to <
=== [[doGenCode]] Generating Java Source Code (ExprCode) For Code-Generated Expression Evaluation -- doGenCode
Method
[source, scala]¶
doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode¶
NOTE: doGenCode
is part of <
doGenCode
requests the CodegenContext
to add a mutable state as count
name and long
Java type.
doGenCode
requests the CodegenContext
to add an immutable state (unless exists already) as partitionMask
name and long
Java type.
doGenCode
requests the CodegenContext
to addPartitionInitializationStatement with [countTerm] = 0L;
statement.
doGenCode
requests the CodegenContext
to addPartitionInitializationStatement with [partitionMaskTerm] = ((long) partitionIndex) << 33;
statement.
In the end, doGenCode
returns the input ExprCode
with the code
as follows and isNull
property disabled (false
):
final [dataType] [value] = [partitionMaskTerm] + [countTerm];
[countTerm]++;
=== [[initializeInternal]] Initializing Nondeterministic Expression -- initializeInternal
Method
[source, scala]¶
initializeInternal(input: InternalRow): Long¶
NOTE: initializeInternal
is part of <Nondeterministic
expression.
initializeInternal
simply sets the <0
and the <partitionIndex.toLong << 33
.
[source, scala]¶
val partitionIndex = 1 val partitionMask = partitionIndex.toLong << 33 scala> println(partitionMask.toBinaryString) 1000000000000000000000000000000000
=== [[evalInternal]] Evaluating Nondeterministic Expression -- evalInternal
Method
[source, scala]¶
evalInternal(input: InternalRow): Long¶
NOTE: evalInternal
is part of <Nondeterministic
expression.
evalInternal
remembers the current value of the <
In the end, evalInternal
returns the sum of the current value of the <