Skip to content


UserDefinedFunction represents a user-defined function.

UserDefinedFunction is <> when:

import org.apache.spark.sql.functions.udf
val lengthUDF = udf { s: String => s.length }

scala> :type lengthUDF

val r = lengthUDF($"name")

scala> :type r

UserDefinedFunction can have an optional <>.

val namedLengthUDF = lengthUDF.withName("lengthUDF")
scala> namedLengthUDF($"name")
res2: org.apache.spark.sql.Column = UDF:lengthUDF(name)

UserDefinedFunction is nullable by default, but can be changed as <>.

[source, scala]

val nonNullableLengthUDF = lengthUDF.asNonNullable assert(nonNullableLengthUDF.nullable == false)

[[deterministic]] UserDefinedFunction is deterministic <<_deterministic, by default>>, i.e. produces the same result for the same input. UserDefinedFunction can be changed to be <>.

[source, scala]

assert(lengthUDF.deterministic) val ndUDF = lengthUDF.asNondeterministic assert(ndUDF.deterministic == false)

[[internal-registries]] .UserDefinedFunction's Internal Properties (e.g. Registries, Counters and Flags) [cols="1m,3",options="header",width="100%"] |=== | Name | Description

| _deterministic a| [[_deterministic]] Flag that controls whether the function is <> (true) or not (false).

Default: true

  • Use <> to change it to false

Used when UserDefinedFunction is requested to <>


=== [[apply]] Executing UserDefinedFunction (Creating Column with ScalaUDF Expression) -- apply Method

[source, scala]

apply(exprs: Column*): Column

apply creates a[Column] with[ScalaUDF] expression.

[source, scala]

import org.apache.spark.sql.functions.udf scala> val lengthUDF = udf { s: String => s.length } lengthUDF: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(,IntegerType,Some(List(StringType)))

scala> lengthUDF($"name") res1: org.apache.spark.sql.Column = UDF(name)

NOTE: apply is used when...FIXME

=== [[asNonNullable]] Marking UserDefinedFunction as NonNullable -- asNonNullable Method

[source, scala]

asNonNullable(): UserDefinedFunction


NOTE: asNonNullable is used when...FIXME

=== [[withName]] Naming UserDefinedFunction -- withName Method

[source, scala]

withName(name: String): UserDefinedFunction


NOTE: withName is used when...FIXME

Creating Instance

UserDefinedFunction takes the following when created:

  • [[f]] A Scala function (as Scala's AnyRef)
  • [[dataType]] Output data type
  • [[inputTypes]] Input data types (if available)

UserDefinedFunction initializes the <>.

Last update: 2020-10-21