UserDefinedFunction¶
UserDefinedFunction
represents a user-defined function.
UserDefinedFunction
is <
-
udf function is executed
-
UDFRegistration
is requested to register a Scala function as a user-defined function (inFunctionRegistry
)
import org.apache.spark.sql.functions.udf
val lengthUDF = udf { s: String => s.length }
scala> :type lengthUDF
org.apache.spark.sql.expressions.UserDefinedFunction
val r = lengthUDF($"name")
scala> :type r
org.apache.spark.sql.Column
UserDefinedFunction
can have an optional <
val namedLengthUDF = lengthUDF.withName("lengthUDF")
scala> namedLengthUDF($"name")
res2: org.apache.spark.sql.Column = UDF:lengthUDF(name)
UserDefinedFunction
is nullable by default, but can be changed as <
[source, scala]¶
val nonNullableLengthUDF = lengthUDF.asNonNullable assert(nonNullableLengthUDF.nullable == false)
[[deterministic]] UserDefinedFunction
is deterministic <<_deterministic, by default>>, i.e. produces the same result for the same input. UserDefinedFunction
can be changed to be <
[source, scala]¶
assert(lengthUDF.deterministic) val ndUDF = lengthUDF.asNondeterministic assert(ndUDF.deterministic == false)
[[internal-registries]] .UserDefinedFunction's Internal Properties (e.g. Registries, Counters and Flags) [cols="1m,3",options="header",width="100%"] |=== | Name | Description
| _deterministic a| [[_deterministic]] Flag that controls whether the function is <true
) or not (false
).
Default: true
- Use <
> to change it to false
Used when UserDefinedFunction
is requested to <
|===
=== [[apply]] Executing UserDefinedFunction (Creating Column with ScalaUDF Expression) -- apply
Method
apply(
exprs: Column*): Column
apply
creates a Column with ScalaUDF expression.
import org.apache.spark.sql.functions.udf
scala> val lengthUDF = udf { s: String => s.length }
lengthUDF: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(<function1>,IntegerType,Some(List(StringType)))
scala> lengthUDF($"name")
res1: org.apache.spark.sql.Column = UDF(name)
apply
is used when...FIXME
=== [[asNonNullable]] Marking UserDefinedFunction as NonNullable -- asNonNullable
Method
[source, scala]¶
asNonNullable(): UserDefinedFunction¶
asNonNullable
...FIXME
asNonNullable
is used when...FIXME
=== [[withName]] Naming UserDefinedFunction -- withName
Method
[source, scala]¶
withName(name: String): UserDefinedFunction¶
withName
...FIXME
NOTE: withName
is used when...FIXME
Creating Instance¶
UserDefinedFunction
takes the following when created:
- [[f]] A Scala function (as Scala's
AnyRef
) - [[dataType]] Output data type
- [[inputTypes]] Input data types (if available)
UserDefinedFunction
initializes the <