Skip to content

Encoders Utility

Demo

import org.apache.spark.sql.Encoders
val encoder = Encoders.LOCALDATE
scala> :type encoder
org.apache.spark.sql.Encoder[java.time.LocalDate]

Creating Generic ExpressionEncoder using Java Serialization

javaSerialization[T: ClassTag]: Encoder[T]
javaSerialization[T](
  clazz: Class[T]): Encoder[T]

javaSerialization creates a generic ExpressionEncoder (with useKryo flag off).

Creating Generic ExpressionEncoder using Kryo Serialization

kryo[T: ClassTag]: Encoder[T]
kryo[T](
  clazz: Class[T]): Encoder[T]

kryo creates a generic ExpressionEncoder (with useKryo flag on).

Creating Generic ExpressionEncoder

genericSerializer[T: ClassTag](
  useKryo: Boolean): Encoder[T]

genericSerializer creates an ExpressionEncoder with the following:

Attribute Catalyst Expression
objSerializer EncodeUsingSerializer with BoundReference (ordinal = 0, dataType = ObjectType)
objDeserializer DecodeUsingSerializer with GetColumnByOrdinal leaf expression

genericSerializer is used when:

  • Encoders utility is used for a generic encoder using Kryo and Java serialization
Back to top