Skip to content

ParseToTimestamp Expression

ParseToTimestamp is a spark-sql-Expression-RuntimeReplaceable.md[RuntimeReplaceable] expression that <> the spark-sql-functions-datetime.md#to_timestamp[to_timestamp] function (in logical query plans).

[source, scala]

// DEMO to_timestamp(s: Column): Column import java.sql.Timestamp import java.time.LocalDateTime val times = Seq(Timestamp.valueOf(LocalDateTime.of(2018, 5, 30, 0, 0, 0)).toString).toDF("time") scala> times.printSchema root |-- time: string (nullable = true)

import org.apache.spark.sql.functions.to_timestamp val q = times.select(to_timestamp($"time") as "ts") scala> q.printSchema root |-- ts: timestamp (nullable = true)

val plan = q.queryExecution.analyzed scala> println(plan.numberedTreeString) 00 Project [to_timestamp('time, None) AS ts#29] 01 +- Project [value#16 AS time#18] 02 +- LocalRelation [value#16]

import org.apache.spark.sql.catalyst.expressions.ParseToTimestamp val ptt = plan.expressions.head.children.head.asInstanceOf[ParseToTimestamp] scala> println(ptt.numberedTreeString) 00 to_timestamp('time, None) 01 +- cast(time#18 as timestamp) 02 +- time#18: string

// FIXME DEMO to_timestamp(s: Column, fmt: String): Column

As a RuntimeReplaceable expression, ParseToTimestamp is replaced by Logical Optimizer with the <> expression:

  • Cast(left, TimestampType) for to_timestamp(s: Column): Column function

  • Cast(UnixTimestamp(left, format), TimestampType) for to_timestamp(s: Column, fmt: String): Column function

[source, scala]

// FIXME DEMO Conversion to Cast(left, TimestampType) // FIXME DEMO Conversion to Cast(UnixTimestamp(left, format), TimestampType)


=== [[creating-instance]] Creating ParseToTimestamp Instance

ParseToTimestamp takes the following when created:

  • [[left]] Left Expression.md[expression]
  • [[format]] format Expression.md[expression]
  • [[child]] Child Expression.md[expression]

Last update: 2020-10-21
Back to top