Skip to content

DataFrameWriterV2

DataFrameWriterV2 is an API for Spark SQL developers to describe how to write a Dataset to an external storage using the DataSource V2.

DataFrameWriterV2 is a CreateTableWriter (and thus a WriteConfigMethods).

Demo

val nums = spark.range(5)
scala> :type nums
org.apache.spark.sql.Dataset[Long]

val writerV2 = nums.writeTo("t1")
scala> :type writerV2
org.apache.spark.sql.DataFrameWriterV2[Long]

Creating Instance

DataFrameWriterV2 takes the following to be created:

  • Name of the target table (multi-part table identifier)
  • Dataset

DataFrameWriterV2 is created when:

append

append(): Unit

append loads the table from the catalog and identifier (based on the table name).

When found, append creates an AppendData logical command (by name) with a DataSourceV2Relation logical operator (for the catalog and the identifier).

In the end, append runs the command with append execution name.

Tip

Execution is announced as a SparkListenerSQLExecutionEnd.

append throws a NoSuchTableException when the table could not be found.

scala> spark.range(5).writeTo("my.catalog.t1").append
org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table my.catalog.t1 not found;
  at org.apache.spark.sql.DataFrameWriterV2.append(DataFrameWriterV2.scala:162)
  ... 47 elided

create

create(): Unit

create is part of the CreateTableWriter abstraction.

create...FIXME

createOrReplace

createOrReplace(): Unit

createOrReplace is part of the CreateTableWriter abstraction.

createOrReplace...FIXME

option

option(
  key: String,
  value: String): DataFrameWriterV2[T]

option is part of the WriteConfigMethods abstraction.

option...FIXME

options

options(
  options: Map[String, String]): DataFrameWriterV2[T]

options is part of the WriteConfigMethods abstraction.

options...FIXME

overwrite

overwrite(
  condition: Column): Unit

overwrite...FIXME

overwritePartitions

overwritePartitions(): Unit

overwritePartitions...FIXME

partitionedBy

partitionedBy(
  column: Column,
  columns: Column*): CreateTableWriter[T]

partitionedBy is part of the CreateTableWriter abstraction.

partitionedBy...FIXME

replace

replace(): Unit

replace is part of the CreateTableWriter abstraction.

replace...FIXME

tableProperty

tableProperty(
  property: String,
  value: String): CreateTableWriter[T]

tableProperty is part of the CreateTableWriter abstraction.

tableProperty...FIXME

using

using(
  provider: String): CreateTableWriter[T]

using is part of the CreateTableWriter abstraction.

using...FIXME

Private Methods

internalReplace

internalReplace(
  orCreate: Boolean): Unit

internalReplace...FIXME

Executing Logical Command

runCommand(
  name: String)(
  command: LogicalPlan): Unit

runCommand...FIXME


Last update: 2020-12-28