V2TableWriteExec Unary Physical Commands¶

V2TableWriteExec is an extension of the V2CommandExec abstraction for unary physical commands that writeWithV2.

Contract¶

Physical Query Plan¶

query: SparkPlan

SparkPlan for the data to be written out

Implementations¶

• AppendDataExec
• AtomicTableWriteExec
• CreateTableAsSelectExec
• OverwriteByExpressionExec
• OverwritePartitionsDynamicExec
• ReplaceTableAsSelectExec
• WriteToDataSourceV2Exec

writeWithV2¶

writeWithV2(
  batchWrite: BatchWrite): Seq[InternalRow]

writeWithV2 requests the physical query plan to execute (that gives a RDD[InternalRow]).

writeWithV2 requests the given BatchWrite for a DataWriterFactory.

writeWithV2 prints out the following INFO message to the logs:

Start processing data source write support: [batchWrite]. The input RDD has [n] partitions.

writeWithV2 runs a Spark job (Spark Core) with the DataWritingSparkTask for every partition. writeWithV2 requests the BatchWrite to onDataWriterCommit (with the result WriterCommitMessage) after every partition has been processed successfully.

writeWithV2 prints out the following INFO message to the logs:

Data source write support [batchWrite] is committing.

writeWithV2 requests the BatchWrite to commit (with all the result WriterCommitMessages).

writeWithV2 prints out the following INFO message to the logs:

Data source write support [batchWrite] committed.

In the end, writeWithV2 returns no InternalRows.

writeWithV2 is used when:

• V2TableWriteExec commands are executed