Skip to content

DataWritingCommand -- Logical Commands That Write Query Data

DataWritingCommand is an <> of the <> for <> that write the result of executing <> (query data) to a relation when <>.

DataWritingCommand is resolved to a <> physical operator when BasicOperators execution planning strategy is executed (i.e. plan a <> to a <>).

[[contract]] .DataWritingCommand Contract [cols="1m,2",options="header",width="100%"] |=== | Property | Description

| outputColumnNames a| [[outputColumnNames]]

[source, scala]

outputColumnNames: Seq[String]

The output column names of the <>

Used when DataWritingCommand is requested for the <>

| query a| [[query]]

[source, scala]

query: LogicalPlan

The analyzed <> representing the data to write (i.e. whose result will be inserted into a relation)

Used when DataWritingCommand is requested for the <> and <>.

| run a| [[run]]

[source, scala]

run( sparkSession: SparkSession, child: SparkPlan): Seq[Row]

Executes the command to write query data (the result of executing[structured query])

Used when:

[[children]] When requested for the <>, DataWritingCommand simply returns the <>.

[[extensions]] .DataWritingCommands (Direct Implementations and Extensions Only) [cols="1,2",options="header",width="100%"] |=== | DataWritingCommand | Description

|[CreateDataSourceTableAsSelectCommand] | [[CreateDataSourceTableAsSelectCommand]]

| hive/[CreateHiveTableAsSelectCommand] | [[CreateHiveTableAsSelectCommand]]

| InsertIntoHadoopFsRelationCommand | [[InsertIntoHadoopFsRelationCommand]]

| hive/[SaveAsHiveFile] | [[SaveAsHiveFile]] Commands that write query result as Hive files (i.e. hive/[InsertIntoHiveDirCommand] and hive/[InsertIntoHiveTable])


=== [[basicWriteJobStatsTracker]] basicWriteJobStatsTracker Method

[source, scala]

basicWriteJobStatsTracker(hadoopConf: Configuration): BasicWriteJobStatsTracker

basicWriteJobStatsTracker simply creates and returns a new BasicWriteJobStatsTracker (with the given Hadoop Configuration and the <>).


basicWriteJobStatsTracker is used when:

  • SaveAsHiveFile is requested to <> (when hive/[InsertIntoHiveDirCommand] and hive/[InsertIntoHiveTable] logical commands are executed)

* InsertIntoHadoopFsRelationCommand logical command is executed

=== [[outputColumns]] Output Columns -- outputColumns Method

[source, scala]

outputColumns: Seq[Attribute]



outputColumns is used when:

  • hive/[CreateHiveTableAsSelectCommand], hive/[InsertIntoHiveDirCommand] and InsertIntoHadoopFsRelationCommand logical commands are executed

* SaveAsHiveFile is requested to <>

Performance Metrics

Key Name (in web UI) Description
numFiles number of written files
numOutputBytes bytes of written output
numOutputRows number of output rows
numParts number of dynamic part

Last update: 2021-05-05
Back to top