Skip to content

SortExec Unary Physical Operator

SortExec is a unary physical operator that is <> when:

SortExec supports Java code generation (aka codegen).

val q = Seq((0, "zero"), (1, "one")).toDF("id", "name").sort('id)
val qe = q.queryExecution

val logicalPlan = qe.analyzed
scala> println(logicalPlan.numberedTreeString)
00 Sort [id#72 ASC NULLS FIRST], true
01 +- Project [_1#69 AS id#72, _2#70 AS name#73]
02    +- LocalRelation [_1#69, _2#70]

// BasicOperators does the conversion of Sort logical operator to SortExec
val sparkPlan = qe.sparkPlan
scala> println(sparkPlan.numberedTreeString)
00 Sort [id#72 ASC NULLS FIRST], true, 0
01 +- LocalTableScan [id#72, name#73]

// SortExec supports Whole-Stage Code Generation
val executedPlan = qe.executedPlan
scala> println(executedPlan.numberedTreeString)
00 *(1) Sort [id#72 ASC NULLS FIRST], true, 0
01 +- Exchange rangepartitioning(id#72 ASC NULLS FIRST, 200)
02    +- LocalTableScan [id#72, name#73]

import org.apache.spark.sql.execution.SortExec
val sortExec = executedPlan.collect { case se: SortExec => se }.head
assert(sortExec.isInstanceOf[SortExec])

[[output]] When requested for the <>, SortExec simply gives whatever the <> uses.

[[outputOrdering]] SortExec uses the <> for the <>.

[[outputPartitioning]] When requested for the <>, SortExec simply gives whatever the <> uses.

[[requiredChildDistribution]] When requested for the <>, SortExec gives the OrderedDistribution (with the <> for the ordering) when the <> flag is enabled (true) or the UnspecifiedDistribution.

SortExec operator uses the spark.sql.sort.enableRadixSort internal configuration property (enabled by default) to control...FIXME

=== [[creating-instance]] Creating SortExec Instance

SortExec takes the following when created:

  • [[sortOrder]] <> (Seq[SortOrder])
  • [[global]] global flag
  • [[child]] Child <>
  • [[testSpillFrequency]] testSpillFrequency (default: 0)

Performance Metrics

Key Name (in web UI) Description
peakMemory peak memory
sortTime sort time
spillSize spill size

Last update: 2021-05-05
Back to top