Skip to content

SortExec Unary Physical Operator

SortExec is a unary physical operator that is <> when:

SortExec supports Java code generation (aka codegen).

val q = Seq((0, "zero"), (1, "one")).toDF("id", "name").sort('id)
val qe = q.queryExecution

val logicalPlan = qe.analyzed
scala> println(logicalPlan.numberedTreeString)
00 Sort [id#72 ASC NULLS FIRST], true
01 +- Project [_1#69 AS id#72, _2#70 AS name#73]
02    +- LocalRelation [_1#69, _2#70]

// BasicOperators does the conversion of Sort logical operator to SortExec
val sparkPlan = qe.sparkPlan
scala> println(sparkPlan.numberedTreeString)
00 Sort [id#72 ASC NULLS FIRST], true, 0
01 +- LocalTableScan [id#72, name#73]

// SortExec supports Whole-Stage Code Generation
val executedPlan = qe.executedPlan
scala> println(executedPlan.numberedTreeString)
00 *(1) Sort [id#72 ASC NULLS FIRST], true, 0
01 +- Exchange rangepartitioning(id#72 ASC NULLS FIRST, 200)
02    +- LocalTableScan [id#72, name#73]

import org.apache.spark.sql.execution.SortExec
val sortExec = executedPlan.collect { case se: SortExec => se }.head
assert(sortExec.isInstanceOf[SortExec])

[[output]] When requested for the <>, SortExec simply gives whatever the <> uses.

[[outputOrdering]] SortExec uses the <> for the <>.

[[outputPartitioning]] When requested for the <>, SortExec simply gives whatever the <> uses.

[[requiredChildDistribution]] When requested for the <>, SortExec gives the OrderedDistribution (with the <> for the ordering) when the <> flag is enabled (true) or the UnspecifiedDistribution.

SortExec operator uses the spark.sql.sort.enableRadixSort internal configuration property (enabled by default) to control...FIXME

[[metrics]] .SortExec's Performance Metrics [cols="1,2,2",options="header",width="100%"] |=== | Key | Name (in web UI) | Description

| peakMemory | peak memory | [[peakMemory]]

| sortTime | sort time | [[sortTime]]

| spillSize | spill size | [[spillSize]] |===

=== [[creating-instance]] Creating SortExec Instance

SortExec takes the following when created:

  • [[sortOrder]] <> (Seq[SortOrder])
  • [[global]] global flag
  • [[child]] Child <>
  • [[testSpillFrequency]] testSpillFrequency (default: 0)

Last update: 2020-10-18