Skip to content

ResolveCoalesceHints Logical Resolution Rule

ResolveCoalesceHints is a logical resolution rule to resolve UnresolvedHint logical operators.

Hint Name Arguments Logical Operator
COALESCE Number of partitions Repartition (with shuffle off / false)
REBALANCE RebalancePartitions
REPARTITION Number of partitions alone or like REPARTITION_BY_RANGE Repartition (with shuffle on / true)
REPARTITION_BY_RANGE Column names with an optional number of partitions (default: spark.sql.shuffle.partitions configuration property) RepartitionByExpression

ResolveCoalesceHints is a Catalyst rule for transforming logical plans (Rule[LogicalPlan]).

ResolveCoalesceHints is part of Hints batch of rules of Logical Analyzer.

Creating Instance

ResolveCoalesceHints takes the following to be created:

ResolveCoalesceHints is created when Logical Analyzer is requested for the batches of rules.

Executing Rule

apply(
  plan: LogicalPlan): LogicalPlan

apply resolves UnresolvedHint logical operators with the following hint names (case-insensitive).

Hint Name Trigger
COALESCE createRepartition (with shuffle off)
REBALANCE createRebalance
REPARTITION createRepartition (with shuffle on)
REPARTITION_BY_RANGE createRepartitionByRange

apply is part of the Rule abstraction.

createRebalance

createRebalance(
  hint: UnresolvedHint): LogicalPlan

createRebalance handles a REBALANCE hint and creates a Repartition logical operator.

createRepartition

createRepartition(
  shuffle: Boolean,
  hint: UnresolvedHint): LogicalPlan

createRepartition handles COALESCE and REPARTITION hints (and creates Repartition or RepartitionByExpression logical operators).

createRepartitionByRange

createRepartitionByRange(
  hint: UnresolvedHint): RepartitionByExpression

createRepartitionByRange creates a RepartitionByExpression logical operator.

Examples

Using COALESCE Hint

// Use Catalyst DSL to create a logical plan
import org.apache.spark.sql.catalyst.dsl.plans._
val plan = table("t1").hint(name = "COALESCE", 3)
scala> println(plan.numberedTreeString)
00 'UnresolvedHint COALESCE, [3]
01 +- 'UnresolvedRelation `t1`

import org.apache.spark.sql.catalyst.analysis.ResolveHints.ResolveCoalesceHints
val analyzedPlan = ResolveCoalesceHints(plan)
scala> println(analyzedPlan.numberedTreeString)
00 'Repartition 3, false
01 +- 'UnresolvedRelation `t1`

Using REPARTITION Hint

// Use Catalyst DSL to create a logical plan
import org.apache.spark.sql.catalyst.dsl.plans._
val plan = table("t1").hint(name = "REPARTITION", 3)
scala> println(plan.numberedTreeString)
00 'UnresolvedHint REPARTITION, [3]
01 +- 'UnresolvedRelation `t1`

import org.apache.spark.sql.catalyst.analysis.ResolveHints.ResolveCoalesceHints
val analyzedPlan = ResolveCoalesceHints(plan)
scala> println(analyzedPlan.numberedTreeString)
00 'Repartition 3, true
01 +- 'UnresolvedRelation `t1`

Using COALESCE Hint in SQL

val q = sql("SELECT /*+ COALESCE(10) */ * FROM VALUES 1 t(id)")
val plan = q.queryExecution.logical
scala> println(plan.numberedTreeString)
00 'UnresolvedHint COALESCE, [10]
01 +- 'Project [*]
02    +- 'SubqueryAlias `t`
03       +- 'UnresolvedInlineTable [id], [List(1)]

import org.apache.spark.sql.catalyst.analysis.ResolveHints.ResolveCoalesceHints
val analyzedPlan = ResolveCoalesceHints(plan)
scala> println(analyzedPlan.numberedTreeString)
00 'Repartition 10, false
01 +- 'Project [*]
02    +- 'SubqueryAlias `t`
03       +- 'UnresolvedInlineTable [id], [List(1)]
Back to top