Skip to content

UnresolvedHint Unary Logical Operator

UnresolvedHint is an unary logical operator that represents a hint (by name) for the child logical plan.

Creating Instance

UnresolvedHint takes the following to be created:

UnresolvedHint is created when:

Never Resolved

resolved: Boolean

resolved is part of the LogicalPlan abstraction.

resolved is false.

Logical Analysis

UnresolvedHints cannot be resolved and are supposed to be converted to ResolvedHint unary logical operators at analysis or removed from a logical plan.

The following logical rules are used to act on UnresolvedHint logical operators (the order of executing the rules matters):

Analyzer throws an IllegalStateException for any UnresolvedHints left (unresolved):

Internal error: logical hint operator should have been removed during analysis

Catalyst DSL

UnresolvedHint can be created using the hint operator in Catalyst DSL.

import org.apache.spark.sql.catalyst.plans.logical.LocalRelation
val r1 = LocalRelation('a.int, 'b.timestamp, 'c.boolean)

scala> println(r1.numberedTreeString)
00 LocalRelation <empty>, [a#0, b#1, c#2]

import org.apache.spark.sql.catalyst.dsl.plans._
val plan = r1.hint(name = "myHint", 100, true)

scala> println(plan.numberedTreeString)
00 'UnresolvedHint myHint, [100, true]
01 +- LocalRelation <empty>, [a#0, b#1, c#2]

Demo

// Dataset API
val q = spark.range(1).hint("myHint", 100, true)
val plan = q.queryExecution.logical
scala> println(plan.numberedTreeString)
00 'UnresolvedHint myHint, [100, true]
01 +- Range (0, 1, step=1, splits=Some(8))

// SQL
val q = sql("SELECT /*+ myHint (100, true) */ 1")
val plan = q.queryExecution.logical
scala> println(plan.numberedTreeString)
00 'UnresolvedHint myHint, [100, true]
01 +- 'Project [unresolvedalias(1, None)]
02    +- OneRowRelation

// Let's hint the query twice
// The order of hints matters as every hint operator executes Spark analyzer
// That will resolve all but the last hint
val q = spark.range(100).
  hint("broadcast").
  hint("myHint", 100, true)
val plan = q.queryExecution.logical
scala> println(plan.numberedTreeString)
00 'UnresolvedHint myHint, [100, true]
01 +- ResolvedHint (broadcast)
02    +- Range (0, 100, step=1, splits=Some(8))

// Let's resolve unresolved hints
import org.apache.spark.sql.catalyst.rules.RuleExecutor
import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
import org.apache.spark.sql.catalyst.analysis.ResolveHints
import org.apache.spark.sql.internal.SQLConf
object HintResolver extends RuleExecutor[LogicalPlan] {
  lazy val batches =
    Batch("Hints", FixedPoint(maxIterations = 100),
      new ResolveHints.ResolveJoinStrategyHints(SQLConf.get),
      ResolveHints.RemoveAllHints) :: Nil
}
val resolvedPlan = HintResolver.execute(plan)
scala> println(resolvedPlan.numberedTreeString)
00 ResolvedHint (broadcast)
01 +- Range (0, 100, step=1, splits=Some(8))
Last update: 2021-05-24
Back to top