Skip to content

LogicalRelation Leaf Logical Operator -- Representing BaseRelations in Logical Plan

LogicalRelation is a[leaf logical operator] that represents a <> in a[logical query plan].

[source, scala]

val q1 ="header", true).csv("../datasets/people.csv") scala> println(q1.queryExecution.logical.numberedTreeString) 00 Relation[id#72,name#73,age#74] csv

val q2 = sql("select * from csv.../datasets/people.csv") scala> println(q2.queryExecution.optimizedPlan.numberedTreeString) 00 Relation[_c0#175,_c1#176,_c2#177] csv

LogicalRelation is <> when:

  • DataFrameReader loads data from a data source that supports multiple paths (through[SparkSession.baseRelationToDataFrame])
  • DataFrameReader is requested to load data from an external table using JDBC (through[SparkSession.baseRelationToDataFrame])
  • TextInputCSVDataSource and TextInputJsonDataSource are requested to infer schema
  • ResolveSQLOnFile converts a logical plan
  • FindDataSourceTable logical evaluation rule is executed
  • hive/[RelationConversions] logical evaluation rule is executed
  • CreateTempViewUsing logical command is requested to <>
  • Structured Streaming's FileStreamSource creates batches of records

[[simpleString]] The catalyst/[simple text representation] of a LogicalRelation (aka simpleString) is Relation[output] [relation] (that uses the <> and <>).

[source, scala]

val q ="") val logicalPlan = q.queryExecution.logical scala> println(logicalPlan.simpleString) Relation[value#2] text

Creating Instance

LogicalRelation takes the following when created:

=== [[apply]] apply Factory Utility

[source, scala]

apply( relation: BaseRelation, isStreaming: Boolean = false): LogicalRelation apply( relation: BaseRelation, table: CatalogTable): LogicalRelation

apply <> a LogicalRelation for the input BaseRelation (and CatalogTable or optional isStreaming flag).

apply is used when:

=== [[refresh]] refresh Method

[source, scala]

refresh(): Unit

NOTE: refresh is part of[LogicalPlan Contract] to refresh itself.

refresh requests the FileIndex of a HadoopFsRelation <> to refresh.


refresh does the work for HadoopFsRelation relations only.

Last update: 2020-11-13