StreamingExecutionRelation Leaf Logical Operator¶
StreamingExecutionRelation
is a leaf logical operator (i.e. LogicalPlan
) that represents a streaming source in the logical query plan of a streaming Dataset
.
The main use of StreamingExecutionRelation
logical operator is to be a "placeholder" in a logical query plan that will be replaced with the real relation (with new data that has arrived since the last batch) or an empty LocalRelation
when StreamExecution
is requested to <
StreamingExecutionRelation
is <
Note
Right after StreamExecution
MicroBatchExecution.md#runStream-initializing-sources[has started running streaming batches] it initializes the streaming sources by transforming the analyzed logical plan of the streaming Dataset so that every StreamingRelation logical operator is replaced by the corresponding StreamingExecutionRelation
.
Note
StreamingExecutionRelation
is also resolved (planned) to a StreamingRelationExec physical operator in StreamingRelationStrategy execution planning strategy only when explaining a streaming Dataset
.
Creating Instance¶
StreamingExecutionRelation
takes the following when created:
- [[source]] Streaming source
- [[output]] Output attributes
=== [[apply]] Creating StreamingExecutionRelation (based on a Source) -- apply
Object Method
[source, scala]¶
apply(source: Source): StreamingExecutionRelation¶
apply
creates a StreamingExecutionRelation
for the input source
and with the attributes of the schema of the source
.
NOTE: apply
seems to be used for tests only.