Structured Streaming

Internals

StreamExecution — execution environment of a single continuous query (aka streaming Dataset)

StreamExecution has multiple streaming sources but only one streaming sink

Executes every trigger and adds results to the sink

Created exclusively when DataStreamWriter is started.

StreamExecution starts a thread of execution that runs the streaming query continuously and concurrently

StreamExecution collects duration for the execution units of a streaming batch

Use StreamingQuery.lastProgress or StreamingQuery.recentProgress

IncrementalExecution — QueryExecution of a streaming Dataset

Created (in queryPlanning Phase) for incremental execution of the logical query plan (every trigger)