OutputMode of a streaming query describes what data is written to a streaming sink.
The output mode is specified on the writing side of a streaming query using DataStreamWriter.outputMode method (by alias or a value of
import org.apache.spark.sql.streaming.OutputMode.Update val inputStream = spark .readStream .format("rate") .load .writeStream .format("console") .outputMode(Update) // <-- update output mode .start
Append Output Mode¶
Append (alias: append) is the default output mode that writes "new" rows only.
In streaming aggregations, a "new" row is when the intermediate state becomes final, i.e. when new events for the grouping key can only be considered late which is when watermark moves past the event time of the key.
Append output mode requires that a streaming query defines event-time watermark (using withWatermark operator) on the event time column that is used in aggregation (directly or using window standard function).
Required for datasets with
FileFormat format (to create FileStreamSink)
Append is mandatory when multiple
flatMapGroupsWithState operators are used in a structured query.
Complete Output Mode¶
Complete (alias: complete) writes all the rows of a Result Table (and corresponds to a traditional batch structured query).
Complete mode does not drop old aggregation state and preserves all data in the Result Table.
Update Output Mode¶
Update (alias: update) writes only the rows that were updated (every time there are updates).