ContinuousExecution¶

ContinuousExecution is the stream execution engine of Continuous Stream Processing.

ContinuousExecution can only run streaming queries with StreamingRelationV2 leaf logical operators with ContinuousReadSupport data source.

When created (for a streaming query), ContinuousExecution is given the <>. The analyzed logical plan is immediately transformed to include a ContinuousExecutionRelation for every StreamingRelationV2 leaf logical operator with ContinuousReadSupport data source (and is the logical plan internally).

When requested to <>, ContinuousExecution collects ContinuousReadSupport data sources (inside ContinuousExecutionRelation) from the <> and requests each and every ContinuousReadSupport to create a ContinuousReader (that are stored in <> internal registry).

Local Properties¶

__epoch_coordinator_id¶

ContinuousExecution uses __epoch_coordinator_id local property for...FIXME

__continuous_start_epoch¶

ContinuousExecution uses __continuous_start_epoch local property for...FIXME

__continuous_epoch_interval¶

ContinuousExecution uses __continuous_epoch_interval local property for...FIXME

TriggerExecutor¶

TriggerExecutor for the Trigger:

• ProcessingTimeExecutor for ContinuousTrigger

Used when...FIXME

Running Activated Streaming Query¶

runActivatedStream(
  sparkSessionForStream: SparkSession): Unit

runActivatedStream simply runs the streaming query in continuous mode as long as the state is ACTIVE.

runActivatedStream is part of StreamExecution abstraction.

Running Streaming Query in Continuous Mode¶

runContinuous(
  sparkSessionForQuery: SparkSession): Unit

runContinuous initializes the continuousSources internal registry by traversing the analyzed logical plan to find ContinuousExecutionRelation leaf logical operators and requests their ContinuousReadSupport data sources to create a ContinuousReader (with the sources metadata directory under the checkpoint directory).

runContinuous initializes the uniqueSources internal registry to be the continuousSources distinct.

runContinuous gets the start offsets (they may or may not be available).

runContinuous transforms the analyzed logical plan. For every ContinuousExecutionRelation runContinuous finds the corresponding ContinuousReader (in the continuousSources), requests it to deserialize the start offsets (from their JSON representation), and then setStartOffset. In the end, runContinuous creates a StreamingDataSourceV2Relation (with the read schema of the ContinuousReader and the ContinuousReader itself).

runContinuous rewires the transformed plan (with the StreamingDataSourceV2Relation) to use the new attributes from the source (the reader).

runContinuous...FIXME

runContinuous finds the only ContinuousReader (of the only StreamingDataSourceV2Relation) in the query plan with the WriteToContinuousDataSource.

queryPlanning Phase¶

In queryPlanning time-tracking section, runContinuous creates an IncrementalExecution (that becomes the lastExecution) that is immediately executed (the entire query execution pipeline is executed up to and including executedPlan).

runContinuous sets the following local properties:

• __is_continuous_processing as true

• __continuous_start_epoch as the currentBatchId

• __epoch_coordinator_id as the currentEpochCoordinatorId, i.e. runId followed by -- with a random UUID

• __continuous_epoch_interval as the interval of the ContinuousTrigger

runContinuous uses the EpochCoordinatorRef helper to create a remote reference to the EpochCoordinator RPC endpoint (with the ContinuousReader, the currentEpochCoordinatorId, and the currentBatchId).

runContinuous creates a daemon epoch update thread and starts it immediately.

runContinuous Phase¶

In runContinuous time-tracking section, runContinuous requests the physical query plan (of the IncrementalExecution) to execute (that simply requests the physical operator to doExecute and generate an RDD[InternalRow]).

runContinuous is used when ContinuousExecution is requested to run an activated streaming query.

==== [[runContinuous-epoch-update-thread]] Epoch Update Thread

runContinuous creates an epoch update thread that...FIXME

==== [[getStartOffsets]] Getting Start Offsets From Checkpoint -- getStartOffsets Internal Method

[source, scala]¶

getStartOffsets(sparkSessionToRunBatches: SparkSession): OffsetSeq¶

getStartOffsets...FIXME

NOTE: getStartOffsets is used exclusively when ContinuousExecution is requested to <>.

Committing Epoch¶

commit(
  epoch: Long): Unit

In essence, commit adds the given epoch to commit log and the committedOffsets, and requests the <> to commit the corresponding offset. In the end, commit removes old log entries from the offset and commit logs (to keep spark.sql.streaming.minBatchesToRetain entries only).

Internally, commit recordTriggerOffsets (with the from and to offsets as the committedOffsets and availableOffsets, respectively).

At this point, commit may simply return when the stream execution thread is no longer alive (died).

commit requests the commit log to store a metadata for the epoch.

commit requests the single <> to deserialize the offset for the epoch (from the offset write-ahead log).

commit adds the single <> and the offset (for the epoch) to the committedOffsets registry.

commit requests the single <> to commit the offset.

commit requests the offset and commit logs to remove log entries to keep spark.sql.streaming.minBatchesToRetain only.

commit then acquires the awaitProgressLock, wakes up all threads waiting for the awaitProgressLockCondition and in the end releases the awaitProgressLock.

NOTE: commit supports only one continuous source (registered in the <> internal registry).

commit asserts that the given epoch is available in the offsetLog internal registry (i.e. the offset for the given epoch has been reported before).

commit is used when EpochCoordinator is requested to commitEpoch.

=== [[addOffset]] addOffset Method

[source, scala]¶

addOffset( epoch: Long, reader: ContinuousReader, partitionOffsets: Seq[PartitionOffset]): Unit

In essense, addOffset requests the given ContinuousReader to mergeOffsets (with the given PartitionOffsets) and then requests the OffsetSeqLog to register the offset with the given epoch.

Internally, addOffset requests the given ContinuousReader to mergeOffsets (with the given PartitionOffsets) and to get the current "global" offset back.

addOffset then requests the OffsetSeqLog to add the current "global" offset for the given epoch.

addOffset requests the OffsetSeqLog for the offset at the previous epoch.

If the offsets at the current and previous epochs are the same, addOffset turns the noNewData internal flag on.

addOffset then acquires the awaitProgressLock, wakes up all threads waiting for the awaitProgressLockCondition and in the end releases the awaitProgressLock.

NOTE: addOffset supports exactly one continuous source.

addOffset is used when EpochCoordinator is requested to handle a ReportPartitionOffset message.

Analyzed Logical Plan of Streaming Query¶

logicalPlan: LogicalPlan

logicalPlan resolves StreamingRelationV2 leaf logical operators (with a ContinuousReadSupport source) to ContinuousExecutionRelation leaf logical operators.

Internally, logicalPlan transforms the <> as follows:

. For every StreamingRelationV2 leaf logical operator with a ContinuousReadSupport source, logicalPlan looks it up for the corresponding ContinuousExecutionRelation (if available in the internal lookup registry) or creates a ContinuousExecutionRelation (with the ContinuousReadSupport source, the options and the output attributes of the StreamingRelationV2 operator)

. For any other StreamingRelationV2, logicalPlan throws an UnsupportedOperationException: +

Data source [name] does not support continuous processing.

logicalPlan is part of the StreamExecution abstraction.

Creating Instance¶

ContinuousExecution takes the following when created:

• [[sparkSession]] SparkSession
• [[name]] The name of the structured query
• [[checkpointRoot]] Path to the checkpoint directory (aka metadata directory)
• [[analyzedPlan]] Analyzed logical query plan (LogicalPlan)
• [[trigger]] Trigger
• [[triggerClock]] Clock
• [[outputMode]] OutputMode
• [[extraOptions]] Options (Map[String, String])
• [[deleteCheckpointOnStop]] deleteCheckpointOnStop flag to control whether to delete the checkpoint directory on stop

ContinuousExecution is created when StreamingQueryManager is requested to create a streaming query with a StreamWriteSupport sink and a ContinuousTrigger (when DataStreamWriter is requested to start an execution of the streaming query).

Stopping Stream Processing¶

stop(): Unit

stop is part of the StreamingQuery abstraction.

stop transitions the streaming query to TERMINATED state.

If the queryExecutionThread is alive (i.e. it has been started and has not yet died), stop interrupts it and waits for this thread to die.

In the end, stop prints out the following INFO message to the logs:

Query [prettyIdString] was stopped

=== [[awaitEpoch]] awaitEpoch Internal Method

[source, scala]¶

awaitEpoch(epoch: Long): Unit¶

awaitEpoch...FIXME

NOTE: awaitEpoch seems to be used exclusively in tests.

continuousSources¶

continuousSources: Seq[ContinuousReader]

Registry of ContinuousReaders (in the analyzed logical plan of the streaming query)

As asserted in <> and <> there could only be exactly one ContinuousReaders registered.

Used when ContinuousExecution is requested to <>, <>, and <>

Use <> to access the current value

sources¶

ContinuousExecution supports one <> only in a <> (and asserts it when <> and <>). When requested for available streaming sources, ContinuousExecution simply gives the <>.

import org.apache.spark.sql.streaming.Trigger
import scala.concurrent.duration._
val sq = spark
  .readStream
  .format("rate")
  .load
  .writeStream
  .format("console")
  .option("truncate", false)
  .trigger(../Trigger.Continuous(1.minute)) // <-- Gives ContinuousExecution
  .queryName("rate2console")
  .start

import org.apache.spark.sql.streaming.StreamingQuery
assert(sq.isInstanceOf[StreamingQuery])

// The following gives access to the internals
// And to ContinuousExecution
import org.apache.spark.sql.execution.streaming.StreamingQueryWrapper
val engine = sq.asInstanceOf[StreamingQueryWrapper].streamingQuery
import org.apache.spark.sql.execution.streaming.StreamExecution
assert(engine.isInstanceOf[StreamExecution])

import org.apache.spark.sql.execution.streaming.continuous.ContinuousExecution
val continuousEngine = engine.asInstanceOf[ContinuousExecution]
assert(continuousEngine.trigger == Trigger.Continuous(1.minute))

Logging¶

Enable ALL logging level for org.apache.spark.sql.execution.streaming.continuous.ContinuousExecution logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.sql.execution.streaming.continuous.ContinuousExecution=ALL

Refer to Logging.