Skip to content

ContinuousReader — Data Source Readers in Continuous Stream Processing

ContinuousReader is the <> of Spark SQL's DataSourceReader abstraction for <> in Continuous Stream Processing.

ContinuousReader is part of the novel Data Source API V2 in Spark SQL.

Tip

Read up on Data Source API V2 in The Internals of Spark SQL online book.

[[contract]] .ContinuousReader Contract [cols="1m,3",options="header",width="100%"] |=== | Method | Description

| commit a| [[commit]]

[source, java]

void commit(Offset end)

Commits the specified offset

Used exclusively when ContinuousExecution is requested to commit an epoch

| deserializeOffset a| [[deserializeOffset]]

[source, java]

Offset deserializeOffset(String json)

Deserializes an offset from JSON representation

Used when ContinuousExecution is requested to run a streaming query and commit an epoch

| getStartOffset a| [[getStartOffset]]

[source, java]

Offset getStartOffset()

NOTE: Used exclusively in tests.

| mergeOffsets a| [[mergeOffsets]]

[source, java]

Offset mergeOffsets(PartitionOffset[] offsets)

Used exclusively when ContinuousExecution is requested to addOffset

| needsReconfiguration a| [[needsReconfiguration]]

[source, java]

boolean needsReconfiguration()

Indicates that the reader needs reconfiguration (e.g. to generate new input partitions)

Used exclusively when ContinuousExecution is requested to run a streaming query in continuous mode

| setStartOffset a| [[setStartOffset]]

[source, java]

void setStartOffset(Optional start)

Used exclusively when ContinuousExecution is requested to run the streaming query in continuous mode.

|===

[[implementations]] .ContinuousReaders [cols="1,2",options="header",width="100%"] |=== | ContinuousReader | Description

| ContinuousMemoryStream | [[ContinuousMemoryStream]]

| KafkaContinuousReader | [[KafkaContinuousReader]]

| RateStreamContinuousReader | [[RateStreamContinuousReader]]

| TextSocketContinuousReader | [[TextSocketContinuousReader]]

|===


Last update: 2021-02-07