Skip to content

StreamSourceProvider

StreamSourceProvider is an abstraction of data source providers that can create a streaming source for a data format or system.

StreamSourceProvider is part of Data Source API V1 for Micro-Batch Stream Processing.

Contract

Creating Streaming Source

createSource(
  sqlContext: SQLContext,
  metadataPath: String,
  schema: Option[StructType],
  providerName: String,
  parameters: Map[String, String]): Source

Creates a Streaming Source

metadataPath is the value of the optional user-specified checkpointLocation option or resolved by StreamingQueryManager.

Used when:

Source Schema

sourceSchema(
  sqlContext: SQLContext,
  schema: Option[StructType],
  providerName: String,
  parameters: Map[String, String]): (String, StructType)

Name and schema of the Streaming Source

Used when:

Implementations


Last update: 2021-02-07