StreamSourceProvider¶
StreamSourceProvider
is an abstraction of data source providers that can create a streaming source for a data format or system.
StreamSourceProvider
is part of Data Source API V1 for Micro-Batch Stream Processing.
Contract¶
Creating Streaming Source¶
createSource(
sqlContext: SQLContext,
metadataPath: String,
schema: Option[StructType],
providerName: String,
parameters: Map[String, String]): Source
Creates a streaming source
metadataPath
is the value of the optional user-specified checkpointLocation
option or resolved by StreamingQueryManager.
Used when DataSource
is requested to create a streaming source (when MicroBatchExecution
is requested to initialize the analyzed logical plan)
Source Schema¶
sourceSchema(
sqlContext: SQLContext,
schema: Option[StructType],
providerName: String,
parameters: Map[String, String]): (String, StructType)
Name and schema of the streaming source
Used when DataSource
is requested for metadata of a streaming source (when MicroBatchExecution
is requested to initialize the analyzed logical plan)