Skip to content

SQLConf — Internal Configuration Store

SQLConf is an internal configuration store for parameters and hints used to configure a Spark Structured Streaming application (and Spark SQL applications in general).

Tip

Find out more on SQLConf in The Internals of Spark SQL

streamingFileCommitProtocolClass

spark.sql.streaming.commitProtocolClass configuration property

Used when FileStreamSink is requested to "add" a batch of data

streamingMetricsEnabled

spark.sql.streaming.metricsEnabled configuration property

Used when StreamExecution is requested to runStream

fileSinkLogCleanupDelay

spark.sql.streaming.fileSink.log.cleanupDelay configuration property

Used when FileStreamSinkLog is created

fileSinkLogDeletion

spark.sql.streaming.fileSink.log.deletion configuration property

Used when FileStreamSinkLog is created

fileSinkLogCompactInterval

spark.sql.streaming.fileSink.log.compactInterval configuration property

Used when FileStreamSinkLog is created

minBatchesToRetain

spark.sql.streaming.minBatchesToRetain configuration property

Used when:

[[accessor-methods]] .SQLConf's Property Accessor Methods [cols="1,1",options="header",width="100%"] |=== | Method Name / Property | Description

| continuousStreamingExecutorQueueSize

spark.sql.streaming.continuous.executorQueueSize

a| [[continuousStreamingExecutorQueueSize]] Used when:

  • DataSourceV2ScanExec leaf physical operator is requested for the input RDDs (and creates a <>)

  • ContinuousCoalesceExec unary physical operator is requested to execute

| continuousStreamingExecutorPollIntervalMs

spark.sql.streaming.continuous.executorPollIntervalMs

a| [[continuousStreamingExecutorPollIntervalMs]] Used exclusively when DataSourceV2ScanExec leaf physical operator is requested for the input RDDs (and creates a <>)

| disabledV2StreamingMicroBatchReaders

spark.sql.streaming.disabledV2MicroBatchReaders

a| [[disabledV2StreamingMicroBatchReaders]] Used exclusively when MicroBatchExecution is requested for the <> (of a streaming query)

| fileSourceLogDeletion

spark.sql.streaming.fileSource.log.deletion

a| [[fileSourceLogDeletion]][[FILE_SOURCE_LOG_DELETION]] Used exclusively when FileStreamSourceLog is requested for the isDeletingExpiredLog

| fileSourceLogCleanupDelay

spark.sql.streaming.fileSource.log.cleanupDelay

a| [[fileSourceLogCleanupDelay]][[FILE_SOURCE_LOG_CLEANUP_DELAY]] Used exclusively when FileStreamSourceLog is requested for the fileCleanupDelayMs

| fileSourceLogCompactInterval

spark.sql.streaming.fileSource.log.compactInterval

a| [[fileSourceLogCompactInterval]][[FILE_SOURCE_LOG_COMPACT_INTERVAL]] Used exclusively when FileStreamSourceLog is requested for the default compaction interval

| FLATMAPGROUPSWITHSTATE_STATE_FORMAT_VERSION

spark.sql.streaming.flatMapGroupsWithState.stateFormatVersion

a| [[FLATMAPGROUPSWITHSTATE_STATE_FORMAT_VERSION]] Used when:

| SHUFFLE_PARTITIONS

spark.sql.shuffle.partitions a| [[SHUFFLE_PARTITIONS]] See https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-properties.html#spark.sql.shuffle.partitions[spark.sql.shuffle.partitions] in The Internals of Spark SQL.

| stateStoreMinDeltasForSnapshot

spark.sql.streaming.stateStore.minDeltasForSnapshot

a| [[stateStoreMinDeltasForSnapshot]] Used (as StateStoreConf.minDeltasForSnapshot) exclusively when HDFSBackedStateStoreProvider is requested to doSnapshot

| stateStoreProviderClass

spark.sql.streaming.stateStore.providerClass

a| [[stateStoreProviderClass]] Used when:

| STREAMING_AGGREGATION_STATE_FORMAT_VERSION

spark.sql.streaming.aggregation.stateFormatVersion

a| [[STREAMING_AGGREGATION_STATE_FORMAT_VERSION]] Used when:

| STREAMING_CHECKPOINT_FILE_MANAGER_CLASS

spark.sql.streaming.checkpointFileManagerClass a| [[STREAMING_CHECKPOINT_FILE_MANAGER_CLASS]] Used exclusively when CheckpointFileManager helper object is requested to create a CheckpointFileManager

| streamingMetricsEnabled

spark.sql.streaming.metricsEnabled

a| [[streamingMetricsEnabled]] Used exclusively when StreamExecution is requested for runStream (to control whether to register a metrics reporter for a streaming query)

| STREAMING_MULTIPLE_WATERMARK_POLICY

spark.sql.streaming.multipleWatermarkPolicy

a| [[STREAMING_MULTIPLE_WATERMARK_POLICY]]

| streamingNoDataMicroBatchesEnabled

spark.sql.streaming.noDataMicroBatches.enabled

a| [[streamingNoDataMicroBatchesEnabled]][[STREAMING_NO_DATA_MICRO_BATCHES_ENABLED]] Used exclusively when MicroBatchExecution stream execution engine is requested to <>

| streamingNoDataProgressEventInterval

spark.sql.streaming.noDataProgressEventInterval

a| [[streamingNoDataProgressEventInterval]] Used exclusively for ProgressReporter

| streamingPollingDelay

spark.sql.streaming.pollingDelay

a| [[streamingPollingDelay]][[STREAMING_POLLING_DELAY]] Used exclusively when StreamExecution is created

| streamingProgressRetention

spark.sql.streaming.numRecentProgressUpdates

a| [[streamingProgressRetention]][[STREAMING_PROGRESS_RETENTION]] Used exclusively when ProgressReporter is requested to update progress of streaming query (and possibly remove an excess)

|===