Skip to content
The Internals of Spark SQL
DebugExec
Initializing search
spark-sql-internals
Home
Internals
AQE
New & Noteworthy
High-Level APIs
Demos
Misc
The Internals of Spark SQL
spark-sql-internals
Home
Internals
Internals
Overview
Tags
Configuration Properties
Configuration Properties
Configuration Properties
SQLConf
StaticSQLConf
DataSource
Connector API
Connector API
ApplyTransform
Batch
BatchWrite
DataSourceV2Implicits
DataWriter
DataWriterFactory
InputPartition
FileTable
MetadataColumn
MetadataColumnHelper
MetadataColumnsHelper
OptionsHelper
Partitioning
PartitionReader
PartitionReaderFactory
PartitionSpecsHelper
RewritableTransform
Scan
ScanBuilder
SessionConfigSupport
SimpleTableProvider
StagedTable
SupportsAtomicPartitionManagement
SupportsDelete
SupportsDynamicOverwrite
SupportsMetadataColumns
SupportsOverwrite
SupportsPartitionManagement
SupportsPushDownFilters
SupportsPushDownRequiredColumns
SupportsRead
SupportsReportStatistics
SupportsReportPartitioning
SupportsStreamingUpdate
SupportsTruncate
SupportsWrite
Table
TableCapability
TableHelper
TableProvider
Transform
V1Scan
V1Table
V1WriteBuilder
WriteBuilder
DataSource V1 API
DataSource V1 API
DataSourceRegister
CreatableRelationProvider
RelationProvider
SchemaRelationProvider
BaseRelation
FileRelation
InsertableRelation
PrunedFilteredScan
PrunedScan
TableScan
Filter
Catalog Plugin API
Catalog Plugin API
CatalogManager
CatalogPlugin
CatalogHelper
CatalogExtension
DelegatingCatalogExtension
StagingTableCatalog
SupportsNamespaces
SupportsCatalogOptions
TableCatalog
TableChange
Utilities
Utilities
Catalogs
CatalogV2Util
Query Execution
Query Execution
QueryExecution
Logical Analyzer
SparkPlanner
SparkOptimizer
QueryPlanningTracker
SharedState
SQL Support
SQL Support
ParserInterface
AbstractSqlParser
AstBuilder
CatalystSqlParser
SparkSqlParser
SparkSqlAstBuilder
CatalogV2Implicits
LogicalExpressions
Expressions
FieldReference
Statistics and Hints
Statistics and Hints
Statistics
LogicalPlanStats
LogicalPlanVisitor
SizeInBytesOnlyStatsPlanVisitor
BasicStatsPlanVisitor
Estimation Utilities
Estimation Utilities
AggregateEstimation
EstimationUtils
FilterEstimation
JoinEstimation
ProjectEstimation
Logical Operators
Logical Operators
LogicalPlan
AlterTable
CommentOnTable
CreateTableAsSelect
CTERelationRef
CacheTableCommand
IgnoreCachedData
MapPartitions
ObjectConsumer
ObjectProducer
Command
DataWritingCommand
LeafNode
MultiInstanceRelation
NamedRelation
OrderPreservingUnaryNode
ParsedStatement
RepartitionOperation
ResolvedTable
ResolvedView
RunnableCommand
SupportsSubquery
V2CreateTablePlan
V2WriteCommand
Aggregate
AlterTableRecoverPartitionsCommand
AlterTableRecoverPartitionsStatement
AnalyzeColumn
AnalyzeColumnCommand
AnalyzePartitionCommand
AnalyzeTable
AnalyzeTableCommand
AppendData
ClearCacheCommand
CollectMetrics
CreateDataSourceTableAsSelectCommand
CreateDataSourceTableCommand
CreateTable
CreateTableAsSelectStatement
CreateTableCommand
CreateTableStatement
CreateTempViewUsing
CreateV2Table
CreateViewCommand
CTERelationDef
DataSourceV2Relation
DataSourceV2ScanRelation
DeleteFromTable
DescribeColumnCommand
DescribeRelation
DescribeTableCommand
DeserializeToObject
Except
Expand
ExplainCommand
ExternalRDD
Filter
Generate
GlobalLimit
GroupingSets
InMemoryRelation
InsertIntoDataSourceCommand
InsertIntoDir
InsertIntoHadoopFsRelationCommand
InsertIntoStatement
InsertIntoTable
Join
LocalRelation
LogicalRDD
LogicalRelation
MergeIntoTable
OneRowRelation
OverwriteByExpression
OverwritePartitionsDynamic
Pivot
Project
Range
RebalancePartitions
RepairTableStatement
Repartition
RepartitionByExpression
ResolvedHint
SaveIntoDataSourceCommand
SetCatalogAndNamespace
ShowCreateTableCommand
ShowCurrentNamespace
ShowCurrentNamespaceStatement
ShowTableProperties
ShowTablePropertiesCommand
ShowTables
Sort
Subquery
SubqueryAlias
TruncateTableCommand
TypedFilter
Union
UnresolvedCatalogRelation
UnresolvedHaving
UnresolvedHint
UnresolvedInlineTable
UnresolvedRelation
UnresolvedTable
UnresolvedTableOrView
UnresolvedTableValuedFunction
UnresolvedV2Relation
UnresolvedWith
UpdateTable
UseStatement
Window
WithCTE
WithWindowDefinition
WriteToDataSourceV2
View
SparkSession Registries
SparkSession Registries
Catalog
Catalog
Catalog
CatalogImpl
ExperimentalMethods
ExternalCatalog
ExternalCatalog
ExternalCatalog
InMemoryCatalog
FunctionRegistry
GlobalTempViewManager
SessionCatalog
SessionCatalog
SessionCatalog
CatalogTable
CatalogStorageFormat
CatalogTablePartition
BucketSpec
V2SessionCatalog
SessionState
SessionState
SessionState
BaseSessionStateBuilder
SessionStateBuilder
CacheManager
CacheManager
CacheManager
CachedRDDBuilder
RuntimeConfig
UDFRegistration
Data Sources
Data Sources
Noop
Noop
NoopDataSource
NoopTable
NoopWriteBuilder
NoopBatchWrite
NoopStreamingWrite
Files
Files
FileScan
FileScanBuilder
FileDataSourceV2
HadoopFsRelation
FileIndex
FileIndex
FileIndex
CatalogFileIndex
InMemoryFileIndex
PartitioningAwareFileIndex
PrunedInMemoryFileIndex
FileFormatWriter
FileWriteBuilder
SQLHadoopMapReduceCommitProtocol
PartitionedFile
RecordReaderIterator
FileFormat
TextBasedFileFormat
Kafka
Kafka
Configuration Properties
KafkaSourceProvider
KafkaTable
Options
KafkaScan
KafkaBatch
KafkaBatchWrite
KafkaRelation
KafkaSourceRDD
KafkaSourceRDD
KafkaSourceRDD
KafkaSourceRDDPartition
ConsumerStrategy
KafkaOffsetReader
KafkaOffsetRangeLimit
KafkaDataConsumer
InternalKafkaConsumer
KafkaWriter
KafkaWriteTask
JsonUtils
InternalKafkaProducerPool
Avro
Avro
AvroOptions
AvroFileFormat
CatalystDataToAvro
AvroDataToCatalyst
JDBC
JDBC
JDBCOptions
JdbcRelationProvider
JDBCRelation
JDBCScan
JDBCRDD
JdbcDialect
JdbcUtils
Console
Console
ConsoleSinkProvider
CSV
CSV
CSVFileFormat
CSVScanBuilder
JSON
JSON
JsonFileFormat
JsonDataSource
ORC
ORC
OrcScanBuilder
OrcFileFormat
Parquet
Parquet
ParquetFileFormat
ParquetReadSupport
ParquetScanBuilder
ParquetTable
SpecificParquetRecordReaderBase
VectorizedColumnReader
VectorizedParquetRecordReader
Text
Text
TextFileFormat
Hive Integration
Hive Integration
Configuration Properties
Hive Metastore
DataSinks
HiveFileFormat
HiveClient
HiveClient
HiveClient
HiveClientImpl
HiveUtils
IsolatedClientLoader
HiveTableRelation
CreateHiveTableAsSelectCommand
SaveAsHiveFile
InsertIntoHiveDirCommand
InsertIntoHiveTable
HiveTableScans
HiveTableScanExec
TableReader
HadoopTableReader
HiveSessionStateBuilder
HiveExternalCatalog
HiveSessionCatalog
HiveMetastoreCatalog
RelationConversions
ResolveHiveSerdeTable
DetermineTableStats
HiveAnalysis
Catalyst
Catalyst
GenericStrategy
Optimizer
PlanChangeLogger
QueryPlan
QueryPlanner
Rule
RuleExecutor
TreeNode
TreePattern
TreePatternBits
Expressions
Expressions
AggregateExpression
AggregateFunction
AggregateWindowFunction
Aggregator
AttributeReference
Alias
Attribute
BinaryComparison
BinaryOperator
BoundReference
CallMethodViaReflection
Coalesce
CodeGeneratorWithInterpretedFallback
CodegenFallback
CollectionGenerator
CreateArray
CreateNamedStruct
CreateNamedStructLike
CreateNamedStructUnsafe
CumeDist
DeclarativeAggregate
DecodeUsingSerializer
DynamicPruningExpression
DynamicPruningSubquery
EncodeUsingSerializer
EqualNullSafe
EqualTo
ExecSubqueryExpression
Exists
ExpectsInputTypes
ExplodeBase
Expression
First
Generator
GetArrayStructFields
GetArrayItem
GetMapValue
GetStructField
HashExpression
HashPartitioning
Higher-Order Functions
Higher-Order Functions
HigherOrderFunction
ArrayBasedSimpleHigherOrderFunction
MapBasedSimpleHigherOrderFunction
SimpleHigherOrderFunction
ImperativeAggregate
In
Inline
InSet
InSubquery
InSubqueryExec
InterpretedProjection
JsonToStructs
ListQuery
Literal
MonotonicallyIncreasingID
Murmur3Hash
MutableProjection
NamedExpression
Nondeterministic
NonSQLExpression
OffsetWindowFunction
ParseToDate
ParseToTimestamp
PlanExpression
Predicate
PrettyAttribute
Projection
RangePartitioning
RankLike
ResolvedStar
RowNumberLike
RowOrdering
RuntimeReplaceable
SafeProjection
ScalarSubquery
ScalarSubquery (ExecSubqueryExpression) Expression
ScalaUDF
ScalaUDAF
SizeBasedWindowFunction
SortOrder
Stack
Star
StaticInvoke
SubqueryExpression
TimeWindow
TypedImperativeAggregate
UnaryExpression
Unevaluable
UnixTimestamp
UnresolvedAttribute
UnresolvedFunction
UnresolvedGenerator
UnresolvedOrdinal
UnresolvedRegex
UnresolvedStar
UnresolvedWindowExpression
UnsafeProjection
UserDefinedExpression
WindowExpression
WindowFunction
WindowSpecDefinition
Physical Operators
Physical Operators
AliasAwareOutputExpression
AliasAwareOutputOrdering
AliasAwareOutputPartitioning
AlterTableExec
AppendDataExec
AppendDataExecV1
AtomicTableWriteExec
AtomicCreateTableAsSelectExec
AtomicReplaceTableAsSelectExec
AtomicReplaceTableExec
BaseAggregateExec
BaseJoinExec
BaseLimitExec
BaseSubqueryExec
BatchWriteHelper
BatchScanExec
BinaryExecNode
BlockingOperatorWithCodegen
BroadcastExchangeExec
BroadcastExchangeLike
BroadcastHashJoinExec
BroadcastNestedLoopJoinExec
CoalesceExec
CollectLimitExec
CollectMetricsExec
ColumnarToRowExec
CreateTableAsSelectExec
CreateTableExec
CustomShuffleReaderExec
CodegenSupport
ColumnarBatchScan
DataSourceScanExec
DataSourceV2ScanExec
DataSourceV2ScanExecBase
DataWritingCommandExec
DebugExec
DeleteFromTableExec
DescribeTableExec
DeserializeToObjectExec
DropNamespaceExec
DropTableExec
EvalPythonExec
Exchange
ExecutedCommandExec
ExternalRDDScanExec
HashedRelation
HashJoin
LeafExecNode
LimitExec
LongHashedRelation
ObjectConsumerExec
ReusedSubqueryExec
ShuffledJoin
ShuffleOrigin
SparkPlan
SQLMetric
SQLShuffleWriteMetricsReporter
SubqueryAdaptiveBroadcastExec
SupportsV1Write
TableWriteExecHelper
UnaryExecNode
UnsafeHashedRelation
V2CommandExec
V2TableWriteExec
V1FallbackWriters
Distribution and Partitioning
Distribution and Partitioning
Distribution
Partitioning
Distribution Specifications
Distribution Specifications
AllTuples
BroadcastDistribution
ClusteredDistribution
HashClusteredDistribution
OrderedDistribution
UnspecifiedDistribution
Broadcast Modes
Broadcast Modes
BroadcastMode
HashedRelationBroadcastMode
IdentityBroadcastMode
FileSourceScanExec
FilterExec
GenerateExec
HashAggregateExec
InMemoryTableScanExec
InputAdapter
InputRDDCodegen
LocalTableScanExec
MapElementsExec
MapPartitionsExec
ObjectHashAggregateExec
ObjectProducerExec
OverwriteByExpressionExec
OverwriteByExpressionExecV1
OverwritePartitionsDynamicExec
ProjectExec
RangeExec
RefreshTableExec
RenameTableExec
ReplaceTableAsSelectExec
ReplaceTableExec
ReusedExchangeExec
RowDataSourceScanExec
RowToColumnarExec
SerializeFromObjectExec
SetCatalogAndNamespaceExec
ShowCurrentNamespaceExec
ShowTablesExec
ShowTablePropertiesExec
ShuffleExchangeExec
ShuffleExchangeLike
ShuffledHashJoinExec
SortAggregateExec
SortMergeJoinExec
SortExec
SubqueryExec
SubqueryBroadcastExec
TakeOrderedAndProjectExec
WindowExec
WindowExec
WindowExec
AggregateProcessor
WindowFunctionFrame
WholeStageCodegenExec
WriteToDataSourceV2Exec
Logical Analysis Rules
Logical Analysis Rules
AddMetadataColumns
AliasViewChild
CleanupAliases
CTESubstitution
DataSourceAnalysis
ExtractWindowExpressions
FindDataSourceTable
HandleNullInputsForUDF
InConversion
LookupFunctions
PreprocessTableCreation
PreWriteCheck
RemoveAllHints
ResolveAggregateFunctions
ResolveAliases
ResolveCatalogs
ResolveCoalesceHints
ResolveCreateNamedStruct
ResolveFunctions
ResolveGroupingAnalytics
ResolveInlineTables
ResolveInsertInto
ResolveJoinStrategyHints
ResolveMissingReferences
ResolveOrdinalInOrderByAndGroupBy
ResolveOutputRelation
ResolveReferences
ResolveRelations
ResolveSessionCatalog
ResolveSQLOnFile
ResolveSubquery
ResolveTables
ResolveTempViews
ResolveTimeZone
ResolveWindowFrame
ResolveWindowOrder
ResolveWithCTE
TableCapabilityCheck
TimeWindowing
UpdateOuterReferences
WindowFrameCoercion
WidenSetOperationTypes
WindowsSubstitution
Logical Optimizations
Logical Optimizations
CleanupDynamicPruningFilters
CollapseWindow
ColumnPruning
CombineTypedFilters
CombineUnions
ComputeCurrentTime
ConstantFolding
CostBasedJoinReorder
DecimalAggregates
EliminateResolvedHint
EliminateSerialization
EliminateSubqueryAliases
EliminateView
ExtractPythonUDFFromAggregate
GetCurrentDatabase
InferFiltersFromConstraints
InlineCTE
LimitPushDown
NullPropagation
OptimizeIn
OptimizeMetadataOnlyQuery
OptimizeSubqueries
PartitionPruning
PropagateEmptyRelation
PruneFileSourcePartitions
PruneFilters
PullupCorrelatedPredicates
PushDownOperatorsToDataSource
PushDownPredicate
PushDownPredicates
PushPredicateThroughJoin
ReorderJoin
ReplaceExpressions
ReplaceExceptWithAntiJoin
ReplaceExceptWithFilter
RewriteCorrelatedScalarSubquery
RewriteExceptAll
RewritePredicateSubquery
SchemaPruning
SimplifyCasts
UpdateCTERelationStats
V2ScanRelationPushDown
Execution Planning Strategies
Execution Planning Strategies
SparkStrategy
Aggregation
BasicOperators
DataSourceStrategy
DataSourceV2Strategy
FileSourceStrategy
InMemoryScans
JoinSelection
LogicalQueryStageStrategy
SpecialLimits
SparkStrategies
WithCTEStrategy
Physical Optimizations
Physical Optimizations
ApplyColumnarRulesAndInsertTransitions
CoalesceBucketsInJoin
CollapseCodegenStages
DisableUnnecessaryBucketedScan
EnsureRequirements
ExtractPythonUDFs
PlanDynamicPruningFilters
PlanSubqueries
RemoveRedundantProjects
RemoveRedundantSorts
ReuseExchange
ReuseExchangeAndSubquery
ReuseSubquery
Tungsten Execution Backend
Tungsten Execution Backend
InternalRow
UnsafeRow
UnsafeRowSerializerInstance
CatalystSerde
ExternalAppendOnlyUnsafeRowArray
UnsafeFixedWidthAggregationMap
Encoder
Encoder
Encoder
ExpressionEncoder
RowEncoder
ScalaReflection
AQE
AQE
AQEUtils
LogicalQueryStage
InsertAdaptiveSparkPlan
AdaptiveSparkPlanExec
AQEShuffleReadExec
AdaptiveExecutionContext
PlanAdaptiveSubqueries
ShuffleStage
Adaptive Logical Optimizer
Adaptive Logical Optimizer
AQEOptimizer
DemoteBroadcastHashJoin
EliminateJoinToEmptyRelation
DynamicJoinSelection
Physical Optimizations
Physical Optimizations
AQEShuffleReadRule
CoalesceShufflePartitions
OptimizeShuffleWithLocalRead
OptimizeSkewInRebalancePartitions
OptimizeSkewedJoin
PlanAdaptiveDynamicPruningFilters
ReuseAdaptiveSubquery
QueryStageExecs
QueryStageExecs
QueryStageExec
BroadcastQueryStageExec
ShuffleQueryStageExec
CostEvaluator
SimpleCostEvaluator
Demo
New & Noteworthy
New & Noteworthy
Common Table Expressions
Join Queries
Join Queries
Joins
Broadcast Joins
Statistics
Metadata Columns
Catalog Plugin API and Multi-Catalog Support
Dynamic Partition Pruning
Explaining Query Plans Improved
Hints
Hints
Hint Framework
JoinHint
HintInfo
Join Strategy Hints
HintErrorHandler
Observable Metrics
Columnar Processing
DataSource V2
Hive Integration
Vectorized Parquet Decoding
Vectorized Parquet Decoding
Vectorized Parquet Decoding
ColumnVectors
ColumnVectors
ColumnVector
WritableColumnVector
OnHeapColumnVector
OffHeapColumnVector
Dynamic Partition Inserts
Bucketing
Whole-Stage CodeGen
Whole-Stage CodeGen
BufferedRowIterator
CodegenContext
CodeGenerator
Block
GenerateUnsafeProjection
GenerateMutableProjection
GenerateColumnAccessor
GenerateOrdering
GeneratePredicate
GenerateSafeProjection
BytesToBytesMap
Subexpression Elimination
Subexpression Elimination
Subexpression Elimination In Code-Generated Expression Evaluation (Common Expression Reuse)
EquivalentExpressions
Vectorized Query Execution
Vectorized Query Execution
Vectorized Query Execution
ColumnarBatch
Subqueries
Cost-Based Optimization
Cost-Based Optimization
Cost-Based Optimization (CBO) of Logical Query Plan
CatalogStatistics
ColumnStat
CommandUtils
Catalyst DSL
Catalyst DSL
DslLogicalPlan
High-Level APIs
High-Level APIs
SparkSession
SparkSession.Builder
SparkSessionExtensions
ExecutionListenerManager
ColumnarRule
Dataset
DataFrame
DataFrameReader
DataFrameWriter
DataFrameWriterV2
Encoders
Demos
Demos
Developing CatalogPlugin
Connecting Spark SQL to Hive Metastore
Hive Partitioned Parquet Table and Partition Pruning
Using JDBC Data Source to Access PostgreSQL
Misc
Misc
JoinSelectionHelper
ExplainUtils
DataSourceRDD
DataSourceRDDPartition
DataWritingSparkTask
Dataset, DataFrame and RDD
Dataset and SQL
DataSourceV2Utils
Implicits
Row
Data Source API
Data Source API
DataSource API
CreateTableWriter
WriteConfigMethods
Dataset API
Dataset API
Dataset API — Dataset Operators
Dataset API — Typed Transformations
Untyped Transformations
Dataset API — Basic Actions
Actions
DataFrameNaFunctions — Working With Missing Data
DataFrameStatFunctions
Column
Column
Column
Column Operators
TypedColumn
Basic Aggregation
Basic Aggregation
Basic Aggregation — Typed and Untyped Grouping Operators
RelationalGroupedDataset
KeyValueGroupedDataset
Window Aggregation
Window Aggregation
Window Aggregation
WindowSpec
Window Utility
Standard Functions
Standard Functions
Standard Functions — functions Object
Aggregate Functions
Standard Functions for Collections (Collection Functions)
Date and Time Functions
Regular Functions
Standard Functions for Window Aggregation (Window Functions)
User-Defined Functions
User-Defined Functions
User-Defined Functions
UDFs are Blackbox
UserDefinedFunction
Data Types
Data Types
AbstractDataType
ArrayType
AtomicType
DataType
StructField
StructType
Multi-Dimensional Aggregation
Caching and Persistence
Caching and Persistence
Caching and Persistence
User-Friendly Names Of Cached Queries in web UI's Storage Tab
Checkpointing
UserDefinedAggregateFunction
AggregationIterators
AggregationIterators
AggregationIterator
ObjectAggregationIterator
SortBasedAggregationIterator
TungstenAggregationIterator
FileScanRDD
Monitoring
Monitoring
SQLTab
SQLListener
QueryExecutionListener
SQLAppStatusListener
SQLAppStatusPlugin
SQLAppStatusStore
WriteTaskStats
WriteTaskStats
WriteTaskStats
BasicWriteTaskStats
WriteTaskStatsTracker
WriteTaskStatsTracker
WriteTaskStatsTracker
BasicWriteTaskStatsTracker
WriteJobStatsTracker
WriteJobStatsTracker
WriteJobStatsTracker
BasicWriteJobStatsTracker
Logging
Performance Tuning and Debugging
Performance Tuning and Debugging
Performance Tuning
Case Study
Debugging Query Execution
Tools
Spark Thrift Server
Spark Thrift Server
HiveThriftServer2
SparkSQLEnv
SQLExecution
RDDConversions
ShuffledRowRDD
UnsupportedOperationChecker
CheckAnalysis
CatalystTypeConverters
StatFunctions
SubExprUtils
PredicateHelper
DDLUtils
SchemaUtils
AggUtils
CreateStruct
TypeCoercion
TypeCoercionRule
ExtractEquiJoinKeys
ExtractSingleColumnNullAwareAntiJoin
ExtractJoinWithBuckets
PhysicalAggregation
PhysicalOperation
KnownSizeEstimation
SizeEstimator
PartitioningUtils
HadoopFileLinesReader
CatalogUtils
ExternalCatalogUtils
CompressionCodecs
JoinStrategyHint
PushDownUtils
SQLContext
DebugExec Unary Physical Operator
¶
DebugExec
is a
unary physical operator
.
Back to top