Skip to content

The Internals of Spark SQL

SQLHadoopMapReduceCommitProtocol

Initializing search

spark-sql-internals

Home
Internals
New & Noteworthy
High-Level APIs
PySpark
Demos
Extras

The Internals of Spark SQL

spark-sql-internals

Home
Home
- Welcome
Internals
Internals
- DataSource
- Connector API
  Connector API
- Data Source API
  Data Source API
- Catalog Plugin API
  Catalog Plugin API
  - CatalogManager
  - CatalogPlugin
  - CatalogHelper
  - CatalogExtension
  - DelegatingCatalogExtension
  - StagingTableCatalog
  - SupportsNamespaces
  - SupportsCatalogOptions
  - TableCatalog
  - Utilities
    Utilities
    
    Catalogs
    
    CatalogV2Util
- Structured Query Execution
  Structured Query Execution
- SparkSession
  SparkSession
- Configuration Properties
- SQL Support
  SQL Support
- Logical Operators
  Logical Operators
- SparkSession Registries
  SparkSession Registries
  - Catalog
    Catalog
    
    Catalog
    
    CatalogImpl
  - ExecutionListenerManager
  - ExperimentalMethods
  - ExternalCatalog
    ExternalCatalog
    
    ExternalCatalog
    
    InMemoryCatalog
  - FunctionRegistry
  - GlobalTempViewManager
  - SessionCatalog
    SessionCatalog
    
    SessionCatalog
    
    CatalogTable
    
    CatalogStorageFormat
    
    CatalogTablePartition
    
    BucketSpec
  - V2SessionCatalog
  - SessionState
    SessionState
    
    SessionState
    
    BaseSessionStateBuilder
    
    SessionStateBuilder
  - CacheManager
    CacheManager
    
    CacheManager
    
    CachedRDDBuilder
  - RuntimeConfig
  - SQLConf
    SQLConf
    
    SQLConf
    
    StaticSQLConf
    
    CatalystConf
  - UDFRegistration
- Data Sources
  Data Sources
  - Noop
    Noop
    
    Noop Data Source
    
    NoopDataSource
    
    NoopTable
    
    NoopWriteBuilder
    
    NoopBatchWrite
    
    NoopStreamingWrite
  - Files
    Files
    
    FileScan
    
    FileScanBuilder
    
    FileDataSourceV2
    
    HadoopFsRelation
    
    FileIndex
    FileIndex
    
    FileIndex
    
    CatalogFileIndex
    
    InMemoryFileIndex
    
    PartitioningAwareFileIndex
    
    PrunedInMemoryFileIndex
    
    FileFormatWriter
    
    FileWriteBuilder
    
    FileBatchWrite
    
    SQLHadoopMapReduceCommitProtocol
    
    PartitionedFile
    
    RecordReaderIterator
    
    FileFormat
    
    TextBasedFileFormat
  - Kafka
    Kafka
    
    Kafka Data Source
    
    Configuration Properties
    
    KafkaSourceProvider
    
    KafkaTable
    
    Options
    
    KafkaScan
    
    KafkaBatch
    
    KafkaBatchWrite
    
    KafkaRelation
    
    KafkaSourceRDD
    KafkaSourceRDD
    
    KafkaSourceRDD
    
    KafkaSourceRDDPartition
    
    ConsumerStrategy
    
    KafkaOffsetReader
    
    KafkaOffsetRangeLimit
    
    KafkaDataConsumer
    
    InternalKafkaConsumer
    
    KafkaWriter
    
    KafkaWriteTask
    
    JsonUtils
    
    InternalKafkaProducerPool
  - Avro
    Avro
    
    Avro Data Source
    
    AvroOptions
    
    AvroFileFormat
    
    CatalystDataToAvro
    
    AvroDataToCatalyst
  - JDBC
    JDBC
    
    JDBC Data Source
    
    JDBCOptions
    
    JdbcRelationProvider
    
    JDBCRelation
    
    JDBCRDD
    
    JdbcDialect
    
    JdbcUtils
  - Console
    Console
    
    Console Data Source
    
    ConsoleSinkProvider
  - CSV
    CSV
    
    CSVFileFormat
    
    CSVScanBuilder
  - JSON
    JSON
    
    JsonFileFormat
    
    JsonDataSource
  - ORC
    ORC
    
    OrcScanBuilder
    
    OrcFileFormat
  - Parquet
    Parquet
    
    ParquetTable
    
    ParquetScanBuilder
    
    ParquetFileFormat
    
    ParquetReadSupport
    
    VectorizedParquetRecordReader
    
    VectorizedColumnReader
    
    SpecificParquetRecordReaderBase
  - Text
    Text
    
    TextFileFormat
- Hive Integration
  Hive Integration
- Catalyst
  Catalyst
- Catalyst Expressions
  Catalyst Expressions
  - Expression
  - ExecSubqueryExpression
  - PlanExpression
  - Unevaluable
  - Projection
  - UnsafeProjection
  - InterpretedProjection
  - CodeGeneratorWithInterpretedFallback
  - Concrete Expressions
    Concrete Expressions
    
    AggregateExpression
    
    AggregateFunction
    
    AggregateWindowFunction
    
    AttributeReference
    
    Alias
    
    Attribute
    
    BoundReference
    
    CallMethodViaReflection
    
    Coalesce
    
    CodegenFallback
    
    CollectionGenerator
    
    ComplexTypedAggregateExpression
    
    CreateArray
    
    CreateNamedStruct
    
    CreateNamedStructLike
    
    CreateNamedStructUnsafe
    
    CumeDist
    
    DeclarativeAggregate
    
    DynamicPruningExpression
    
    DynamicPruningSubquery
    
    Exists
    
    ExpectsInputTypes
    
    ExplodeBase
    
    First
    
    Generator
    
    GetArrayStructFields
    
    GetArrayItem
    
    GetMapValue
    
    GetStructField
    
    HashPartitioning
    
    Higher-Order Functions
    Higher-Order Functions
    
    HigherOrderFunction
    
    ArrayBasedSimpleHigherOrderFunction
    
    MapBasedSimpleHigherOrderFunction
    
    SimpleHigherOrderFunction
    
    ImperativeAggregate
    
    In
    
    Inline
    
    InSet
    
    InSubquery
    
    InSubqueryExec
    
    JsonToStructs
    
    JsonTuple
    
    ListQuery
    
    Literal
    
    MonotonicallyIncreasingID
    
    Murmur3Hash
    
    NamedExpression
    
    Nondeterministic
    
    OffsetWindowFunction
    
    ParseToDate
    
    ParseToTimestamp
    
    PrettyAttribute
    
    RangePartitioning
    
    RankLike
    
    ResolvedStar
    
    RowNumberLike
    
    RuntimeReplaceable
    
    ScalarSubquery
    
    ScalarSubquery (ExecSubqueryExpression) Expression
    
    ScalaUDF
    
    ScalaUDAF
    
    SimpleTypedAggregateExpression
    
    SizeBasedWindowFunction
    
    SortOrder
    
    Stack
    
    Star
    
    StaticInvoke
    
    SubqueryExpression
    
    TimeWindow
    
    TypedAggregateExpression
    
    TypedImperativeAggregate
    
    UnaryExpression
    
    UnixTimestamp
    
    UnresolvedAttribute
    
    UnresolvedFunction
    
    UnresolvedGenerator
    
    UnresolvedOrdinal
    
    UnresolvedRegex
    
    UnresolvedStar
    
    UnresolvedWindowExpression
    
    WindowExpression
    
    WindowFunction
    
    WindowSpecDefinition
- Physical Operators
  Physical Operators
  - SparkPlan
  - AliasAwareOutputPartitioning
  - BaseAggregateExec
  - BaseLimitExec
  - BaseSubqueryExec
  - BinaryExecNode
  - CodegenSupport
  - ColumnarBatchScan
  - DataSourceScanExec
  - DataSourceV2ScanExecBase
  - EvalPythonExec
  - Exchange
  - LeafExecNode
  - LimitExec
  - ObjectConsumerExec
  - QueryStageExec
  - UnaryExecNode
  - V2CommandExec
  - V2TableWriteExec
  - V1FallbackWriters
  - SQLMetric
  - SQLShuffleWriteMetricsReporter
  - Distribution and Partitioning
    Distribution and Partitioning
    
    Distribution
    
    Partitioning
    
    Distribution Specifications
    Distribution Specifications
    
    AllTuples
    
    BroadcastDistribution
    
    ClusteredDistribution
    
    HashClusteredDistribution
    
    OrderedDistribution
    
    UnspecifiedDistribution
    
    Broadcast Modes
    Broadcast Modes
    
    BroadcastMode
    
    HashedRelationBroadcastMode
    
    IdentityBroadcastMode
  - HashedRelation
    HashedRelation
    
    HashedRelation
    
    LongHashedRelation
    
    UnsafeHashedRelation
  - HashJoin
  - AtomicTableWriteExec
  - BatchWriteHelper
  - SupportsV1Write
  - Concrete Operators
    Concrete Operators
    
    AdaptiveSparkPlanExec
    
    AppendDataExec
    
    AppendDataExecV1
    
    AlterTableExec
    
    AtomicCreateTableAsSelectExec
    
    AtomicReplaceTableAsSelectExec
    
    AtomicReplaceTableExec
    
    BatchScanExec
    
    BlockingOperatorWithCodegen
    
    BroadcastExchangeExec Unary Physical Operator for Broadcast Joins
    
    BroadcastHashJoinExec Binary Physical Operator
    
    BroadcastNestedLoopJoinExec
    
    BroadcastQueryStageExec
    
    CartesianProductExec
    
    CoalesceExec
    
    CoGroupExec
    
    CollectLimitExec
    
    CollectMetricsExec
    
    ColumnarToRowExec
    
    CreateTableAsSelectExec
    
    CreateTableExec
    
    CustomShuffleReaderExec
    
    DataSourceV2ScanExec
    
    DataWritingCommandExec Physical Operator
    
    DebugExec
    
    DescribeTableExec
    
    DeserializeToObjectExec
    
    DropNamespaceExec
    
    DropTableExec
    
    ExecutedCommandExec Physical Operator
    
    ExpandExec
    
    ExternalRDDScanExec
    
    FileSourceScanExec
    
    FilterExec Unary Physical Operator
    
    GenerateExec Unary Physical Operator
    
    HashAggregateExec
    
    InMemoryTableScanExec Leaf Physical Operator
    
    InputAdapter
    
    InputRDDCodegen
    
    LocalTableScanExec Leaf Physical Operator
    
    MapElementsExec
    
    ObjectHashAggregateExec
    
    ObjectProducerExec
    
    OverwriteByExpressionExec
    
    OverwriteByExpressionExecV1
    
    OverwritePartitionsDynamicExec
    
    ProjectExec Unary Physical Operator
    
    RangeExec
    
    RDDScanExec
    
    RefreshTableExec
    
    RenameTableExec
    
    ReplaceTableAsSelectExec
    
    ReplaceTableExec
    
    ReusedExchangeExec
    
    RowDataSourceScanExec Leaf Physical Operator
    
    RowToColumnarExec
    
    SampleExec
    
    SerializeFromObjectExec Unary Physical Operator
    
    SetCatalogAndNamespaceExec
    
    ShowCurrentNamespaceExec
    
    ShowTablesExec
    
    ShuffleExchangeExec
    
    ShuffledHashJoinExec Binary Physical Operator for Shuffled Hash Join
    
    ShuffleQueryStageExec
    
    SortAggregateExec Aggregate Physical Operator
    
    SortMergeJoinExec
    
    SortExec Unary Physical Operator
    
    SubqueryExec
    
    SubqueryBroadcastExec
    
    TakeOrderedAndProjectExec
    
    WindowExec
    WindowExec
    
    WindowExec Unary Physical Operator
    
    AggregateProcessor
    
    WindowFunctionFrame
    
    WholeStageCodegenExec Unary Physical Operator
    
    WriteToDataSourceV2Exec Physical Operator
- Logical Analysis Rules
  Logical Analysis Rules
- Logical Optimizations
  Logical Optimizations
- Execution Planning Strategies
  Execution Planning Strategies
- Physical Optimizations
  Physical Optimizations
- Project Tungsten
  Project Tungsten
  - Tungsten Execution Backend
  - InternalRow
  - UnsafeRow
  - AggregationIterator
    AggregationIterator
    
    AggregationIterator
    
    ObjectAggregationIterator
    
    SortBasedAggregationIterator
    
    TungstenAggregationIterator
  - CatalystSerde
  - ExternalAppendOnlyUnsafeRowArray
  - UnsafeFixedWidthAggregationMap
- DataSourceRDD
- DataSourceRDDPartition
- DataWritingSparkTask
New & Noteworthy
New & Noteworthy
- Adaptive Query Execution
- Catalog Plugin API and Multi-Catalog Support
- Dynamic Partition Pruning
- Explaining Query Plans Improved
- Hint Framework
- Join Strategy Hints
- Observable Metrics
- Columnar Processing
- DataSource V2
- Hive Integration
- Vectorized Parquet Decoding
  Vectorized Parquet Decoding
  - Vectorized Parquet Decoding
  - ColumnVectors
    ColumnVectors
    
    ColumnVector
    
    WritableColumnVector
    
    OnHeapColumnVector
    
    OffHeapColumnVector
- Dynamic Partition Inserts
- Bucketing
- Whole-Stage CodeGen
  Whole-Stage CodeGen
  - Whole-Stage CodeGen
  - BufferedRowIterator
  - CodegenContext
  - CodeGenerator
  - Block
  - GenerateUnsafeProjection
  - GenerateMutableProjection
  - GenerateColumnAccessor
  - GenerateOrdering
  - GeneratePredicate
  - GenerateSafeProjection
  - BytesToBytesMap
  - Subexpression Elimination
    Subexpression Elimination
    
    Subexpression Elimination In Code-Generated Expression Evaluation (Common Expression Reuse)
    
    EquivalentExpressions
- Vectorized Query Execution
  Vectorized Query Execution
  - Vectorized Query Execution
  - ColumnarBatch
- Subqueries
- Cost-Based Optimization
  Cost-Based Optimization
- Catalyst DSL
  Catalyst DSL
  - Catalyst DSL
  - DslLogicalPlan
High-Level APIs
High-Level APIs
PySpark
PySpark
- Setup
- Configuration Properties
- PythonWorkerFactory
- MonitorThread
- PythonUDF
- PythonFunction
- AggregateInPandasExec
- PythonRDD
- PythonForeachWriter
- Python Runners
  Python Runners
Demos
Demos
Extras
Extras
- Spark SQL
- Dataset, DataFrame and RDD
- Dataset and SQL
- DataSourceV2Utils
- Developing Applications
- Encoder
  Encoder
- DataFrame
- Row
- Data Source API
  Data Source API
- Dataset API
  Dataset API
- Column
  Column
- Basic Aggregation
  Basic Aggregation
- Joins
  Joins
  - Joins
  - Broadcast Joins
- Window Aggregation
  Window Aggregation
- Standard Functions
  Standard Functions
- User-Defined Functions
  User-Defined Functions
- Schema
  Schema
- Multi-Dimensional Aggregation
- Caching and Persistence
  Caching and Persistence
  - Caching and Persistence
  - User-Friendly Names Of Cached Queries in web UI's Storage Tab
- Checkpointing
- UserDefinedAggregateFunction
- Aggregator
- FileScanRDD
- Monitoring
  Monitoring
  - SQLTab
  - SQLListener
  - QueryExecutionListener
  - SQLAppStatusListener
  - SQLAppStatusPlugin
  - SQLAppStatusStore
  - WriteTaskStats
    WriteTaskStats
    
    WriteTaskStats
    
    BasicWriteTaskStats
  - WriteTaskStatsTracker
    WriteTaskStatsTracker
    
    WriteTaskStatsTracker
    
    BasicWriteTaskStatsTracker
  - WriteJobStatsTracker
    WriteJobStatsTracker
    
    WriteJobStatsTracker
    
    BasicWriteJobStatsTracker
  - Logging
- Performance Tuning and Debugging
  Performance Tuning and Debugging
- Tools
- Spark Thrift Server
  Spark Thrift Server
  - HiveThriftServer2
  - SparkSQLEnv
- SQLExecution
- RDDConversions
- ShuffledRowRDD
- UnsupportedOperationChecker
- CheckAnalysis
- CatalystTypeConverters
- StatFunctions
- SubExprUtils
- PredicateHelper
- DDLUtils
- SchemaUtils
- AggUtils
- ScalaReflection
- CreateStruct
- MultiInstanceRelation
- TypeCoercion
- TypeCoercionRule
- ExtractEquiJoinKeys
- PhysicalAggregation
- PhysicalOperation
- KnownSizeEstimation
- SizeEstimator
- PartitioningUtils
- HadoopFileLinesReader
- CatalogUtils
- ExternalCatalogUtils
- CompressionCodecs
- JoinStrategyHint
- PushDownUtils
- SQLContext

SQLHadoopMapReduceCommitProtocol¶

SQLHadoopMapReduceCommitProtocol is...FIXME

Last update: 2020-10-21

Previous FileBatchWrite

Next PartitionedFile

Copyright © 2021 Jacek Laskowski

Made with Material for MkDocs Insiders