Skip to content

The Internals of Spark SQL

SQLHadoopMapReduceCommitProtocol

Initializing search

spark-sql-internals

Home
Internals
New & Noteworthy
High-Level APIs
Demos
Misc

The Internals of Spark SQL

spark-sql-internals

Home
Internals
Internals
- Overview
- Configuration Properties
  Configuration Properties
- DataSource
- Connector API
  Connector API
- DataSource V1 API
  DataSource V1 API
- Catalog Plugin API
  Catalog Plugin API
  - CatalogManager
  - CatalogPlugin
  - CatalogHelper
  - CatalogExtension
  - DelegatingCatalogExtension
  - StagingTableCatalog
  - SupportsNamespaces
  - SupportsCatalogOptions
  - TableCatalog
  - TableChange
  - Utilities
    Utilities
    
    Catalogs
    
    CatalogV2Util
- Query Execution
  Query Execution
- SharedState
- SQL Support
  SQL Support
- Statistics and Hints
  Statistics and Hints
  - Statistics
  - LogicalPlanStats
  - LogicalPlanVisitor
  - SizeInBytesOnlyStatsPlanVisitor
  - BasicStatsPlanVisitor
  - Estimation Utilities
    Estimation Utilities
    
    AggregateEstimation
    
    EstimationUtils
    
    FilterEstimation
    
    JoinEstimation
    
    ProjectEstimation
- Logical Operators
  Logical Operators
- SparkSession Registries
  SparkSession Registries
  - Catalog
    Catalog
    
    Catalog
    
    CatalogImpl
  - ExperimentalMethods
  - ExternalCatalog
    ExternalCatalog
    
    ExternalCatalog
    
    InMemoryCatalog
  - FunctionRegistry
  - GlobalTempViewManager
  - SessionCatalog
    SessionCatalog
    
    SessionCatalog
    
    CatalogTable
    
    CatalogStorageFormat
    
    CatalogTablePartition
    
    BucketSpec
  - V2SessionCatalog
  - SessionState
    SessionState
    
    SessionState
    
    BaseSessionStateBuilder
    
    SessionStateBuilder
  - CacheManager
    CacheManager
    
    CacheManager
    
    CachedRDDBuilder
  - RuntimeConfig
  - UDFRegistration
- Data Sources
  Data Sources
  - Noop
    
    Noop
    
    NoopDataSource
    
    NoopTable
    
    NoopWriteBuilder
    
    NoopBatchWrite
    
    NoopStreamingWrite
  - Files
    Files
    
    FileScan
    
    FileScanBuilder
    
    FileDataSourceV2
    
    HadoopFsRelation
    
    FileIndex
    FileIndex
    
    FileIndex
    
    CatalogFileIndex
    
    InMemoryFileIndex
    
    PartitioningAwareFileIndex
    
    PrunedInMemoryFileIndex
    
    FileFormatWriter
    
    FileWriteBuilder
    
    FileBatchWrite
    
    SQLHadoopMapReduceCommitProtocol
    
    PartitionedFile
    
    RecordReaderIterator
    
    FileFormat
    
    TextBasedFileFormat
  - Kafka
    
    Kafka
    
    Configuration Properties
    
    KafkaSourceProvider
    
    KafkaTable
    
    Options
    
    KafkaScan
    
    KafkaBatch
    
    KafkaBatchWrite
    
    KafkaRelation
    
    KafkaSourceRDD
    KafkaSourceRDD
    
    KafkaSourceRDD
    
    KafkaSourceRDDPartition
    
    ConsumerStrategy
    
    KafkaOffsetReader
    
    KafkaOffsetRangeLimit
    
    KafkaDataConsumer
    
    InternalKafkaConsumer
    
    KafkaWriter
    
    KafkaWriteTask
    
    JsonUtils
    
    InternalKafkaProducerPool
  - Avro
    
    Avro
    
    AvroOptions
    
    AvroFileFormat
    
    CatalystDataToAvro
    
    AvroDataToCatalyst
  - JDBC
    
    JDBC
    
    JDBCOptions
    
    JdbcRelationProvider
    
    JDBCRelation
    
    JDBCScan
    
    JDBCRDD
    
    JdbcDialect
    
    JdbcUtils
  - Console
    
    Console
    
    ConsoleSinkProvider
  - CSV
    CSV
    
    CSVFileFormat
    
    CSVScanBuilder
  - JSON
    JSON
    
    JsonFileFormat
    
    JsonDataSource
  - ORC
    ORC
    
    OrcScanBuilder
    
    OrcFileFormat
  - Parquet
    
    Parquet
    
    ParquetFileFormat
    
    ParquetReadSupport
    
    ParquetScanBuilder
    
    ParquetTable
    
    SpecificParquetRecordReaderBase
    
    VectorizedColumnReader
    
    VectorizedParquetRecordReader
  - Text
    Text
    
    TextFileFormat
- Hive Integration
  Hive Integration
- Catalyst
  Catalyst
- Expressions
  Expressions
- Physical Operators
  Physical Operators
  - SparkPlan
  - ShuffledJoin
  - AliasAwareOutputPartitioning
  - BaseAggregateExec
  - BaseLimitExec
  - BaseSubqueryExec
  - BinaryExecNode
  - CodegenSupport
  - ColumnarBatchScan
  - DataSourceScanExec
  - DataSourceV2ScanExecBase
  - EvalPythonExec
  - Exchange
  - LeafExecNode
  - LimitExec
  - ObjectConsumerExec
  - UnaryExecNode
  - V2CommandExec
  - V2TableWriteExec
  - V1FallbackWriters
  - SQLMetric
  - SQLShuffleWriteMetricsReporter
  - TableWriteExecHelper
  - Distribution and Partitioning
    Distribution and Partitioning
    
    Distribution
    
    Partitioning
    
    Distribution Specifications
    Distribution Specifications
    
    AllTuples
    
    BroadcastDistribution
    
    ClusteredDistribution
    
    HashClusteredDistribution
    
    OrderedDistribution
    
    UnspecifiedDistribution
    
    Broadcast Modes
    Broadcast Modes
    
    BroadcastMode
    
    HashedRelationBroadcastMode
    
    IdentityBroadcastMode
  - HashedRelation
    HashedRelation
    
    HashedRelation
    
    LongHashedRelation
    
    UnsafeHashedRelation
  - HashJoin
  - BaseJoinExec
  - AtomicTableWriteExec
  - BatchWriteHelper
  - SupportsV1Write
  - AppendDataExec
  - AppendDataExecV1
  - AlterTableExec
  - AtomicCreateTableAsSelectExec
  - AtomicReplaceTableAsSelectExec
  - AtomicReplaceTableExec
  - BatchScanExec
  - BlockingOperatorWithCodegen
  - BroadcastExchangeExec
  - BroadcastExchangeLike
  - BroadcastHashJoinExec
  - BroadcastNestedLoopJoinExec
  - CartesianProductExec
  - CoalesceExec
  - CoGroupExec
  - CollectLimitExec
  - CollectMetricsExec
  - ColumnarToRowExec
  - CreateTableAsSelectExec
  - CreateTableExec
  - CustomShuffleReaderExec
  - DataSourceV2ScanExec
  - DataWritingCommandExec Physical Operator
  - DebugExec
  - DeleteFromTableExec
  - DescribeTableExec
  - DeserializeToObjectExec
  - DropNamespaceExec
  - DropTableExec
  - ExecutedCommandExec Physical Operator
  - ExpandExec
  - ExternalRDDScanExec
  - FileSourceScanExec
  - FilterExec Unary Physical Operator
  - GenerateExec Unary Physical Operator
  - HashAggregateExec
  - InMemoryTableScanExec Leaf Physical Operator
  - InputAdapter
  - InputRDDCodegen
  - LocalTableScanExec Leaf Physical Operator
  - MapElementsExec
  - MapPartitionsExec
  - ObjectHashAggregateExec
  - ObjectProducerExec
  - OverwriteByExpressionExec
  - OverwriteByExpressionExecV1
  - OverwritePartitionsDynamicExec
  - ProjectExec
  - RangeExec
  - RDDScanExec
  - RefreshTableExec
  - RenameTableExec
  - ReplaceTableAsSelectExec
  - ReplaceTableExec
  - ReusedExchangeExec
  - RowDataSourceScanExec
  - RowToColumnarExec
  - SampleExec
  - SerializeFromObjectExec Unary Physical Operator
  - SetCatalogAndNamespaceExec
  - ShowCurrentNamespaceExec
  - ShowTablesExec
  - ShowTablePropertiesExec
  - ShuffleExchangeExec
  - ShuffleExchangeLike
  - ShuffledHashJoinExec
  - SortAggregateExec Aggregate Physical Operator
  - SortMergeJoinExec
  - SortExec Unary Physical Operator
  - SubqueryExec
  - SubqueryBroadcastExec
  - TakeOrderedAndProjectExec
  - WindowExec
    WindowExec
    
    WindowExec Unary Physical Operator
    
    AggregateProcessor
    
    WindowFunctionFrame
  - WholeStageCodegenExec Unary Physical Operator
  - WriteToDataSourceV2Exec Physical Operator
- Logical Analysis Rules
  Logical Analysis Rules
- Logical Optimizations
  Logical Optimizations
- Execution Planning Strategies
  Execution Planning Strategies
- Physical Optimizations
  Physical Optimizations
- Tungsten Execution Backend
  Tungsten Execution Backend
- Encoder
  Encoder
New & Noteworthy
New & Noteworthy
- Join Queries
  Join Queries
  - Joins
  - Broadcast Joins
- Adaptive Query Execution
  Adaptive Query Execution
  - LogicalQueryStage
  - InsertAdaptiveSparkPlan
  - AdaptiveSparkPlanExec
  - AdaptiveExecutionContext
  - PlanAdaptiveSubqueries
  - ShuffleStage
  - Adaptive Logical Optimizer
    Adaptive Logical Optimizer
    
    AQEOptimizer
    
    DemoteBroadcastHashJoin
    
    EliminateJoinToEmptyRelation
  - Physical Optimizations
    Physical Optimizations
    
    CoalesceShufflePartitions
    
    CustomShuffleReaderRule
    
    OptimizeSkewedJoin
    
    OptimizeLocalShuffleReader
    
    ReuseAdaptiveSubquery
  - QueryStageExecs
    QueryStageExecs
    
    QueryStageExec
    
    BroadcastQueryStageExec
    
    ShuffleQueryStageExec
  - CostEvaluator
  - SimpleCostEvaluator
  - Demo
- Statistics
- Metadata Columns
- Catalog Plugin API and Multi-Catalog Support
- Dynamic Partition Pruning
- Explaining Query Plans Improved
- Hints
  Hints
- Observable Metrics
- Columnar Processing
- DataSource V2
- Hive Integration
- Vectorized Parquet Decoding
  Vectorized Parquet Decoding
  - Vectorized Parquet Decoding
  - ColumnVectors
    ColumnVectors
    
    ColumnVector
    
    WritableColumnVector
    
    OnHeapColumnVector
    
    OffHeapColumnVector
- Dynamic Partition Inserts
- Bucketing
- Whole-Stage CodeGen
  Whole-Stage CodeGen
  - BufferedRowIterator
  - CodegenContext
  - CodeGenerator
  - Block
  - GenerateUnsafeProjection
  - GenerateMutableProjection
  - GenerateColumnAccessor
  - GenerateOrdering
  - GeneratePredicate
  - GenerateSafeProjection
  - BytesToBytesMap
  - Subexpression Elimination
    Subexpression Elimination
    
    Subexpression Elimination In Code-Generated Expression Evaluation (Common Expression Reuse)
    
    EquivalentExpressions
- Vectorized Query Execution
  Vectorized Query Execution
  - Vectorized Query Execution
  - ColumnarBatch
- Subqueries
- Cost-Based Optimization
  Cost-Based Optimization
- Catalyst DSL
  Catalyst DSL
  - DslLogicalPlan
High-Level APIs
High-Level APIs
Demos
Demos
Misc
Misc
- DataSourceRDD
- DataSourceRDDPartition
- DataWritingSparkTask
- Dataset, DataFrame and RDD
- Dataset and SQL
- DataSourceV2Utils
- Implicits
- Row
- Data Source API
  Data Source API
- Dataset API
  Dataset API
- Column
  Column
- Basic Aggregation
  Basic Aggregation
- Window Aggregation
  Window Aggregation
- Standard Functions
  Standard Functions
- User-Defined Functions
  User-Defined Functions
- Data Types
  Data Types
- Multi-Dimensional Aggregation
- Caching and Persistence
  Caching and Persistence
  - Caching and Persistence
  - User-Friendly Names Of Cached Queries in web UI's Storage Tab
- Checkpointing
- UserDefinedAggregateFunction
- AggregationIterators
  AggregationIterators
- FileScanRDD
- Monitoring
  Monitoring
  - SQLTab
  - SQLListener
  - QueryExecutionListener
  - SQLAppStatusListener
  - SQLAppStatusPlugin
  - SQLAppStatusStore
  - WriteTaskStats
    WriteTaskStats
    
    WriteTaskStats
    
    BasicWriteTaskStats
  - WriteTaskStatsTracker
    WriteTaskStatsTracker
    
    WriteTaskStatsTracker
    
    BasicWriteTaskStatsTracker
  - WriteJobStatsTracker
    WriteJobStatsTracker
    
    WriteJobStatsTracker
    
    BasicWriteJobStatsTracker
  - Logging
- Performance Tuning and Debugging
  Performance Tuning and Debugging
- Tools
- Spark Thrift Server
  Spark Thrift Server
  - HiveThriftServer2
  - SparkSQLEnv
- SQLExecution
- RDDConversions
- ShuffledRowRDD
- UnsupportedOperationChecker
- CheckAnalysis
- CatalystTypeConverters
- StatFunctions
- SubExprUtils
- PredicateHelper
- DDLUtils
- SchemaUtils
- AggUtils
- CreateStruct
- TypeCoercion
- TypeCoercionRule
- ExtractEquiJoinKeys
- ExtractSingleColumnNullAwareAntiJoin
- ExtractJoinWithBuckets
- PhysicalAggregation
- PhysicalOperation
- KnownSizeEstimation
- SizeEstimator
- PartitioningUtils
- HadoopFileLinesReader
- CatalogUtils
- ExternalCatalogUtils
- CompressionCodecs
- JoinStrategyHint
- PushDownUtils
- SQLContext

SQLHadoopMapReduceCommitProtocol¶

SQLHadoopMapReduceCommitProtocol is...FIXME

Last update: 2020-10-21

Previous FileBatchWrite

Next PartitionedFile

Copyright © 2021 Jacek Laskowski

Made with Material for MkDocs Insiders