January 16, 2026: Weekly Status Update in Gluten

[GlutenPerfBot]

This weekly update is generated by LLMs. You're welcome to join our Github for in-depth discussions.

Overall Activity Summary

The Gluten community has been intensely focused on Spark version compatibility and infrastructure improvements. This week saw significant progress on Spark 4.1 support, daily Velox version updates, and cleanup of deprecated Spark 3.2 code. The project merged 287k+ lines of Spark 4.1 unit tests and continued aggressive deprecation of Spark 3.2 support.

Key Ongoing Projects

• Spark 4.1 Support Initiative - The team delivered comprehensive Spark 4.1 compatibility:

  • [GLUTEN-11343][CORE][VL] Support Spark 4.1 UT #11353 by @baibaichen added 287k+ lines of Spark 4.1 unit tests with extensive test exclusions for later fixes
  • [GLUTEN-11346][CORE][VL] Add Spark 4.1 Shim Layer #11347 by @baibaichen introduced the Spark 4.1 shim layer with 62 files of compatibility changes
  • [GLUTEN-11355][UT] Add new Spark 4.1 tests #11380 by @baibaichen continues expanding Spark 4.1 test coverage (currently in draft)

• Spark 3.2 Deprecation - Major cleanup effort led by @QCLyu:

  • [CORE] Deprecate and remove Spark 3.2 support #11351 removes Spark 3.2 support comprehensively from source code, build profiles, CI pipelines, and documentation
  • Remove remaining Spark 3.2-specific compatibility code #11379 tracks remaining Spark 3.2 compatibility code that needs removal

• Infrastructure & Performance:

  • Daily Velox version updates continue ([GLUTEN-6887][VL] Daily Update Velox Version (2026_01_09) #11385, [GLUTEN-6887][VL] Daily Update Velox Version (2026_01_08) #11378, [GLUTEN-6887][VL] Daily Update Velox Version (2026_01_07) #11375, [GLUTEN-6887][VL] Daily Update Velox Version (2026_01_06) #11366, [GLUTEN-6887][VL] Daily Update Velox Version (2026_01_05) #11356) by @GlutenPerfBot
  • [GLUTEN-9619][VL] Add HDFS integration tests to gluten-it #11373 by @infvg adds HDFS integration tests to improve storage backend coverage
  • WIP: add bolt backend in gluten #11261 by @WangGuangxin introduces experimental Bolt backend (269k+ lines, 1055 files)

Priority Items

Critical Bug Fixes Needed:

• OOM due to Hash based shuffle fallback #11421 by @FelixYBW - OOM due to Hash based shuffle during Delta Lake table creation
• [VL] docker image build failed due to maven download #11432 by @zhouyuan - Docker image build failed due to Maven download timeout
• [CH] Build clickhouse failed with clang-17 #11427 by @luomh1998 - ClickHouse build fails with Clang 17 compatibility issues
• [Qualification]Operator Count Inflation in UnsupportedOperators Report #11394 by @xumanbu - Operator count inflation in qualification tool reports

Performance Issues:

• [VL][BUG] Exception in evaluating deprecated dataset sum #11403 by @Surbhi-Vijay - Exception in evaluating deprecated dataset sum operations
• [VL][BUG] Incorrect decimal casting from floating point value #11402 by @Surbhi-Vijay - Incorrect decimal casting from floating point values

Notable Discussions

• Gluten Slack Channel #incubator-gluten #8429: Gluten Slack Channel setup on ASF workspace for community communication
• January 09, 2026: Weekly Status Update in Gluten #11388: Weekly status updates showing systematic tracking of project progress
• Gluten 2025 Roadmap #8226: 2025 Roadmap discussion covering Spark 4.0 support, upstream Velox adoption, and TLP submission

Emerging Trends

• Spark Version Acceleration - Rapid support for Spark 4.1 following 4.0 pattern, suggesting annual Spark version support cycle
• Backend Diversification - Bolt backend introduction alongside Velox/ClickHouse indicates multi-backend strategy
• Memory Management Focus - Multiple memory-related issues and optimizations suggest this is a critical adoption blocker
• Test Infrastructure Maturation - Systematic daily Velox updates and comprehensive test suite additions show production readiness focus

Good First Issues

• [VL] Add ANSI mode support #10134: Add ANSI mode support - Well-documented issue with clear task breakdown for implementing Spark ANSI compliance across type casting, arithmetic functions, and date/time operations
• [VL] Enhance the qualification tool to calculate scan on lakehouse formats #11417: Enhance qualification tool for lakehouse formats - Update qualification tool to properly report scan benefits for Iceberg/Delta/Hudi/Paimon formats
• [VL] Track on Spark-4.1.x failed unit tests #11400: Track Spark 4.1.x failed unit tests - Systematic tracking and fixing of failing tests with clear categorization of issues