4.1.1 Release Notes

[yiguolei]

New Features

Storage & Compaction

• Add adaptive batch size for scan path ([feature](be) Add adaptive batch size for scan path #62835)
• Add CompactionTaskTracker with system table and HTTP API ([Feature](compaction) add CompactionTaskTracker with system table and HTTP API #61696)
• Add value predicate pushdown control for MOR tables ([feature](scan) Add value predicate pushdown control for MOR tables #60513)

Other

• Add json_object_flatten scalar function ([feature](jsonb) add json_object_flatten scalar function #62825)

Improvement

Storage & Compaction

• Skip zero-length files in FileSplitter to avoid sending empty splits to BE ([improvement](fe) Skip zero-length files in FileSplitter to avoid sending empty splits to BE #62482)
• Avoid single-point read/write during sequentially reading key ([improvement](recycler) Avoid single-point read/write during sequentially reading key #62476)
• Add stampede protection for AnnIndexIVFListCache ([improvement](be) Add stampede protection for AnnIndexIVFListCache #62442)
• Limit packed file writes to rowset first segment ([improvement](be) Limit packed file writes to rowset first segment #62342)
• Unify current query runtime statistics and expose task progress ([improvement](executor) unify current query runtime statistics and expose task progress #60567)

Load

• Make thirdparty jar download mirror configurable ([improvement](docker) Make thirdparty jar download mirror configurable #62376)

Cloud Native

• Add HashCRC32Return32 and make join_hash_table use 32-bit hash value ([Improvement](join) add HashCRC32Return32 and make join_hash_table use 32-bit hash value #62512)

Other

• Add current_database builtin alias ([improvement](fe) Add current_database builtin alias #62591)

Bugfix

Index & Search

• Fix deep nested complex type subtype validation bypass ([fix](fe) Fix deep nested complex type subtype validation bypass #63208)
• Fix potential NPE in getDeleteBitmapUpdateLock ([fix](fe) fix potential NPE in getDeleteBitmapUpdateLock #62809)
• Fix ANN query vector extraction to handle all constant expression types ([fix](be) Fix ANN query vector extraction to handle all constant expression types #62637)
• Split monolithic function_string.h into domain-specific files ([refine](function) split monolithic function_string.h into domain-specific files #62262)
• Fix OOB crash in partition column generation for Iceberg/Paimon tables ([fix](scan) Fix OOB crash in partition column generation for Iceberg/Paimon tables #62177)
• Fix null pointer exception in sessionVariables after upgrade ([fix](fe) Fix null pointer exception in sessionVariables after upgrade #61959)
• Optimize the compilation time of aggregate_function_orthogonal_bitmap.cpp ([env](compiler)Optimize the compilation time of aggregate_function_orthogonal_bitmap.cpp #61606)
• Drop index should set db_id ([fix](snapshot) drop index should set db_id #58401)

Query & Execution

• Fix remote Flight SQL result receiver initialization ([fix](fe) Fix remote Flight SQL result receiver initialization #63136)
• Keep LogicalOlapScan for short-circuit point query on empty table ([fix](point query) Keep LogicalOlapScan for short-circuit point query on empty table #62948)
• Fix DST spring-forward gap handling in timestamptz literal ([fix](nereids) Fix DST spring-forward gap handling in timestamptz literal #62945)
• Fix UB and param order in assert_num_rows_operator ([fix](be) Fix UB and param order in assert_num_rows_operator #62800)
• Preserve TIMESTAMPTZ in LEAD/LAG by adding TimeStampTzType signature matching ([fix](functions)Preserve TIMESTAMPTZ in LEAD/LAG by adding TimeStampTzType signature matching #62779)
• Fix exchange receiver dependency race ([fix](be) Fix exchange receiver dependency race #62777)
• Add fast path for canonical format datetime parse ([Opt](parse) Add fast path for canonical format datetime parse #62757)
• Fix isCountStar incorrectly treating count(null) as count(*) ([fix](fe) Fix isCountStar incorrectly treating count(null) as count(*) #62548)
• Fix view columns losing colUniqueId in lazy materialization ([fix](fe) Fix view columns losing colUniqueId in lazy materialization #62533)
• Align extra-join elimination safety check ([fix](mv) Align extra-join elimination safety check #62527)
• Fix Doris query service fails after the file handles on the BE node are used up ([fix](Reliability)Fix Doris query service fails after the file handles on the BE node are used up #62393)
• Scale num_nulls in col stats when partition pruned ([opt](nereids) scale num_nulls in col stats when partition pruned #62265)
• Fix prepared statement QPS metrics not counted when audit log disabled ([fix](metrics) Fix prepared statement QPS metrics not counted when audit log disabled #61621)
• Update license-maven-plugin to 2.1.0 ([fix](pipeline) update license-maven-plugin to 2.1.0 #58951)

Storage & Compaction

• Avoid nested task attach in cloud snapshot manager ([fix](memory-tracker) Avoid nested task attach in cloud snapshot manager #63189)
• Materialize NG compaction regular paths ([fix](variant) materialize NG compaction regular paths #63104)
• Rebuild broker load storage properties after Gson replay ([fix](load) rebuild broker load storage properties after Gson replay #63094)
• Skip statistics cache for system dbs ([fix](statistics) Skip statistics cache for system dbs #63089)
• Fix variant flat-leaf root read plan ([fix](variant) Fix variant flat-leaf root read plan #63086)
• Resolve variant sub-column indexes for score() ([fix](inverted index) resolve variant sub-column indexes for score() #62992)
• Avoid deleting nonexistent delete bitmap files ([fix](cloud) Avoid deleting nonexistent delete bitmap files #62967)
• Skip skewed warmup rowset latency samples ([fix](cloud) Skip skewed warmup rowset latency samples #62941)
• Return unknown stats for system tables ([fix](fe) Return unknown stats for system tables #62913)
• Mark internal query failures as ERR in audit log ([fix](audit) Mark internal query failures as ERR in audit log #62908)
• Fix warm up don't retry due to error message change ([Fix](warm up) Fix warm up don't retry due to error message change #62886)
• Reject oversized block size in add_cell ([fix](filecache) reject oversized block size in add_cell #62878)
• Move #include directives outside namespace blocks to avoid ODR violations ([fix](be) Move #include directives outside namespace blocks to avoid ODR violations #62871)
• Bound cdc_client RPCs with per-category timeouts ([fix](streaming-job) bound cdc_client RPCs with per-category timeouts #62870)
• Add NestedGroup path metadata support ([Refactor](Variant) add NestedGroup path metadata support #62848)
• Reduce point-query network overhead by resending query context on cache miss ([opt](point query) reduce point-query network overhead by resending query context on cache miss #62836)
• Skip full footer scan when constructing VariantStatsCaculator ([fix](variant) Skip full footer scan when constructing VariantStatsCaculator #62819)
• Avoid NPE and clear stale cache on warmup job cancel/expire ([fix](cloud) avoid NPE and clear stale cache on warmup job cancel/expire #62805)
• Hold table write lock across first-time dynamic partition setup to prevent CREATE MV race ((cloud) Hold table write lock across first-time dynamic partition setup to prevent CREATE MV race #62755)
• Support specifying compute_group for StreamingJob ([improve](streaming-job) support specifying compute_group for StreamingJob #62747)
• Attach MemTracker to cache background threads to fix orphan crash ([fix](memory) attach MemTracker to cache background threads to fix orphan crash #62739)
• Treat empty cancel alter job list as all rollup jobs ([fix](sync) Treat empty cancel alter job list as all rollup jobs #62712)
• Remove enable_nereids_load switch ([refactor](fe) Remove enable_nereids_load switch #62703)
• Add some Sanity-check for avoid npe ([Chore](check) add some Sanity-check for avoid npe #62691)
• RestoreJob cannot deserialize Tablet when upgrade ([cloud](fe) RestoreJob cannot deserialize Tablet when upgrade #62673)
• The profile hitcache not correct in force refresh and sink id not correct ([Profile][exec] The profile hitcache not correct in force refresh and sink id not correct #62645)
• Aggregate non-MOW segment key bounds to reduce rowset meta size ([opt](rowset) Aggregate non-MOW segment key bounds to reduce rowset meta size #62604)
• Support ADMIN COMPACT TABLE type='FULL' and enable it in cloud mode ([feat](compaction) Support ADMIN COMPACT TABLE type='FULL' and enable it in cloud mode #62596)
• Refine revocable memory accounting for spill ([fix](be) Refine revocable memory accounting for spill #62581)
• Fix compaction failure on no-key table with variant column uid=0 ([fix](variant) Fix compaction failure on no-key table with variant column uid=0 #62571)
• Propagate ALTER source/target properties to runtime and persistence ([fix](streaming-job) propagate ALTER source/target properties to runtime and persistence #62553)
• Avoid per-row Field temporaries in TypedZoneMapIndexWriter ([fix](zone-map) Avoid per-row Field temporaries in TypedZoneMapIndexWriter #62544)
• Fix incorrect for-loop ([fix](column) Fix incorrect for-loop #62517)
• Normalize legacy single-part dot-key subcolumn paths on read ([fix](variant) normalize legacy single-part dot-key subcolumn paths on read #62409)
• Avoid BE crash when finalize misses local cache writer ([fix](filecache) avoid BE crash when finalize misses local cache writer #62389)
• Fix InsertLoadJob memory leak caused by jobs permanently stuck in PENDING state ([fix](insert) fix InsertLoadJob memory leak caused by jobs permanently stuck in PENDING state #62282)
• Add SC_COMPACTION_CONFLICT error code to retry cross-V1 compaction failures ([fix](cloud) Add SC_COMPACTION_CONFLICT error code to retry cross-V1 compaction failures #62272)
• Add per-job metrics for streaming insert jobs ([improve](streaming-job) Add per-job metrics for streaming insert jobs #62224)
• Reduce compile time of aggregate_function_reader_replace.cpp ([env](compiler) Reduce compile time of aggregate_function_reader_replace.cpp #62047)
• Reduce HybridSet/InListPredicate template bloat and fix create_string_value_set bug ([refine](predicate) reduce HybridSet/InListPredicate template bloat and fix create_string_value_set bug #61999)
• Deprecate TYPE_LAMBDA_FUNCTION in PrimitiveType enum ([refine](be) Deprecate TYPE_LAMBDA_FUNCTION in PrimitiveType enum #61941)
• Add get_version and get_tablet_stats case ([fix](cloud) add get_version and get_tablet_stats case #61915)
• Migrate old VExpr execute interface to new execute_column (Part 1) ([refine](expr) Migrate old VExpr execute interface to new execute_column (Part 1) #61912)
• Reject upload snapshots on broken storage path ([fix](backup) reject upload snapshots on broken storage path #61251)
• Submit manual full compaction task to thread pool instead of detached thread ([refactor](compaction) submit manual full compaction task to thread pool instead of detached thread #61222)
• Add fe-foundation module with zero-dependency JDK-only utilities ([refactor](fe) Add fe-foundation module with zero-dependency JDK-only utilities #61175)
• Add show_segment_data operation to meta_tool ((tools) Add show_segment_data operation to meta_tool #60608)
• Avoid SIGSEGV in background LRU update when clear cache ([fix](filecache) avoid SIGSEGV in background LRU update when clear cache #60533)
• Fix IOContext Use-After-Free ([fix](storage) fix IOContext Use-After-Free #59947)
• Prioritize scheduling the most recently active tablets in cloud ([opt](cloud) Prioritize scheduling the most recently active tablets in cloud #59539)
• Support config hot reload of file cache microbench in running state ([opt](filecache) support config hot reload of file cache microbench in running state #58922)
• Skip agg delete bitmap v1 to ms when v2 ([fix](mow) skip agg delete bitmap v1 to ms when v2 #57990)
• Check delete bitmap version ([fix](mow) check delete bitmap version #57989)
• Exposes cloud balance metrics ([opt](cloud) Exposes cloud balance metrics  #57200)

Load

• Start counting task max interval after the first record is received ([fix](streaming-job) start counting task max interval after the first record is received #63141)
• Load JNI log4j2 properties config ([fix](be) Load JNI log4j2 properties config #63063)
• Fix broker load silently loaded only the first file when parsing multiple files path ([fix](load) fix broker load silently loaded only the first file when parsing multiple files path #62969)
• Poll packed file async close without blocking ([fix](be) Poll packed file async close without blocking #62938)
• Avoid copying ANN search results ([refactor](ann) Avoid copying ANN search results #62924)
• Fix filteredRows always 0 on single-table S3 streaming ([fix](streaming-job) fix filteredRows always 0 on single-table S3 streaming #62816)
• Report physical file count in LoadStatistic.FileNumber ([fix](insert) Report physical file count in LoadStatistic.FileNumber #62804)
• Use CPU metrics delta for CPU pressure detection in adaptive flush controller ([fix](memtable) use CPU metrics delta for CPU pressure detection in adaptive flush controller #62744)
• Reject silent-no-op ALTER keys and unsupported load.* properties ([fix](streaming-job) reject silent-no-op ALTER keys and unsupported load.* properties #62680)
• Roll sample window before accumulating in checkDataQuality ([fix](streaming-job) roll sample window before accumulating in checkDataQuality #62636)
• Use per-table publication instead of ALL TABLES for PostgreSQL CDC ([Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC #62526)
• Support specifying offset for StreamingInsertJob create and alter ([Improve](streaming-job) support specifying offset for StreamingInsertJob create and alter #62490)
• Fix S3 offset and job statistics lost after FE checkpoint restart ([fix](streaming-job) fix S3 offset and job statistics lost after FE checkpoint restart #62449)
• Fix NPE in StreamingInsertJob.replayOnCommitted during EditLog replay ([fix](streaming-job) Fix NPE in StreamingInsertJob.replayOnCommitted during EditLog replay #62416)
• Cap auto-resume attempts and expose structured FailureReason ([improve](streaming-job) Cap auto-resume attempts and expose structured FailureReason #62345)
• Fix INSERT job statistics lost in show load after FE restart ([fix](insert) fix INSERT job statistics lost in show load after FE restart #62331)
• Fix invalid String.format pattern in NereidsStreamLoadTask ([fix](load): fix invalid String.format pattern in NereidsStreamLoadTask #62225)
• Update flink cdc version for streaming job ([Improve](job) update flink cdc version for streaming job #62212)
• Return early for non-master stream load precommit ([fix](fe) Return early for non-master stream load precommit #62109)
• Validate stream load content length before group commit ([fix](be) Validate stream load content length before group commit #62110)
• Mask sensitive headers in stream load logs ([fix](fe) Mask sensitive headers in stream load logs #62108)
• Reject invalid stream load tokens on commit and rollback ([fix](fe) Reject invalid stream load tokens on commit and rollback #62111)
• Using memtable memory instead of load memory to calculate adaptive write buffer size ([fix](memtable) using memtable memory instead of load memory to calculate adaptive write buffer size #62104)
• Fix GenericPool reopen blocking 18min on stale TLS connections ([fix](rpc) fix GenericPool reopen blocking 18min on stale TLS connections #61951)
• Fix IllegalMonitorStateException in routine load afterAborted when coordinate BE restarts ([fix](transaction) fix IllegalMonitorStateException in routine load afterAborted when coordinate BE restarts #61881)
• Reduce template instantiation in scan_operator to improve compile time ([env](compiler) reduce template instantiation in scan_operator to improve compile time #61227)
• Support adaptive memtable write buffer size ([enhance](memtable) support adaptive memtable write buffer size #56948)
• Remove NDEBUG for groupcommit's memory tracker ([fix](groupcommit) remove NDEBUG for groupcommit's memory tracker #56577)

Cloud Native

• Preserve correct DST fold branch to go cross the transition point ([fix](timestamptz) Preserve correct DST fold branch to go cross the transition point #63034)
• Clear warmup jobs on available backends ([fix](fe) Clear warmup jobs on available backends #62931)
• Extract snapshot integration hooks ([refactor](cloud) Extract snapshot integration hooks #62859)
• Refresh event warmup backends ([fix](cloud) refresh event warmup backends #62839)
• Avoid missing result packet for query during master switch ([fix](query) avoid missing result packet for query during master switch #62721)
• Fix redundant copy in collect_set merge ([fix](agg) fix redundant copy in collect_set merge #62640)
• Support IAM role auth in S3 filesystem ([fix](fe) Support IAM role auth in S3 filesystem #62584)
• Fix typo in different_serialize test data directory introduced by [fix](cloud)Fix create dynamic table race with insert overwrite #59489 ([fix](regression) Fix typo in different_serialize test data directory introduced by #59489 #62480)
• Fix race condition in Set operator runtime filter processing ([fix](runtime-filter) Fix race condition in Set operator runtime filter processing #62434)
• Enforce final on CRTP aggregate function derived classes to enable compiler devirtualization ([refine](function) enforce final on CRTP aggregate function derived classes to enable compiler devirtualization #62433)
• Support IAM Role for s3 client ut test ([opt](test) Support IAM Role for s3 client ut test #62303)
• Allocate fresh ExprId for constants when pushing project into Union ([fix](nereids) Allocate fresh ExprId for constants when pushing project into Union #62296)
• Hide KV_TXN_MAYBE_COMMITTED from clients ([fix](cloud) Hide KV_TXN_MAYBE_COMMITTED from clients #62244)
• Branch-4.1: feature Treat JuiceFS (jfs://) as HDFS-compatible in FE/BE [feature] Treat JuiceFS (jfs://) as HDFS-compatible in FE/BE #61031 (branch-4.1: [feature](jsf) Treat JuiceFS (jfs://) as HDFS-compatible in FE/BE #61031 #61706)
• Merge ms and recycler http skeleton ([chore](http) Merge ms and recycler http skeleton #61502)

Lakehouse

• Decouple JDBC catalog metadata name ([fix](iceberg) Decouple JDBC catalog metadata name #62806)
• Fix iceberg & maxcompute p2 case. ([fix](case)fix iceberg & maxcompute p2 case. #62483)
• Educe template instantiations in datetime floor/ceil functions ([env](compiler)educe template instantiations in datetime floor/ceil functions #61515)

Security & Authentication

• Restore _applied_rf_num update in late arrival path ([fix](runtime-filter) Restore _applied_rf_num update in late arrival path #62872)
• Support SSL and align MySQL CDC source with PG ([improve](streaming-job) support SSL and align MySQL CDC source with PG #62700)
• Fix empty workload_group value __internal_schema.audit_log ([fix](audit log) fix empty workload_group value __internal_schema.audit_log #62651)
• Fix Ranger column-level privilege bypass when CTE combined ([fix](fe) Fix Ranger column-level privilege bypass when CTE combined #61741)
• Fix HTTP API authentication framework for admin operations ([fix](auth) Fix HTTP API authentication framework for admin operations #60761)

Other

• Fix TIMESTAMPTZ elapsed-time semantics to use UTC ([Fix](ttz) Fix TIMESTAMPTZ elapsed-time semantics to use UTC #63161)
• Correct Arrow UTF8/String size limit ([fix](be) Correct Arrow UTF8/String size limit #63137)
• Remove useless if in is_acting_on_a_slot ([chore](expr) remove useless if in is_acting_on_a_slot #63095)
• No longer throws exceptions when parse datetime failed in from_olap_string ([fix](parse) No longer throws exceptions when parse datetime failed in from_olap_string #63035)
• Remove datetime transfomers check in release mode ([Opt](function) Remove datetime transfomers check in release mode #63003)
• Avoid formatting generated insert errors ([fix](insert) Avoid formatting generated insert errors #62982)
• Doris compose supports docker compose v2 ([fix](docker) doris compose supports docker compose v2 #62851)
• Preserve sign for negative sub-hour TIMESTAMPTZ offsets ([fix](function) preserve sign for negative sub-hour TIMESTAMPTZ offsets #62823)
• Fix -Wtype-limits error in VariantStatsCaculator ([fix](variant) Fix -Wtype-limits error in VariantStatsCaculator #62632)
• Fix FE startup argument forwarding ([fix](fe) Fix FE startup argument forwarding #62587)
• Fix wrong error message of insert overwrite ([fix](errmsg) Fix wrong error message of insert overwrite #62555)
• License check ([ci](fix) license check #62534)
• Move report logic to pipeline fragment context to remove callback parameter from ctor ([refactor](fragment mgr) move report logic to pipeline fragment context to remove callback parameter from ctor #62500)
• Extract isDynamicScheduleTable method to reduce code duplication ([refactor](fe) Extract isDynamicScheduleTable method to reduce code duplication #62477)
• Set parallel_pipeline_task_num explicitly in test_hive_compress_type_large_data. ([fix](test) Set parallel_pipeline_task_num explicitly in test_hive_compress_type_large_data.  #62423)
• Remove classhisto*=trace in JAVA_OPTS_FOR_JDK_17 to prevent printing class histogram in full gc ([opt](fe)remove classhisto*=trace in JAVA_OPTS_FOR_JDK_17 to prevent printing class histogram in full gc #62422)
• Fix RuntimeFilter selectivity sampling_frequency lost during VExprContext recreation ([fix](be) Fix RuntimeFilter selectivity sampling_frequency lost during VExprContext recreation #62355)
• Close snapshot readers on re-prepare and harden REST error handling ([fix](cdc) close snapshot readers on re-prepare and harden REST error handling #62337)
• Remove FixedContainer optimization for StringValueSet ([opt](expr) Remove FixedContainer optimization for StringValueSet #62243)
• Require exact match in DateTimeV2Type.acceptsType ([fix](datetime)Require exact match in DateTimeV2Type.acceptsType #62201)
• Remove io_helper helper indirection from BE cast paths ([refactor](be) Remove io_helper helper indirection from BE cast paths #62179)
• Reduce hash join build template instantiations ([env](compiler)Reduce hash join build template instantiations #61349)
• Optimize the compilation time of cast. ([env](compiler)Optimize the compilation time of cast. #61276)
• Reduce window function template instantiations ([env](compiler)Reduce window function template instantiations #61232)
• Reduce the compilation time of aggregate functions. ([env](compiler) Reduce the compilation time of aggregate functions. #61179)
• FE startup fail due to websocket startup ([fix](web) FE startup fail due to websocket startup #60369)
• Support max/min agg functions for type_array ([enhance](agg) Support max/min agg functions for type_array #58490)
• Fix AdminCreateClusterSnapshotCommand ([feat](snapshot) fix AdminCreateClusterSnapshotCommand #58119)
• Add a lambda functor version for array_sort ([enhance](function) add a lambda functor version for array_sort #57828)
• Clone snapshot support absolute path file ([feat](snapshot) clone snapshot support absolute path file #57685)
• Show the snapshot count ([fix](snapshot) show the snapshot count #56491)
• Move watcher.stop() into locked code block ([Bug](profile) move watcher.stop() into locked code block #56462)
• Process auto partition name when its length exceeds 50 ([opt](partition) process auto partition name when its length exceeds 50 #56304)