Merge branch 'master' into hanfei/remove_if_exists_for_backup_s3_writer

Author: hanfei1991

.github/PULL_REQUEST_TEMPLATE.md

@@ -43,7 +43,11 @@ At a minimum, the following information should be added (but add more as needed)
 
 > Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/
 
-#### CI Settings (Only check the boxes if you know what you are doing):
+#### CI Settings (Only check the boxes if you know what you are doing)
+
+All builds in Builds_1 and Builds_2 stages are always mandatory
+and will run independently of the checks below:
+
 - [ ] <!---ci_set_required--> Allow: All Required Checks
 - [ ] <!---ci_include_stateless--> Allow: Stateless tests
 - [ ] <!---ci_include_stateful--> Allow: Stateful tests
@@ -57,7 +61,9 @@ At a minimum, the following information should be added (but add more as needed)
 - [ ] <!---ci_exclude_fast--> Exclude: Fast test
 - [ ] <!---ci_exclude_asan--> Exclude: All with ASAN
 - [ ] <!---ci_exclude_tsan|msan|ubsan|coverage--> Exclude: All with TSAN, MSAN, UBSAN, Coverage
-- [ ] <!---ci_exclude_aarch64|release|debug--> Exclude: All with aarch64, release, debug
+- [ ] <!---ci_exclude_aarch64|release|debug--> Exclude: All with aarch64
+- [ ] <!---ci_exclude_release--> Exclude: All with release
+- [ ] <!---ci_exclude_debug--> Exclude: All with debug
 ---
 - [ ] <!---ci_include_fuzzer--> Run only fuzzers related jobs (libFuzzer fuzzers, AST fuzzers, etc.)
 - [ ] <!---ci_exclude_ast--> Exclude: AST fuzzers

.github/workflows/auto_releases.yml

@@ -7,8 +7,8 @@ concurrency:
   group: autoreleases
 
 on:
-  #  schedule:
-  #    - cron: '0 9 * * *'
+  schedule:
+    - cron: '45 11 * * 2,4'
   workflow_dispatch:
     inputs:
       dry-run:
@@ -52,7 +52,7 @@ jobs:
               echo 'EOF'
           } >> "$GITHUB_OUTPUT"
           if [[ "${{ github.event_name }}" == "schedule" ]]; then
-            echo "DRY_RUN=true" >> "$GITHUB_OUTPUT"
+            echo "DRY_RUN=false" >> "$GITHUB_OUTPUT"
           else
             echo "DRY_RUN=${{ github.event.inputs.dry-run }}" >> "$GITHUB_OUTPUT"
           fi

CHANGELOG.md

@@ -18,8 +18,8 @@
 ### <a id="2412"></a> ClickHouse release 24.12, 2024-12-19
 
 #### Backward Incompatible Change
-* Functions `greatest` and `least` now ignore NULL input values, whereas they previously returned NULL if one of the arguments was NULL. For example, `SELECT greatest(1, 2, NULL)` now returns 2. This makes the behavior compatible with PostgreSQL, but at the same time it breaks the compatiability with MySQL, who returns NULL. [#65519](https://github.com/ClickHouse/ClickHouse/pull/65519) ([kevinyhzou](https://github.com/KevinyhZou)).
-* The legacy MongoDB integration based on the Poco driver has been removed. Server setting `use_legacy_mongodb_integration` is obsolete and has no effect anymore. [#71997](https://github.com/ClickHouse/ClickHouse/pull/71997) ([Kirill Nikiforov](https://github.com/allmazz)). The new integration is strictly more capable and powerful.
+* Functions `greatest` and `least` now ignore NULL input values, whereas they previously returned NULL if one of the arguments was NULL. For example, `SELECT greatest(1, 2, NULL)` now returns 2. This makes the behavior compatible with PostgreSQL, but at the same time it breaks the compatibility with MySQL which returns NULL. To retain the previous behavior, set setting `least_greatest_legacy_null_behavior` (default: `false`) to `true`. [#65519](https://github.com/ClickHouse/ClickHouse/pull/65519) [#73344](https://github.com/ClickHouse/ClickHouse/pull/73344) ([kevinyhzou](https://github.com/KevinyhZou)).
+* A new MongoDB integration is now the default. Users who like to use the legacy MongoDB driver (based on the Poco driver) can enable server setting `use_legacy_mongodb_integration`. [#73359](https://github.com/ClickHouse/ClickHouse/pull/73359) ([Kirill Nikiforov](https://github.com/allmazz).
 
 #### New Feature
 * Move `JSON`/`Dynamic`/`Variant` types from experimental features to beta. [#72294](https://github.com/ClickHouse/ClickHouse/pull/72294) ([Pavel Kruglov](https://github.com/Avogar)). We also backported all fixes as well as this change to 24.11.
@@ -63,7 +63,7 @@
 * Remove the `allow_experimental_join_condition` setting, allowing non-equi conditions by default. [#69910](https://github.com/ClickHouse/ClickHouse/pull/69910) ([Vladimir Cherkasov](https://github.com/vdimir)).
 * Settings from server config (users.xml) now apply on the client too. Useful for format settings, e.g. `date_time_output_format`. [#71178](https://github.com/ClickHouse/ClickHouse/pull/71178) ([Michael Kolupaev](https://github.com/al13n321)).
 * Automatic `GROUP BY`/`ORDER BY` to disk based on the server/user memory usage. Controlled with `max_bytes_ratio_before_external_group_by`/`max_bytes_ratio_before_external_sort` query settings. [#71406](https://github.com/ClickHouse/ClickHouse/pull/71406) ([Azat Khuzhin](https://github.com/azat)).
-* Adding a new cancellation logic using a singleton object `CancellationChecker` which checks the timeouts for queries. [#69880](https://github.com/ClickHouse/ClickHouse/pull/69880) ([Yarik Briukhovetskyi](https://github.com/yariks5s)).
+* Adding a new cancellation logic: `CancellationChecker` checks timeouts for every started query and stops them once the timeout has reached. [#69880](https://github.com/ClickHouse/ClickHouse/pull/69880) ([Yarik Briukhovetskyi](https://github.com/yariks5s)).
 * Support ALTER from `Object` to `JSON`, which means you can easily migrate from the deprecated Object type. [#71784](https://github.com/ClickHouse/ClickHouse/pull/71784) ([Pavel Kruglov](https://github.com/Avogar)).
 * Allow unknown values in set that are not present in Enum. Fix [#72662](https://github.com/ClickHouse/ClickHouse/issues/72662). [#72686](https://github.com/ClickHouse/ClickHouse/pull/72686) ([zhanglistar](https://github.com/zhanglistar)).
 * Support string search operator (eg., LIKE) for `Enum` data type, implements [#72661](https://github.com/ClickHouse/ClickHouse/issues/72661). [#72732](https://github.com/ClickHouse/ClickHouse/pull/72732) ([zhanglistar](https://github.com/zhanglistar)).
@@ -87,6 +87,7 @@ memory usage issue. [#72490](https://github.com/ClickHouse/ClickHouse/pull/72490
 * Add S3 request settings `http_max_fields`, `http_max_field_name_size`, `http_max_field_value_size` and use them while parsing S3 API responses during making a backup or restoring. [#72778](https://github.com/ClickHouse/ClickHouse/pull/72778) ([Vitaly Baranov](https://github.com/vitlibar)).
 * Delete table metadata in keeper in Storage S3(Azure)Queue only after last table using this metadata was dropped. [#72810](https://github.com/ClickHouse/ClickHouse/pull/72810) ([Kseniia Sumarokova](https://github.com/kssenii)).
 * Added `JoinBuildTableRowCount`/`JoinProbeTableRowCount/JoinResultRowCount` profile events. [#72842](https://github.com/ClickHouse/ClickHouse/pull/72842) ([Vladimir Cherkasov](https://github.com/vdimir)).
+* Support subcolumns in MergeTree sorting key and skip indexes. [#72644](https://github.com/ClickHouse/ClickHouse/pull/72644) ([Pavel Kruglov](https://github.com/Avogar)).
 
 #### Bug Fix (user-visible misbehavior in an official stable release)
 * Fix possible intersecting parts for MergeTree (after an operation of moving part to the detached directory has been failed, possibly due to operation on object storage). [#70476](https://github.com/ClickHouse/ClickHouse/pull/70476) ([Azat Khuzhin](https://github.com/azat)).

CMakeLists.txt

@@ -7,6 +7,9 @@ option(FAIL_ON_UNSUPPORTED_OPTIONS_COMBINATION
    "Stop/Fail CMake configuration if some ENABLE_XXX option is defined (either ON or OFF)
    but is not possible to satisfy" ON)
 
+# This allows to compile some code conditionally in the private build while having byte-identical source files.
+set (CLICKHOUSE_CLOUD 0)
+
 if(FAIL_ON_UNSUPPORTED_OPTIONS_COMBINATION)
     set(RECONFIGURE_MESSAGE_LEVEL FATAL_ERROR)
 else()
@@ -269,6 +272,10 @@ else ()
     set (ENABLE_BUILD_PATH_MAPPING_DEFAULT ON)
 endif ()
 
+if (COMPILER_CACHE STREQUAL "chcache")
+    set (ENABLE_BUILD_PATH_MAPPING_DEFAULT ON)
+endif()
+
 option (ENABLE_BUILD_PATH_MAPPING "Enable remapping of file source paths in debug info, predefined preprocessor macros, and __builtin_FILE(). It's used to generate reproducible builds. See https://reproducible-builds.org/docs/build-path" ${ENABLE_BUILD_PATH_MAPPING_DEFAULT})
 
 if (ENABLE_BUILD_PATH_MAPPING)
@@ -347,10 +354,11 @@ endif()
 set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fdiagnostics-absolute-paths")
 set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fdiagnostics-absolute-paths")
 
-if (NOT ENABLE_TESTS AND NOT SANITIZE AND NOT SANITIZE_COVERAGE AND OS_LINUX)
+if (NOT ENABLE_TESTS AND NOT ENABLE_CLANG_TIDY AND NOT SANITIZE AND NOT SANITIZE_COVERAGE AND OS_LINUX)
     # https://clang.llvm.org/docs/ThinLTO.html
     # Applies to clang and linux only.
     # Disabled when building with tests or sanitizers.
+    # Also disabled with clang-tidy where we don't care about linking
     option(ENABLE_THINLTO "Clang-specific link time optimization" ON)
 endif()
 
@@ -463,6 +471,9 @@ message (STATUS "Building for: ${CMAKE_SYSTEM} ${CMAKE_SYSTEM_PROCESSOR} ${CMAKE
 
 add_subdirectory (contrib EXCLUDE_FROM_ALL)
 
+# Ensure that we always use the dummy launchers for our own code.
+set_dummy_launchers_if_needed()
+
 if (NOT ENABLE_JEMALLOC)
     message (WARNING "Non default allocator is disabled. This is not recommended for production builds.")
 endif ()
@@ -592,8 +603,6 @@ endif()
 include (cmake/sanitize_targets.cmake)
 
 if (COMPILER_CACHE STREQUAL "chcache")
-    set (ENABLE_BUILD_PATH_MAPPING_DEFAULT ON)
-
     get_all_targets(all_targets)
     set (chcache_targets _cargo-build_chcache cargo-build_chcache cargo-prebuild_chcache)
     foreach(target ${all_targets})

README.md

@@ -42,8 +42,10 @@ Every month we get together with the community (users, contributors, customers,
 Keep an eye out for upcoming meetups and events around the world. Somewhere else you want us to be? Please feel free to reach out to tyler `<at>` clickhouse `<dot>` com. You can also peruse [ClickHouse Events](https://clickhouse.com/company/news-events) for a list of all upcoming trainings, meetups, speaking engagements, etc.
 
 Upcoming meetups
-
-* [Dubai Meetup](https://www.meetup.com/clickhouse-dubai-meetup-group/events/303096989/) - Feb 3
+* [Tokyo Meetup](https://www.meetup.com/clickhouse-tokyo-user-group/events/305126993/) - Jan 23
+* [FOSDEM 2025](https://www.fosdem.org/2025/schedule/event/fosdem-2025-6351-how-we-built-a-new-powerful-json-data-type-for-clickhouse/) - Feb 1, 2
+* [London Meetup](https://www.meetup.com/clickhouse-london-user-group/events/305146729/) - Feb 5
+* [Dubai Meetup](https://www.meetup.com/clickhouse-dubai-meetup-group/events/303096989/) - Feb 10
 
 Recently completed meetups
 

SECURITY.md

@@ -14,9 +14,10 @@ The following versions of ClickHouse server are currently supported with securit
 
 | Version | Supported |
 |:-|:-|
+| 24.12 | ✔️ |
 | 24.11 | ✔️ |
 | 24.10 | ✔️ |
-| 24.9 | ✔️ |
+| 24.9 | ❌ |
 | 24.8 | ✔️ |
 | 24.7 | ❌ |
 | 24.6 | ❌ |

base/base/AlignedUnion.h

@@ -0,0 +1,16 @@
+#pragma once
+
+#include <algorithm>
+
+template<std::size_t len, class... Types>
+struct AlignedUnion
+{
+    static constexpr std::size_t alignment_value = std::max({alignof(Types)...});
+    struct Type
+    {
+        alignas(alignment_value) char s[std::max({len, sizeof(Types)...})];
+    };
+};
+
+template<std::size_t len, class... Types>
+using AlignedUnionT = typename AlignedUnion<len, Types...>::Type;

base/base/unit.h

@@ -9,9 +9,9 @@ constexpr size_t GiB = 1024 * MiB;
 #pragma clang diagnostic ignored "-Wreserved-identifier"
 
 // NOLINTBEGIN(google-runtime-int)
-constexpr size_t operator"" _KiB(unsigned long long val) { return val * KiB; }
-constexpr size_t operator"" _MiB(unsigned long long val) { return val * MiB; }
-constexpr size_t operator"" _GiB(unsigned long long val) { return val * GiB; }
+constexpr size_t operator""_KiB(unsigned long long val) { return val * KiB; }
+constexpr size_t operator""_MiB(unsigned long long val) { return val * MiB; }
+constexpr size_t operator""_GiB(unsigned long long val) { return val * GiB; }
 // NOLINTEND(google-runtime-int)
 
 #pragma clang diagnostic pop

base/base/wide_integer_impl.h

@@ -85,7 +85,6 @@ class numeric_limits<wide::integer<Bits, Signed>>
     static constexpr bool has_infinity = false;
     static constexpr bool has_quiet_NaN = false;
     static constexpr bool has_signaling_NaN = true;
-    static constexpr std::float_denorm_style has_denorm = std::denorm_absent;
     static constexpr bool has_denorm_loss = false;
     static constexpr std::float_round_style round_style = std::round_toward_zero;
     static constexpr bool is_iec559 = false;

base/poco/CMakeLists.txt

@@ -3,6 +3,11 @@ add_subdirectory (Data)
 add_subdirectory (Data/ODBC)
 add_subdirectory (Foundation)
 add_subdirectory (JSON)
+
+if (USE_MONGODB)
+    add_subdirectory(MongoDB)
+endif()
+
 add_subdirectory (Net)
 add_subdirectory (NetSSL_OpenSSL)
 add_subdirectory (Redis)