Merge branch 'ClickHouse:master' into buzzhouse-update

Author: PedroTadim

README.md

@@ -51,7 +51,6 @@ Upcoming meetups
 * [Los Gatos Meetup](https://www.meetup.com/clickhouse-silicon-valley-meetup-group/events/306445660) - March 12, 2025
 * [San Francisco Meetup](https://www.meetup.com/clickhouse-silicon-valley-meetup-group/events/306046697) - March 19, 2025
 * [Delhi Meetup](https://www.meetup.com/clickhouse-delhi-user-group/events/306253492/) - March 22, 2025
-* [Zurich Meetup](https://www.meetup.com/clickhouse-switzerland-meetup-group/events/306435122/) - March 24, 2025
 * [Budapest Meetup](https://www.meetup.com/clickhouse-hungary-user-group/events/306435234/) - March 25, 2025
 * [Boston Meetup](https://www.meetup.com/clickhouse-boston-user-group/events/305882607) - March 25, 2025
 * [Sao Paulo Meetup](https://www.meetup.com/clickhouse-brasil-user-group/events/306385974/) - March 25, 2025

base/base/defines.h

@@ -28,6 +28,12 @@
 #define NO_INLINE __attribute__((__noinline__))
 #define MAY_ALIAS __attribute__((__may_alias__))
 
+#if defined(__x86_64__) || defined(__aarch64__)
+#    define PRESERVE_MOST __attribute__((preserve_most))
+#else
+#    define PRESERVE_MOST
+#endif
+
 #if !defined(__x86_64__) && !defined(__aarch64__) && !defined(__PPC__) && !defined(__s390x__) && !(defined(__loongarch64)) && !(defined(__riscv) && (__riscv_xlen == 64))
 #    error "The only supported platforms are x86_64 and AArch64, PowerPC (work in progress), s390x (work in progress), loongarch64 (experimental) and RISC-V 64 (experimental)"
 #endif

ci/defs/job_configs.py

@@ -476,6 +476,7 @@ class JobConfigs:
         digest_config=Job.CacheDigestConfig(
             include_paths=[
                 "./tests/queries/0_stateless/",
+                "./tests/ci/stress.py",
                 "./tests/clickhouse-test",
                 "./tests/config",
                 "./tests/*.txt",

ci/praktika/json.html

@@ -808,7 +808,11 @@
 
             const infoCell = document.createElement('td');
             infoCell.colSpan = row.children.length; // Make it span the full width
-            infoCell.innerHTML = infoText.replace(/\n/g, '<br>'); // Preserve line breaks
+
+            infoCell.innerHTML = '<pre></pre>';
+            // Append content securely, without interpreting HTML tags.
+            infoCell.firstChild.textContent = infoText;
+
             infoCell.classList.add('info-text');
 
             infoRow.appendChild(infoCell);

docs/changelogs/v24.12.6.70-stable.md

@@ -0,0 +1,49 @@
+---
+sidebar_position: 1
+sidebar_label: 2025
+---
+
+# 2025 Changelog
+
+### ClickHouse release v24.12.6.70-stable (834cccbc6e8) FIXME as compared to v24.12.5.81-stable (abbc2854715)
+
+#### Performance Improvement
+* Backported in [#76694](https://github.com/ClickHouse/ClickHouse/issues/76694): Fixed unnecessary contention in `parallel_hash` when `max_rows_in_join = max_bytes_in_join = 0`. [#75155](https://github.com/ClickHouse/ClickHouse/pull/75155) ([Nikita Taranov](https://github.com/nickitat)).
+
+#### Improvement
+* Backported in [#76428](https://github.com/ClickHouse/ClickHouse/issues/76428): Address some clickhouse-disks usability issues addressed by users. Closes [#67136](https://github.com/ClickHouse/ClickHouse/issues/67136). [#73616](https://github.com/ClickHouse/ClickHouse/pull/73616) ([Daniil Ivanik](https://github.com/divanik)).
+* Backported in [#76862](https://github.com/ClickHouse/ClickHouse/issues/76862): Add ability to reload `max_remote_read_network_bandwidth_for_serve` and `max_remote_write_network_bandwidth_for_server` on fly without restart server. [#74206](https://github.com/ClickHouse/ClickHouse/pull/74206) ([Kai Zhu](https://github.com/nauu)).
+* Backported in [#76451](https://github.com/ClickHouse/ClickHouse/issues/76451): Allow to parse endpoints like `localhost:1234/handle` in `postgresql` or `mysql` table functions. This fixes a regression introduced in https://github.com/ClickHouse/ClickHouse/pull/52503. [#75944](https://github.com/ClickHouse/ClickHouse/pull/75944) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
+* Backported in [#76652](https://github.com/ClickHouse/ClickHouse/issues/76652): Added a server setting `throw_on_unknown_workload` that allows to choose behavior on query with `workload` setting set to unknown value: either allow unlimited access (default) or throw a `RESOURCE_ACCESS_DENIED` error. It is useful to force all queries to use workload scheduling. [#75999](https://github.com/ClickHouse/ClickHouse/pull/75999) ([Sergei Trifonov](https://github.com/serxa)).
+* Backported in [#76671](https://github.com/ClickHouse/ClickHouse/issues/76671): Use correct fallback when multipart copy to S3 fails during backup with Access Denied. Multi part copy can generate Access Denied error when backup is done between buckets that have different credentials. [#76515](https://github.com/ClickHouse/ClickHouse/pull/76515) ([Antonio Andelic](https://github.com/antonio2368)).
+* Backported in [#77200](https://github.com/ClickHouse/ClickHouse/issues/77200): Previously database replicated might print credentials specified in a query to logs. This behaviour is fixed. This closes: [#77123](https://github.com/ClickHouse/ClickHouse/issues/77123). [#77133](https://github.com/ClickHouse/ClickHouse/pull/77133) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
+
+#### Bug Fix (user-visible misbehavior in an official stable release)
+* Backported in [#74520](https://github.com/ClickHouse/ClickHouse/issues/74520): Skip `metadata_version.txt` in while restoring parts from a backup. [#73768](https://github.com/ClickHouse/ClickHouse/pull/73768) ([Vitaly Baranov](https://github.com/vitlibar)).
+* Backported in [#76527](https://github.com/ClickHouse/ClickHouse/issues/76527): Respect `materialized_views_ignore_errors` when a materialized view writes to a URL engine and there is a connectivity issue. [#75679](https://github.com/ClickHouse/ClickHouse/pull/75679) ([Christoph Wurm](https://github.com/cwurm)).
+* Backported in [#76538](https://github.com/ClickHouse/ClickHouse/issues/76538): Fix `Block structure mismatch in QueryPipeline stream` error for some queries with `UNION ALL`. [#75715](https://github.com/ClickHouse/ClickHouse/pull/75715) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
+* Backported in [#76149](https://github.com/ClickHouse/ClickHouse/issues/76149): Propagate format settings to JSON as string formatting in Native format. [#75832](https://github.com/ClickHouse/ClickHouse/pull/75832) ([Pavel Kruglov](https://github.com/Avogar)).
+* Backported in [#76995](https://github.com/ClickHouse/ClickHouse/issues/76995): Fix data loss when enable async insert and insert into ... from file ... with unequal block size if the first block size < async_max_size but the second block > async_max_size, the second block will not be inserted. these data is left in `squashing`. [#76343](https://github.com/ClickHouse/ClickHouse/pull/76343) ([Han Fei](https://github.com/hanfei1991)).
+* Backported in [#76401](https://github.com/ClickHouse/ClickHouse/issues/76401): Fix logical error in index analysis if condition in `WHERE` has `pointInPolygon` function. [#76360](https://github.com/ClickHouse/ClickHouse/pull/76360) ([Anton Popov](https://github.com/CurtizJ)).
+* Backported in [#76482](https://github.com/ClickHouse/ClickHouse/issues/76482): Removed allocation from the signal handler. [#76446](https://github.com/ClickHouse/ClickHouse/pull/76446) ([Nikita Taranov](https://github.com/nickitat)).
+* Backported in [#76966](https://github.com/ClickHouse/ClickHouse/issues/76966): Fix THERE_IS_NO_COLUMN exception when selecting boolean literal from distributed tables. [#76656](https://github.com/ClickHouse/ClickHouse/pull/76656) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)).
+* Backported in [#76856](https://github.com/ClickHouse/ClickHouse/issues/76856): Flush output write buffers before finalizing them. Fix `LOGICAL_ERROR` generated during the finalization of some output format, e.g. `JSONEachRowWithProgressRowOutputFormat`. [#76726](https://github.com/ClickHouse/ClickHouse/pull/76726) ([Antonio Andelic](https://github.com/antonio2368)).
+* Backported in [#77254](https://github.com/ClickHouse/ClickHouse/issues/77254): Fix possible crash because of bad JSON column rollback on error during async inserts. [#76908](https://github.com/ClickHouse/ClickHouse/pull/76908) ([Pavel Kruglov](https://github.com/Avogar)).
+* Backported in [#77060](https://github.com/ClickHouse/ClickHouse/issues/77060): Fix sorting of `BFloat16` values. This closes [#75487](https://github.com/ClickHouse/ClickHouse/issues/75487). This closes [#75669](https://github.com/ClickHouse/ClickHouse/issues/75669). [#77000](https://github.com/ClickHouse/ClickHouse/pull/77000) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
+* Backported in [#77218](https://github.com/ClickHouse/ClickHouse/issues/77218): Bug fix json with variant subcolumn by adding check to skip ephemeral subcolumns in part consistency check. [#72187](https://github.com/ClickHouse/ClickHouse/issues/72187). [#77034](https://github.com/ClickHouse/ClickHouse/pull/77034) ([Smita Kulkarni](https://github.com/SmitaRKulkarni)).
+* Backported in [#77246](https://github.com/ClickHouse/ClickHouse/issues/77246): Fix crash during Kafka table creation with exception. [#77121](https://github.com/ClickHouse/ClickHouse/pull/77121) ([Pavel Kruglov](https://github.com/Avogar)).
+* Backported in [#77391](https://github.com/ClickHouse/ClickHouse/issues/77391): `SELECT toBFloat16(-0.0) == toBFloat16(0.0)` now correctly returns `true` (from previously `false`). This makes the behavior consistent with `Float32` and `Float64`. [#77290](https://github.com/ClickHouse/ClickHouse/pull/77290) ([Shankar Iyer](https://github.com/shankar-iyer)).
+* Backported in [#77331](https://github.com/ClickHouse/ClickHouse/issues/77331): Fix comparison between tuples with nullable elements inside and strings. [#77323](https://github.com/ClickHouse/ClickHouse/pull/77323) ([Alexey Katsman](https://github.com/alexkats)).
+
+#### NOT FOR CHANGELOG / INSIGNIFICANT
+
+* Backported in [#76506](https://github.com/ClickHouse/ClickHouse/issues/76506): Add `prefetch` method to `ReadBufferFromEncryptedFile`. [#76322](https://github.com/ClickHouse/ClickHouse/pull/76322) ([Antonio Andelic](https://github.com/antonio2368)).
+* Backported in [#76463](https://github.com/ClickHouse/ClickHouse/issues/76463): Fix `setReadUntilPosition` in `AsynchronousBoundedReadBuffer`. [#76429](https://github.com/ClickHouse/ClickHouse/pull/76429) ([Antonio Andelic](https://github.com/antonio2368)).
+* Backported in [#76600](https://github.com/ClickHouse/ClickHouse/issues/76600): Use `MultiRead` when querying `system.distributed_ddl_queue`. [#76575](https://github.com/ClickHouse/ClickHouse/pull/76575) ([Antonio Andelic](https://github.com/antonio2368)).
+* Backported in [#76892](https://github.com/ClickHouse/ClickHouse/issues/76892): Add log for HTTP Bad Request. [#76594](https://github.com/ClickHouse/ClickHouse/pull/76594) ([Christoph Wurm](https://github.com/cwurm)).
+* Backported in [#76821](https://github.com/ClickHouse/ClickHouse/issues/76821): CI: Disable cross-compile for ARM in release and backport. [#76808](https://github.com/ClickHouse/ClickHouse/pull/76808) ([Max Kainov](https://github.com/maxknv)).
+* Backported in [#76957](https://github.com/ClickHouse/ClickHouse/issues/76957): Allow empty chunks in FinishSortingTransfrom. [#76919](https://github.com/ClickHouse/ClickHouse/pull/76919) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
+* Backported in [#77043](https://github.com/ClickHouse/ClickHouse/issues/77043): fix fast test 02783_parsedatetimebesteffort leap year. [#76940](https://github.com/ClickHouse/ClickHouse/pull/76940) ([Han Fei](https://github.com/hanfei1991)).
+* Backported in [#77080](https://github.com/ClickHouse/ClickHouse/issues/77080): Increase log level for dictionary loading. [#77052](https://github.com/ClickHouse/ClickHouse/pull/77052) ([Michael Lex](https://github.com/mlex)).
+* Backported in [#77265](https://github.com/ClickHouse/ClickHouse/issues/77265): Fix uninitialized-value `CoordinationZnode::last_success_duration` in `RefreshTask`. [#77174](https://github.com/ClickHouse/ClickHouse/pull/77174) ([Tuan Pham Anh](https://github.com/tuanpach)).
+

docs/en/engines/table-engines/integrations/s3.md

@@ -174,6 +174,10 @@ Received exception from server (version 23.4.1):
 Code: 48. DB::Exception: Received from localhost:9000. DB::Exception: Reading from a partitioned S3 storage is not implemented yet. (NOT_IMPLEMENTED)
 ```
 
+## Inserting Data {#inserting-data}
+
+Note that rows can only be inserted into new files. There are no merge cycles or file split operations. Once a file is written, subsequent inserts will fail. To avoid this you can use `s3_truncate_on_insert` and `s3_create_new_file_on_insert` settings. See more details [here](/integrations/s3#inserting-data).
+
 ## Virtual columns {#virtual-columns}
 
 - `_path` — Path to the file. Type: `LowCardinality(String)`.
@@ -340,12 +344,12 @@ FROM s3(
 );
 ```
 
-:::note 
+:::note
 ClickHouse supports three archive formats:
 ZIP
 TAR
 7Z
-While ZIP and TAR archives can be accessed from any supported storage location, 7Z archives can only be read from the local filesystem where ClickHouse is installed.  
+While ZIP and TAR archives can be accessed from any supported storage location, 7Z archives can only be read from the local filesystem where ClickHouse is installed.
 :::
 
 
@@ -367,3 +371,4 @@ For details on optimizing the performance of the s3 function see [our detailed g
 ## See also {#see-also}
 
 - [s3 table function](../../../sql-reference/table-functions/s3.md)
+- [Integrating S3 with ClickHouse](/integrations/s3)

docs/en/engines/table-engines/mergetree-family/annindexes.md

@@ -120,16 +120,11 @@ additional techniques are recommended to speed up index creation:
 - Index creation can be parallelized. The maximum number of threads can be configured using server setting
   [max_build_vector_similarity_index_thread_pool_size](../../../operations/server-configuration-parameters/settings.md#server_configuration_parameters_max_build_vector_similarity_index_thread_pool_size).
 - Index creation on newly inserted parts may be disabled using setting `materialize_skip_indexes_on_insert`. Search on such parts will fall
-  back to exact search but since inserted parts are typically small compared to the total table size, the performance impact is negligible.
-- ClickHouse merges multiple parts incrementally in the background into bigger parts. These new parts are potentially merged later into even
-  bigger parts. Each merge re-builds the vector similarity index the output part (as well as other skip indexes) every time from
-  scratch. This potentially wastes work for creating vector similarity indexes. To avoid that, it is possible to suppress the creation of
-  vector similarity indexes during merge using merge tree setting
-  [materialize_skip_indexes_on_merge](../../../operations/settings/merge-tree-settings.md#materialize_skip_indexes_on_merge). This, in
-  conjunction with statement [ALTER TABLE \[...\] MATERIALIZE INDEX
-  \[...\]](../../../sql-reference/statements/alter/skipping-index.md#materialize-index), provides explicit control over the life cycle of
-  vector similarity indexes. For example, index building can be deferred to periods of low load (e.g. weekends) or after a large data
-  ingestion.
+  back to exact search but as inserted parts are typically small compared to the total table size, the performance impact is negligible.
+- As parts are incrementally merged into bigger parts, and these new parts are merged into even bigger parts ("write amplification"),
+  vector similarity indexes are possibly build multiple times for the same vectors. To avoid that, you may suppress merges during insert
+  using statement [`SYSTEM STOP MERGES`](../../../sql-reference/statements/system.md), respectively start merges once all data has been
+  inserted using `SYSTEM START MERGES`.
 
 Vector similarity indexes support this type of query:
 
@@ -146,9 +141,6 @@ To search using a different value of HNSW parameter `hnsw_candidate_list_size_fo
 original [HNSW paper](https://doi.org/10.1109/TPAMI.2018.2889473), run the `SELECT` query with `SETTINGS hnsw_candidate_list_size_for_search
 = <value>`.
 
-Repeated reads from vector similarity indexes benefit from a large skipping index cache. If needed, you can increase the default cache size
-using server setting [skipping_index_cache_size](../../../operations/server-configuration-parameters/settings.md#skipping_index_cache_size).
-
 **Restrictions**: Approximate vector search algorithms require a limit, hence queries without `LIMIT` clause cannot utilize vector
 similarity indexes. The limit must also be smaller than setting `max_limit_for_ann_queries` (default: 100).
 

docs/en/operations/caches.md

@@ -12,7 +12,6 @@ Main cache types:
 
 - `mark_cache` — Cache of marks used by table engines of the [MergeTree](../engines/table-engines/mergetree-family/mergetree.md) family.
 - `uncompressed_cache` — Cache of uncompressed data used by table engines of the [MergeTree](../engines/table-engines/mergetree-family/mergetree.md) family.
-- `skipping_index_cache` — Cache of in-memory skipping index granules used by table engines of the [MergeTree](../engines/table-engines/mergetree-family/mergetree.md) family.
 - Operating system page cache (used indirectly, for files with actual data).
 
 Additional cache types:

docs/en/operations/cluster-discovery.md

@@ -119,6 +119,48 @@ To enable observer mode, include the `<observer/>` tag within the `<discovery>`
 ```
 
 
+### Discovery of clusters {#discovery-of-clusters}
+
+Sometimes you may need to add and remove not only hosts in clusters, but clusters themselves. You can use the `<multicluster_root_path>` node with root path for several clusters:
+
+```xml
+<remote_servers>
+    <some_unused_name>
+        <discovery>
+            <multicluster_root_path>/clickhouse/discovery</multicluster_root_path>
+            <observer/>
+        </discovery>
+    </some_unused_name>
+</remote_servers>
+```
+
+In this case, when some other host registers itself with the path `/clickhouse/discovery/some_new_cluster`, a cluster with name `some_new_cluster` will be added.
+
+You can use both features simultaneously, the host can register itself in cluster `my_cluster` and discovery any other clusters:
+
+```xml
+<remote_servers>
+    <my_cluster>
+        <discovery>
+            <path>/clickhouse/discovery/my_cluster</path>
+        </discovery>
+    </my_cluster>
+    <some_unused_name>
+        <discovery>
+            <multicluster_root_path>/clickhouse/discovery</multicluster_root_path>
+            <observer/>
+        </discovery>
+    </some_unused_name>
+</remote_servers>
+```
+
+Limitations:
+- You can't use both `<path>` and `<multicluster_root_path>` in the same `remote_servers` subtree.
+- `<multicluster_root_path>` can only be with `<observer/>`.
+- The last part of path from Keeper is used as the cluster name, while during registration the name is taken from the XML tag.
+
+
+
 ## Use-Cases and Limitations {#use-cases-and-limitations}
 
 As nodes are added or removed from the specified ZooKeeper path, they are automatically discovered or removed from the cluster without the need for configuration changes or server restarts.