Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into add_bech32_enc_dec

Author: george-larionov

CHANGELOG.md

@@ -26,9 +26,7 @@
 * If reading from an `URL` involves multiple redirects, setting `enable_url_encoding` is correctly applied across all redirects in the chain. [#79563](https://github.com/ClickHouse/ClickHouse/pull/79563) ([Shankar Iyer](https://github.com/shankar-iyer)). Setting `enble_url_encoding` default value is now set to `false`. [#80088](https://github.com/ClickHouse/ClickHouse/pull/80088) ([Shankar Iyer](https://github.com/shankar-iyer)).
 
 #### New Feature
-* New `Time`/`Time64` data types: `Time` (HHH:MM:SS) and `Time64` (HHH:MM:SS.`<fractional>`) and some basic cast functions and functions to interact with other data types. Also, changed the existing function's name toTime to toTimeWithFixedDate because the function toTime is required for the cast function. [#75735](https://github.com/ClickHouse/ClickHouse/pull/75735) ([Yarik Briukhovetskyi](https://github.com/yariks5s)).
-72459). [#76078](https://github.com/ClickHouse/ClickHouse/pull/76078) ([Dmitry Novik](https://github.com/novikd)).
-* Support scalar correlated subqueries in the WHERE clause. Closes [#6697](https://github.com/ClickHouse/ClickHouse/issues/6697). [#79600](https://github.com/ClickHouse/ClickHouse/pull/79600) ([Dmitry Novik](https://github.com/novikd)). Support correlated subqueries in the projection list in simple cases. [#79925](https://github.com/ClickHouse/ClickHouse/pull/79925) ([Dmitry Novik](https://github.com/novikd)). Now it covers 100% of TPC-H test suite.
+* Support scalar correlated subqueries in the WHERE clause. Closes [#6697](https://github.com/ClickHouse/ClickHouse/issues/6697). [#79600](https://github.com/ClickHouse/ClickHouse/pull/79600) ([Dmitry Novik](https://github.com/novikd)). Support correlated subqueries in the projection list in simple cases. [#79925](https://github.com/ClickHouse/ClickHouse/pull/79925) ([Dmitry Novik](https://github.com/novikd)). [#76078](https://github.com/ClickHouse/ClickHouse/pull/76078) ([Dmitry Novik](https://github.com/novikd)). Now it covers 100% of TPC-H test suite.
 * Vector search using the vector similarity index is now beta (from previously experimental). [#80164](https://github.com/ClickHouse/ClickHouse/pull/80164) ([Robert Schulze](https://github.com/rschu1ze)).
 * Support geo types in `Parquet` format. This closes [#75317](https://github.com/ClickHouse/ClickHouse/issues/75317). [#79777](https://github.com/ClickHouse/ClickHouse/pull/79777) ([scanhex12](https://github.com/scanhex12)).
 * New functions `sparseGrams`, `sparseGramsHashes`, `sparseGramsHashesUTF8`, `sparseGramsUTF8` for calculating "sparse-ngrams" - a robust algorithm for extracting substrings for indexing and search. [#79517](https://github.com/ClickHouse/ClickHouse/pull/79517) ([scanhex12](https://github.com/scanhex12)).
@@ -44,6 +42,8 @@
 * Add [`icebergHash`](https://iceberg.apache.org/spec/#appendix-b-32-bit-hash-requirements) and [`icebergBucketTransform`](https://iceberg.apache.org/spec/#bucket-transform-details) functions. Support data files pruning in `Iceberg` tables partitioned with [`bucket transfom`](https://iceberg.apache.org/spec/#partitioning). [#79262](https://github.com/ClickHouse/ClickHouse/pull/79262) ([Daniil Ivanik](https://github.com/divanik)).
 
 #### Experimental Feature
+* New `Time`/`Time64` data types: `Time` (HHH:MM:SS) and `Time64` (HHH:MM:SS.`<fractional>`) and some basic cast functions and functions to interact with other data types. Also, changed the existing function's name toTime to toTimeWithFixedDate because the function toTime is required for the cast function. [#75735](https://github.com/ClickHouse/ClickHouse/pull/75735) ([Yarik Briukhovetskyi](https://github.com/yariks5s)).
+72459).
 * Hive metastore catalog for Iceberg datalake. [#77677](https://github.com/ClickHouse/ClickHouse/pull/77677) ([scanhex12](https://github.com/scanhex12)).
 * Indexes of type `full_text` were renamed to `gin`. This follows the more familiar terminology of PostgreSQL and other databases. Existing indexes of type `full_text` remain loadable but they will throw an exception (suggesting `gin` indexes instead) when one tries to use them in searches. [#79024](https://github.com/ClickHouse/ClickHouse/pull/79024) ([Robert Schulze](https://github.com/rschu1ze)).
 
@@ -60,7 +60,6 @@
 * Allow parallel merging of `uniqExact` states during the final stage of distributed aggregation. [#78703](https://github.com/ClickHouse/ClickHouse/pull/78703) ([Nikita Taranov](https://github.com/nickitat)).
 * Fix possible performance degradation of the parallel merging of `uniqExact` states for aggregation with key. [#78724](https://github.com/ClickHouse/ClickHouse/pull/78724) ([Nikita Taranov](https://github.com/nickitat)).
 * Reduce the number of List Blobs API calls to Azure storage. [#78860](https://github.com/ClickHouse/ClickHouse/pull/78860) ([Julia Kartseva](https://github.com/jkartseva)).
-* Merge equality conditions from filter query plan step into JOIN condition if possible to allow using them as hash table keys. [#78877](https://github.com/ClickHouse/ClickHouse/pull/78877) ([Dmitry Novik](https://github.com/novikd)).
 * Fix performance of the distributed INSERT SELECT with parallel replicas. [#79441](https://github.com/ClickHouse/ClickHouse/pull/79441) ([Azat Khuzhin](https://github.com/azat)).
 * Prevent `LogSeriesLimiter` from doing cleanup on every construction, avoiding lock contention and performance regressions in high-concurrency scenarios. [#79864](https://github.com/ClickHouse/ClickHouse/pull/79864) ([filimonov](https://github.com/filimonov)).
 * Speedup queries with trivial count optimization. [#79945](https://github.com/ClickHouse/ClickHouse/pull/79945) ([Raúl Marín](https://github.com/Algunenano)).
@@ -73,7 +72,6 @@
 * Lazy Materialization with parallel replicas. [#79401](https://github.com/ClickHouse/ClickHouse/pull/79401) ([Igor Nikonov](https://github.com/devcrafter)).
 
 #### Improvement
-* Add table settings for `SASL` configuration and credentials to the `Kafka` table engine. This allows configuring SASL-based authentication to Kafka and Kafka-compatible systems directly in the CREATE TABLE statement rather than having to use configuration files or named collections. [#78810](https://github.com/ClickHouse/ClickHouse/pull/78810) ([Christoph Wurm](https://github.com/cwurm)).
 * Added an ability to apply lightweight deletes on the fly (with settings `lightweight_deletes_sync = 0`, `apply_mutations_on_fly = 1`. [#79281](https://github.com/ClickHouse/ClickHouse/pull/79281) ([Anton Popov](https://github.com/CurtizJ)).
 * If data in the pretty format is displayed in the terminal, and a subsequent block has the same column widths, it can continue from the previous block, glue it to the previous block by moving the cursor up. This closes [#79333](https://github.com/ClickHouse/ClickHouse/issues/79333). The feature is controlled by the new setting, `output_format_pretty_glue_chunks`. [#79339](https://github.com/ClickHouse/ClickHouse/pull/79339) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
 * Extend the `isIPAddressInRange` function to `String`, `IPv4`, `IPv6`, `Nullable(String)`, `Nullable(IPv4)`, and `Nullable(IPv6)` data types. [#78364](https://github.com/ClickHouse/ClickHouse/pull/78364) ([YjyJeff](https://github.com/YjyJeff)).

ci/docker/integration/integration-test/requirements.txt

@@ -18,8 +18,8 @@ pycurl==7.45.3
 PyJWT==2.3.0
 pyparsing==2.4.7
 SecretStorage==3.3.1
-setuptools==59.6.0
+setuptools==78.1.1
 six==1.16.0
 wadllib==1.3.6
-wheel==0.37.1
+wheel==0.46.1
 zipp==1.0.0

ci/jobs/performance_tests.py

@@ -165,6 +165,7 @@ def run_test(
                 --profile-seconds 10 \
                 {test_file}",
             verbose=True,
+            strip=False,
         )
         duration = sw.duration
         if res != 0:

ci/jobs/scripts/check_style/aspell-ignore/en/aspell-dict.txt

@@ -1437,6 +1437,7 @@ bitmaskToArray
 bitmaskToList
 bitov
 blake
+blobpath
 blockNumber
 blockSerializedSize
 blockSize

ci/jobs/scripts/check_style/check_cpp.sh

@@ -341,3 +341,32 @@ find $ROOT_PATH/{src,base,programs,utils} -name '*.h' -or -name '*.cpp' | grep -
 do
     echo "Found the usage of std::format in '${file}'. Please use fmt::format instead"
 done
+
+# Context.h (and a few similar headers) is included in many parts of the
+# codebase, so any modifications to it trigger a large-scale recompilation.
+# Therefore, it is crucial to avoid unnecessary inclusion of Context.h in
+# headers.
+#
+# In most cases, we can include Context_fwd.h instead, as we usually do not
+# need the full definition of the Context structure in headers - only declaration.
+CONTEXT_H_EXCLUDES=(
+    # For now we have few exceptions (somewhere due to templated code, in other
+    # places just because for now it does not worth it, i.e. the header is not
+    # too generic):
+    --exclude "$ROOT_PATH/src/BridgeHelper/XDBCBridgeHelper.h"
+    --exclude "$ROOT_PATH/src/Interpreters/AddDefaultDatabaseVisitor.h"
+    --exclude "$ROOT_PATH/src/TableFunctions/ITableFunctionCluster.h"
+    --exclude "$ROOT_PATH/src/Core/PostgreSQLProtocol.h"
+    --exclude "$ROOT_PATH/src/Client/ClientBase.h"
+    --exclude "$ROOT_PATH/src/Common/tests/gtest_global_context.h"
+    --exclude "$ROOT_PATH/src/Analyzer/InDepthQueryTreeVisitor.h"
+
+    # For functions we allow it for regular functions (due to lots of
+    # templates), but forbid it in interface (IFunction) part.
+    --exclude "$ROOT_PATH/src/Functions/*"
+    --include "$ROOT_PATH/src/Functions/IFunction*"
+)
+find $ROOT_PATH/src -name '*.h' -print0 | xargs -0 grep -P '#include[\s]*(<|")Interpreters/Context.h(>|")' "${CONTEXT_H_EXCLUDES[@]}" | \
+    grep . && echo '^ Too broad Context.h usage. Consider using Context_fwd.h and Context.h out from .h into .cpp'
+
+exit 0

ci/jobs/scripts/workflow_hooks/trusted.py

@@ -7,7 +7,6 @@
     e.lower()
     for e in [
         "amosbird",
-        "azat",
         "den-crane",  # Documentation contributor
         "taiyang-li",
         "ucasFL",  # Amos Bird's friend

ci/praktika/utils.py

@@ -182,7 +182,7 @@ def get_output(cls, command, strict=False, verbose=False):
         return res.stdout.strip()
 
     @classmethod
-    def get_res_stdout_stderr(cls, command, verbose=True):
+    def get_res_stdout_stderr(cls, command, verbose=True, strip=True):
         if verbose:
             print(f"Run command [{command}]")
         res = subprocess.run(
@@ -192,7 +192,10 @@ def get_res_stdout_stderr(cls, command, verbose=True):
             stderr=subprocess.PIPE,
             text=True,
         )
-        return res.returncode, res.stdout.strip(), res.stderr.strip()
+        if strip:
+            return res.returncode, res.stdout.strip(), res.stderr.strip()
+        else:
+            return res.returncode, res.stdout, res.stderr
 
     @classmethod
     def check(

docker/test/fuzzer/requirements.txt

@@ -19,8 +19,8 @@ pipdeptree==2.23.0
 PyJWT==2.3.0
 pyparsing==2.4.7
 SecretStorage==3.3.1
-setuptools==59.6.0
+setuptools==78.1.1
 six==1.16.0
 wadllib==1.3.6
-wheel==0.37.1
+wheel==0.46.1
 zipp==1.0.0

docker/test/fuzzer/run-fuzzer.sh

@@ -101,6 +101,8 @@ function configure
     mkdir db ||:
     cp -av --dereference "$repo_dir"/programs/server/config* db
     cp -av --dereference "$repo_dir"/programs/server/user* db
+    # for log export
+    cp -av --dereference "$repo_dir"/tests/config/users.d/ci_logs_sender.yaml db/users.d/
     # TODO figure out which ones are needed
     cp -av --dereference "$repo_dir"/tests/config/config.d/listen.xml db/config.d
     cp -av --dereference "$script_dir"/query-fuzzer-tweaks-users.xml db/users.d

docker/test/integration/base/requirements.txt

@@ -18,8 +18,8 @@ pycurl==7.45.3
 PyJWT==2.3.0
 pyparsing==2.4.7
 SecretStorage==3.3.1
-setuptools==59.6.0
+setuptools==78.1.1
 six==1.16.0
 wadllib==1.3.6
-wheel==0.37.1
+wheel==0.46.1
 zipp==1.0.0