Merge remote-tracking branch 'u/master' into HEAD

Author: azat

.gitmodules

@@ -296,9 +296,6 @@
 [submodule "contrib/aws-c-compression"]
 	path = contrib/aws-c-compression
 	url = https://github.com/awslabs/aws-c-compression
-[submodule "contrib/aws-s2n-tls"]
-	path = contrib/aws-s2n-tls
-	url = https://github.com/ClickHouse/s2n-tls
 [submodule "contrib/crc32-vpmsum"]
 	path = contrib/crc32-vpmsum
 	url = https://github.com/antonblanchard/crc32-vpmsum.git

CHANGELOG.md

@@ -18,32 +18,31 @@
 ### <a id="255"></a> ClickHouse release 25.5, 2025-05-22
 
 #### Backward Incompatible Change
-* Function `geoToH3` now accepts the input in the order (lat, lon, res) (which is common for other geometric functions). Users who wish to retain the previous result order (lon, lat, res) can set setting `geotoh3_lon_lat_input_order = true`. [#78852](https://github.com/ClickHouse/ClickHouse/pull/78852) ([Pratima Patel](https://github.com/pratimapatel2008)).
+* Function `geoToH3` now accepts the input in the order (lat, lon, res) (which is common for other geometric functions). Users who wish to retain the previous result order (lon, lat, res) can set setting `geotoh3_argument_order = 'lon_lat'`. [#78852](https://github.com/ClickHouse/ClickHouse/pull/78852) ([Pratima Patel](https://github.com/pratimapatel2008)).
 * Add a filesystem cache setting `allow_dynamic_cache_resize`, by default `false`, to allow dynamic resize of filesystem cache. Why: in certain environments (ClickHouse Cloud) all the scaling events happen through the restart of the process and we would love this feature to be explicitly disabled to have more control over the behaviour + as a safety measure. This PR is marked as backward incompatible, because in older versions dynamic cache resize worked by default without special setting. [#79148](https://github.com/ClickHouse/ClickHouse/pull/79148) ([Kseniia Sumarokova](https://github.com/kssenii)).
 * Removed support for legacy index types `annoy` and `usearch`. Both have been stubs for a long time, i.e. every attempt to use the legacy indexes returned an error anyways. If you still have `annoy` and `usearch` indexes, please drop them. [#79802](https://github.com/ClickHouse/ClickHouse/pull/79802) ([Robert Schulze](https://github.com/rschu1ze)).
 * Remove `format_alter_commands_with_parentheses` server setting. The setting was introduced and disabled by default in 24.2. It was enabled by default in 25.2. As there are no LTS versions that don't support the new format, we can remove the setting. [#79970](https://github.com/ClickHouse/ClickHouse/pull/79970) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)).
 * Enable `DeltaLake` storage `delta-kernel-rs` implementation by default. [#79541](https://github.com/ClickHouse/ClickHouse/pull/79541) ([Kseniia Sumarokova](https://github.com/kssenii)).
 * If reading from an `URL` involves multiple redirects, setting `enable_url_encoding` is correctly applied across all redirects in the chain. [#79563](https://github.com/ClickHouse/ClickHouse/pull/79563) ([Shankar Iyer](https://github.com/shankar-iyer)). Setting `enble_url_encoding` default value is now set to `false`. [#80088](https://github.com/ClickHouse/ClickHouse/pull/80088) ([Shankar Iyer](https://github.com/shankar-iyer)).
 
 #### New Feature
-* New `Time`/`Time64` data types: `Time` (HHH:MM:SS) and `Time64` (HHH:MM:SS.`<fractional>`) and some basic cast functions and functions to interact with other data types. Also, changed the existing function's name toTime to toTimeWithFixedDate because the function toTime is required for the cast function. [#75735](https://github.com/ClickHouse/ClickHouse/pull/75735) ([Yarik Briukhovetskyi](https://github.com/yariks5s)).
-72459). [#76078](https://github.com/ClickHouse/ClickHouse/pull/76078) ([Dmitry Novik](https://github.com/novikd)).
-* Support scalar correlated subqueries in the WHERE clause. Closes [#6697](https://github.com/ClickHouse/ClickHouse/issues/6697). [#79600](https://github.com/ClickHouse/ClickHouse/pull/79600) ([Dmitry Novik](https://github.com/novikd)). Support correlated subqueries in the projection list in simple cases. [#79925](https://github.com/ClickHouse/ClickHouse/pull/79925) ([Dmitry Novik](https://github.com/novikd)). Now it covers 100% of TPC-H test suite.
+* Support scalar correlated subqueries in the WHERE clause. Closes [#6697](https://github.com/ClickHouse/ClickHouse/issues/6697). [#79600](https://github.com/ClickHouse/ClickHouse/pull/79600) ([Dmitry Novik](https://github.com/novikd)). Support correlated subqueries in the projection list in simple cases. [#79925](https://github.com/ClickHouse/ClickHouse/pull/79925) ([Dmitry Novik](https://github.com/novikd)). [#76078](https://github.com/ClickHouse/ClickHouse/pull/76078) ([Dmitry Novik](https://github.com/novikd)). Now it covers 100% of TPC-H test suite.
 * Vector search using the vector similarity index is now beta (from previously experimental). [#80164](https://github.com/ClickHouse/ClickHouse/pull/80164) ([Robert Schulze](https://github.com/rschu1ze)).
 * Support geo types in `Parquet` format. This closes [#75317](https://github.com/ClickHouse/ClickHouse/issues/75317). [#79777](https://github.com/ClickHouse/ClickHouse/pull/79777) ([scanhex12](https://github.com/scanhex12)).
 * New functions `sparseGrams`, `sparseGramsHashes`, `sparseGramsHashesUTF8`, `sparseGramsUTF8` for calculating "sparse-ngrams" - a robust algorithm for extracting substrings for indexing and search. [#79517](https://github.com/ClickHouse/ClickHouse/pull/79517) ([scanhex12](https://github.com/scanhex12)).
 * `clickhouse-local` (and its shorthand alias, `ch`) now use an implicit `FROM table` when there is input data for processing. This closes [#65023](https://github.com/ClickHouse/ClickHouse/issues/65023). Also enabled format inference in clickhouse-local if `--input-format` is not specified and it processes a regular file. [#79085](https://github.com/ClickHouse/ClickHouse/pull/79085) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
 * Add `stringBytesUniq` and `stringBytesEntropy` functions to search for possibly random or encrypted data. [#79350](https://github.com/ClickHouse/ClickHouse/pull/79350) ([Sachin Kumar Singh](https://github.com/sachinkumarsingh092)).
 * Added functions for encoding and decoding base32. [#79809](https://github.com/ClickHouse/ClickHouse/pull/79809) ([Joanna Hulboj](https://github.com/jh0x)).
 * Add `getServerSetting` and `getMergeTreeSetting` function. Closes #78318. [#78439](https://github.com/ClickHouse/ClickHouse/pull/78439) ([NamNguyenHoai](https://github.com/NamHoaiNguyen)).
-* Add `system.iceberg_history` table. [#78244](https://github.com/ClickHouse/ClickHouse/pull/78244) ([Smita Kulkarni](https://github.com/SmitaRKulkarni)).
 * Add new `iceberg_enable_version_hint` setting to leverage `version-hint.text` file. [#78594](https://github.com/ClickHouse/ClickHouse/pull/78594) ([Arnaud Briche](https://github.com/arnaudbriche)).
 * Gives the possibility to truncate specific tables from a database, filtered with the `LIKE` keyword. [#78597](https://github.com/ClickHouse/ClickHouse/pull/78597) ([Yarik Briukhovetskyi](https://github.com/yariks5s)).
 * Support `_part_starting_offset` virtual column in `MergeTree`-family tables. This column represents the cumulative row count of all preceding parts, calculated at query time based on the current part list. The cumulative values are retained throughout query execution and remain effective even after part pruning. Related internal logic has been refactored to support this behavior. [#79417](https://github.com/ClickHouse/ClickHouse/pull/79417) ([Amos Bird](https://github.com/amosbird)).
 * Add functions `divideOrNull`,`moduloOrNull`, `intDivOrNull`,`positiveModuloOrNull` to return NULL when right argument is zero. [#78276](https://github.com/ClickHouse/ClickHouse/pull/78276) ([kevinyhzou](https://github.com/KevinyhZou)).
-* TODO: WTF is that? Explain it in a way your gradma will understand. Support for Iceberg partition pruning bucket transform. [#79262](https://github.com/ClickHouse/ClickHouse/pull/79262) ([Daniil Ivanik](https://github.com/divanik)).
+* Add [`icebergHash`](https://iceberg.apache.org/spec/#appendix-b-32-bit-hash-requirements) and [`icebergBucket`](https://iceberg.apache.org/spec/#bucket-transform-details) functions. Support data files pruning in `Iceberg` tables partitioned with [`bucket transfom`](https://iceberg.apache.org/spec/#partitioning). [#79262](https://github.com/ClickHouse/ClickHouse/pull/79262) ([Daniil Ivanik](https://github.com/divanik)).
 
 #### Experimental Feature
+* New `Time`/`Time64` data types: `Time` (HHH:MM:SS) and `Time64` (HHH:MM:SS.`<fractional>`) and some basic cast functions and functions to interact with other data types. Also, changed the existing function's name toTime to toTimeWithFixedDate because the function toTime is required for the cast function. [#75735](https://github.com/ClickHouse/ClickHouse/pull/75735) ([Yarik Briukhovetskyi](https://github.com/yariks5s)).
+72459).
 * Hive metastore catalog for Iceberg datalake. [#77677](https://github.com/ClickHouse/ClickHouse/pull/77677) ([scanhex12](https://github.com/scanhex12)).
 * Indexes of type `full_text` were renamed to `gin`. This follows the more familiar terminology of PostgreSQL and other databases. Existing indexes of type `full_text` remain loadable but they will throw an exception (suggesting `gin` indexes instead) when one tries to use them in searches. [#79024](https://github.com/ClickHouse/ClickHouse/pull/79024) ([Robert Schulze](https://github.com/rschu1ze)).
 
@@ -60,7 +59,6 @@
 * Allow parallel merging of `uniqExact` states during the final stage of distributed aggregation. [#78703](https://github.com/ClickHouse/ClickHouse/pull/78703) ([Nikita Taranov](https://github.com/nickitat)).
 * Fix possible performance degradation of the parallel merging of `uniqExact` states for aggregation with key. [#78724](https://github.com/ClickHouse/ClickHouse/pull/78724) ([Nikita Taranov](https://github.com/nickitat)).
 * Reduce the number of List Blobs API calls to Azure storage. [#78860](https://github.com/ClickHouse/ClickHouse/pull/78860) ([Julia Kartseva](https://github.com/jkartseva)).
-* Merge equality conditions from filter query plan step into JOIN condition if possible to allow using them as hash table keys. [#78877](https://github.com/ClickHouse/ClickHouse/pull/78877) ([Dmitry Novik](https://github.com/novikd)).
 * Fix performance of the distributed INSERT SELECT with parallel replicas. [#79441](https://github.com/ClickHouse/ClickHouse/pull/79441) ([Azat Khuzhin](https://github.com/azat)).
 * Prevent `LogSeriesLimiter` from doing cleanup on every construction, avoiding lock contention and performance regressions in high-concurrency scenarios. [#79864](https://github.com/ClickHouse/ClickHouse/pull/79864) ([filimonov](https://github.com/filimonov)).
 * Speedup queries with trivial count optimization. [#79945](https://github.com/ClickHouse/ClickHouse/pull/79945) ([Raúl Marín](https://github.com/Algunenano)).
@@ -73,7 +71,6 @@
 * Lazy Materialization with parallel replicas. [#79401](https://github.com/ClickHouse/ClickHouse/pull/79401) ([Igor Nikonov](https://github.com/devcrafter)).
 
 #### Improvement
-* Add table settings for `SASL` configuration and credentials to the `Kafka` table engine. This allows configuring SASL-based authentication to Kafka and Kafka-compatible systems directly in the CREATE TABLE statement rather than having to use configuration files or named collections. [#78810](https://github.com/ClickHouse/ClickHouse/pull/78810) ([Christoph Wurm](https://github.com/cwurm)).
 * Added an ability to apply lightweight deletes on the fly (with settings `lightweight_deletes_sync = 0`, `apply_mutations_on_fly = 1`. [#79281](https://github.com/ClickHouse/ClickHouse/pull/79281) ([Anton Popov](https://github.com/CurtizJ)).
 * If data in the pretty format is displayed in the terminal, and a subsequent block has the same column widths, it can continue from the previous block, glue it to the previous block by moving the cursor up. This closes [#79333](https://github.com/ClickHouse/ClickHouse/issues/79333). The feature is controlled by the new setting, `output_format_pretty_glue_chunks`. [#79339](https://github.com/ClickHouse/ClickHouse/pull/79339) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
 * Extend the `isIPAddressInRange` function to `String`, `IPv4`, `IPv6`, `Nullable(String)`, `Nullable(IPv4)`, and `Nullable(IPv6)` data types. [#78364](https://github.com/ClickHouse/ClickHouse/pull/78364) ([YjyJeff](https://github.com/YjyJeff)).

base/poco/NetSSL_OpenSSL/include/Poco/Net/SecureSocketImpl.h

@@ -236,21 +236,16 @@ namespace Net
         /// to be able to re-use it again.
 
     private:
-        using MutexT = Poco::FastMutex;
-        using LockT = MutexT::ScopedLock;
-        using UnLockT = Poco::ScopedLockWithUnlock<MutexT>;
-
         SecureSocketImpl(const SecureSocketImpl &);
         SecureSocketImpl & operator=(const SecureSocketImpl &);
 
         mutable std::recursive_mutex _mutex;
-        std::atomic<SSL *> _pSSL;
+        SSL * _pSSL; // GUARDED_BY _mutex
         Poco::AutoPtr<SocketImpl> _pSocket;
         Context::Ptr _pContext;
         bool _needHandshake;
         std::string _peerHostName;
         Session::Ptr _pSession;
-        mutable MutexT _ssl_mutex;
 
         friend class SecureStreamSocketImpl;
 

base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp

@@ -103,8 +103,6 @@ void SecureSocketImpl::acceptSSL()
 	std::lock_guard<std::recursive_mutex> lock(_mutex);
 	poco_assert (!_pSSL);
 
-	LockT l(_ssl_mutex);
-
 	BIO* pBIO = BIO_new(BIO_s_socket());
 	if (!pBIO) throw SSLException("Cannot create BIO object");
 	BIO_set_fd(pBIO, static_cast<int>(_pSocket->sockfd()), BIO_NOCLOSE);
@@ -171,8 +169,6 @@ void SecureSocketImpl::connectSSL(bool performHandshake)
 	poco_assert (!_pSSL);
 	poco_assert (_pSocket->initialized());
 
-	LockT l(_ssl_mutex);
-
 	BIO* pBIO = BIO_new(BIO_s_socket());
 	if (!pBIO) throw SSLException("Cannot create SSL BIO object");
 	BIO_set_fd(pBIO, static_cast<int>(_pSocket->sockfd()), BIO_NOCLOSE);
@@ -250,8 +246,6 @@ void SecureSocketImpl::shutdown()
 	std::lock_guard<std::recursive_mutex> lock(_mutex);
 	if (_pSSL)
 	{
-        UnLockT l(_ssl_mutex);
-
         // Don't shut down the socket more than once.
         int shutdownState = SSL_get_shutdown(_pSSL);
         bool shutdownSent = (shutdownState & SSL_SENT_SHUTDOWN) == SSL_SENT_SHUTDOWN;
@@ -266,7 +260,6 @@ void SecureSocketImpl::shutdown()
 			// done with it.
 			int rc = SSL_shutdown(_pSSL);
 			if (rc < 0) handleError(rc);
-			l.unlock();
 			if (_pSocket->getBlocking())
 			{
 				_pSocket->shutdown();
@@ -297,9 +290,6 @@ int SecureSocketImpl::sendBytes(const void* buffer, int length, int flags)
 	poco_check_ptr (_pSSL);
 
 	int rc;
-
-	LockT l(_ssl_mutex);
-
 	if (_needHandshake)
 	{
 		rc = completeHandshake();
@@ -341,8 +331,6 @@ int SecureSocketImpl::receiveBytes(void* buffer, int length, int flags)
 	poco_assert (_pSocket->initialized());
 	poco_check_ptr (_pSSL);
 
-	LockT l(_ssl_mutex);
-
 	/// Special case: just check that we can read from socket
 	if ((flags & MSG_DONTWAIT) && (flags & MSG_PEEK))
 		return _pSocket->receiveBytes(buffer, length, flags);
@@ -380,8 +368,6 @@ int SecureSocketImpl::available() const
 	std::lock_guard<std::recursive_mutex> lock(_mutex);
 	poco_check_ptr (_pSSL);
 
-	LockT l(_ssl_mutex);
-
 	return SSL_pending(_pSSL);
 }
 
@@ -478,20 +464,10 @@ bool SecureSocketImpl::isLocalHost(const std::string& hostName)
 X509* SecureSocketImpl::peerCertificate() const
 {
 	std::lock_guard<std::recursive_mutex> lock(_mutex);
-    LockT l(_ssl_mutex);
-
-	X509* pCert = nullptr;
-
 	if (_pSSL)
-	{
-		pCert = ::SSL_get_peer_certificate(_pSSL);
-
-		if (X509_V_OK != SSL_get_verify_result(_pSSL))
-			throw CertificateValidationException("SecureSocketImpl::peerCertificate(): "
-				"Certificate verification error " + Utility::getLastError());
-	}
-
-	return pCert;
+		return SSL_get1_peer_certificate(_pSSL);
+	else
+		return 0;
 }
 
 Poco::Timespan SecureSocketImpl::getMaxTimeoutOrLimit()
@@ -632,8 +608,6 @@ void SecureSocketImpl::reset()
 	close();
 	if (_pSSL)
 	{
-		LockT l(_ssl_mutex);
-
 		SSL_free(_pSSL);
 		_pSSL = nullptr;
 	}
@@ -678,12 +652,9 @@ bool SecureSocketImpl::sessionWasReused()
 {
 	std::lock_guard<std::recursive_mutex> lock(_mutex);
 	if (_pSSL)
-	{
-		LockT l(_ssl_mutex);
-		return ::SSL_session_reused(_pSSL) != 0;
-	}
-
-	return false;
+		return SSL_session_reused(_pSSL) != 0;
+	else
+		return false;
 }
 
 void SecureSocketImpl::setBlocking(bool flag)

ci/docker/integration/integration-test/requirements.txt

@@ -18,8 +18,8 @@ pycurl==7.45.3
 PyJWT==2.3.0
 pyparsing==2.4.7
 SecretStorage==3.3.1
-setuptools==59.6.0
+setuptools==78.1.1
 six==1.16.0
 wadllib==1.3.6
-wheel==0.37.1
+wheel==0.46.1
 zipp==1.0.0

ci/jobs/performance_tests.py

@@ -165,6 +165,7 @@ def run_test(
                 --profile-seconds 10 \
                 {test_file}",
             verbose=True,
+            strip=False,
         )
         duration = sw.duration
         if res != 0:

ci/jobs/scripts/check_style/aspell-ignore/en/aspell-dict.txt

@@ -1431,6 +1431,7 @@ bitmaskToArray
 bitmaskToList
 bitov
 blake
+blobpath
 blockNumber
 blockSerializedSize
 blockSize

ci/jobs/scripts/check_style/check_cpp.sh

@@ -341,3 +341,32 @@ find $ROOT_PATH/{src,base,programs,utils} -name '*.h' -or -name '*.cpp' | grep -
 do
     echo "Found the usage of std::format in '${file}'. Please use fmt::format instead"
 done
+
+# Context.h (and a few similar headers) is included in many parts of the
+# codebase, so any modifications to it trigger a large-scale recompilation.
+# Therefore, it is crucial to avoid unnecessary inclusion of Context.h in
+# headers.
+#
+# In most cases, we can include Context_fwd.h instead, as we usually do not
+# need the full definition of the Context structure in headers - only declaration.
+CONTEXT_H_EXCLUDES=(
+    # For now we have few exceptions (somewhere due to templated code, in other
+    # places just because for now it does not worth it, i.e. the header is not
+    # too generic):
+    --exclude "$ROOT_PATH/src/BridgeHelper/XDBCBridgeHelper.h"
+    --exclude "$ROOT_PATH/src/Interpreters/AddDefaultDatabaseVisitor.h"
+    --exclude "$ROOT_PATH/src/TableFunctions/ITableFunctionCluster.h"
+    --exclude "$ROOT_PATH/src/Core/PostgreSQLProtocol.h"
+    --exclude "$ROOT_PATH/src/Client/ClientBase.h"
+    --exclude "$ROOT_PATH/src/Common/tests/gtest_global_context.h"
+    --exclude "$ROOT_PATH/src/Analyzer/InDepthQueryTreeVisitor.h"
+
+    # For functions we allow it for regular functions (due to lots of
+    # templates), but forbid it in interface (IFunction) part.
+    --exclude "$ROOT_PATH/src/Functions/*"
+    --include "$ROOT_PATH/src/Functions/IFunction*"
+)
+find $ROOT_PATH/src -name '*.h' -print0 | xargs -0 grep -P '#include[\s]*(<|")Interpreters/Context.h(>|")' "${CONTEXT_H_EXCLUDES[@]}" | \
+    grep . && echo '^ Too broad Context.h usage. Consider using Context_fwd.h and Context.h out from .h into .cpp'
+
+exit 0

ci/jobs/scripts/workflow_hooks/trusted.py

@@ -7,7 +7,6 @@
     e.lower()
     for e in [
         "amosbird",
-        "azat",
         "den-crane",  # Documentation contributor
         "taiyang-li",
         "ucasFL",  # Amos Bird's friend

ci/praktika/utils.py

@@ -182,7 +182,7 @@ def get_output(cls, command, strict=False, verbose=False):
         return res.stdout.strip()
 
     @classmethod
-    def get_res_stdout_stderr(cls, command, verbose=True):
+    def get_res_stdout_stderr(cls, command, verbose=True, strip=True):
         if verbose:
             print(f"Run command [{command}]")
         res = subprocess.run(
@@ -192,7 +192,10 @@ def get_res_stdout_stderr(cls, command, verbose=True):
             stderr=subprocess.PIPE,
             text=True,
         )
-        return res.returncode, res.stdout.strip(), res.stderr.strip()
+        if strip:
+            return res.returncode, res.stdout.strip(), res.stderr.strip()
+        else:
+            return res.returncode, res.stdout, res.stderr
 
     @classmethod
     def check(