Merge branch 'ClickHouse:master' into array_functions_part_8

Author: Blargian

.gitmodules

@@ -296,9 +296,6 @@
 [submodule "contrib/aws-c-compression"]
 	path = contrib/aws-c-compression
 	url = https://github.com/awslabs/aws-c-compression
-[submodule "contrib/aws-s2n-tls"]
-	path = contrib/aws-s2n-tls
-	url = https://github.com/ClickHouse/s2n-tls
 [submodule "contrib/crc32-vpmsum"]
 	path = contrib/crc32-vpmsum
 	url = https://github.com/antonblanchard/crc32-vpmsum.git

CHANGELOG.md

@@ -26,9 +26,7 @@
 * If reading from an `URL` involves multiple redirects, setting `enable_url_encoding` is correctly applied across all redirects in the chain. [#79563](https://github.com/ClickHouse/ClickHouse/pull/79563) ([Shankar Iyer](https://github.com/shankar-iyer)). Setting `enble_url_encoding` default value is now set to `false`. [#80088](https://github.com/ClickHouse/ClickHouse/pull/80088) ([Shankar Iyer](https://github.com/shankar-iyer)).
 
 #### New Feature
-* New `Time`/`Time64` data types: `Time` (HHH:MM:SS) and `Time64` (HHH:MM:SS.`<fractional>`) and some basic cast functions and functions to interact with other data types. Also, changed the existing function's name toTime to toTimeWithFixedDate because the function toTime is required for the cast function. [#75735](https://github.com/ClickHouse/ClickHouse/pull/75735) ([Yarik Briukhovetskyi](https://github.com/yariks5s)).
-72459). [#76078](https://github.com/ClickHouse/ClickHouse/pull/76078) ([Dmitry Novik](https://github.com/novikd)).
-* Support scalar correlated subqueries in the WHERE clause. Closes [#6697](https://github.com/ClickHouse/ClickHouse/issues/6697). [#79600](https://github.com/ClickHouse/ClickHouse/pull/79600) ([Dmitry Novik](https://github.com/novikd)). Support correlated subqueries in the projection list in simple cases. [#79925](https://github.com/ClickHouse/ClickHouse/pull/79925) ([Dmitry Novik](https://github.com/novikd)). Now it covers 100% of TPC-H test suite.
+* Support scalar correlated subqueries in the WHERE clause. Closes [#6697](https://github.com/ClickHouse/ClickHouse/issues/6697). [#79600](https://github.com/ClickHouse/ClickHouse/pull/79600) ([Dmitry Novik](https://github.com/novikd)). Support correlated subqueries in the projection list in simple cases. [#79925](https://github.com/ClickHouse/ClickHouse/pull/79925) ([Dmitry Novik](https://github.com/novikd)). [#76078](https://github.com/ClickHouse/ClickHouse/pull/76078) ([Dmitry Novik](https://github.com/novikd)). Now it covers 100% of TPC-H test suite.
 * Vector search using the vector similarity index is now beta (from previously experimental). [#80164](https://github.com/ClickHouse/ClickHouse/pull/80164) ([Robert Schulze](https://github.com/rschu1ze)).
 * Support geo types in `Parquet` format. This closes [#75317](https://github.com/ClickHouse/ClickHouse/issues/75317). [#79777](https://github.com/ClickHouse/ClickHouse/pull/79777) ([scanhex12](https://github.com/scanhex12)).
 * New functions `sparseGrams`, `sparseGramsHashes`, `sparseGramsHashesUTF8`, `sparseGramsUTF8` for calculating "sparse-ngrams" - a robust algorithm for extracting substrings for indexing and search. [#79517](https://github.com/ClickHouse/ClickHouse/pull/79517) ([scanhex12](https://github.com/scanhex12)).
@@ -44,6 +42,8 @@
 * Add [`icebergHash`](https://iceberg.apache.org/spec/#appendix-b-32-bit-hash-requirements) and [`icebergBucketTransform`](https://iceberg.apache.org/spec/#bucket-transform-details) functions. Support data files pruning in `Iceberg` tables partitioned with [`bucket transfom`](https://iceberg.apache.org/spec/#partitioning). [#79262](https://github.com/ClickHouse/ClickHouse/pull/79262) ([Daniil Ivanik](https://github.com/divanik)).
 
 #### Experimental Feature
+* New `Time`/`Time64` data types: `Time` (HHH:MM:SS) and `Time64` (HHH:MM:SS.`<fractional>`) and some basic cast functions and functions to interact with other data types. Also, changed the existing function's name toTime to toTimeWithFixedDate because the function toTime is required for the cast function. [#75735](https://github.com/ClickHouse/ClickHouse/pull/75735) ([Yarik Briukhovetskyi](https://github.com/yariks5s)).
+72459).
 * Hive metastore catalog for Iceberg datalake. [#77677](https://github.com/ClickHouse/ClickHouse/pull/77677) ([scanhex12](https://github.com/scanhex12)).
 * Indexes of type `full_text` were renamed to `gin`. This follows the more familiar terminology of PostgreSQL and other databases. Existing indexes of type `full_text` remain loadable but they will throw an exception (suggesting `gin` indexes instead) when one tries to use them in searches. [#79024](https://github.com/ClickHouse/ClickHouse/pull/79024) ([Robert Schulze](https://github.com/rschu1ze)).
 
@@ -60,7 +60,6 @@
 * Allow parallel merging of `uniqExact` states during the final stage of distributed aggregation. [#78703](https://github.com/ClickHouse/ClickHouse/pull/78703) ([Nikita Taranov](https://github.com/nickitat)).
 * Fix possible performance degradation of the parallel merging of `uniqExact` states for aggregation with key. [#78724](https://github.com/ClickHouse/ClickHouse/pull/78724) ([Nikita Taranov](https://github.com/nickitat)).
 * Reduce the number of List Blobs API calls to Azure storage. [#78860](https://github.com/ClickHouse/ClickHouse/pull/78860) ([Julia Kartseva](https://github.com/jkartseva)).
-* Merge equality conditions from filter query plan step into JOIN condition if possible to allow using them as hash table keys. [#78877](https://github.com/ClickHouse/ClickHouse/pull/78877) ([Dmitry Novik](https://github.com/novikd)).
 * Fix performance of the distributed INSERT SELECT with parallel replicas. [#79441](https://github.com/ClickHouse/ClickHouse/pull/79441) ([Azat Khuzhin](https://github.com/azat)).
 * Prevent `LogSeriesLimiter` from doing cleanup on every construction, avoiding lock contention and performance regressions in high-concurrency scenarios. [#79864](https://github.com/ClickHouse/ClickHouse/pull/79864) ([filimonov](https://github.com/filimonov)).
 * Speedup queries with trivial count optimization. [#79945](https://github.com/ClickHouse/ClickHouse/pull/79945) ([Raúl Marín](https://github.com/Algunenano)).
@@ -73,7 +72,6 @@
 * Lazy Materialization with parallel replicas. [#79401](https://github.com/ClickHouse/ClickHouse/pull/79401) ([Igor Nikonov](https://github.com/devcrafter)).
 
 #### Improvement
-* Add table settings for `SASL` configuration and credentials to the `Kafka` table engine. This allows configuring SASL-based authentication to Kafka and Kafka-compatible systems directly in the CREATE TABLE statement rather than having to use configuration files or named collections. [#78810](https://github.com/ClickHouse/ClickHouse/pull/78810) ([Christoph Wurm](https://github.com/cwurm)).
 * Added an ability to apply lightweight deletes on the fly (with settings `lightweight_deletes_sync = 0`, `apply_mutations_on_fly = 1`. [#79281](https://github.com/ClickHouse/ClickHouse/pull/79281) ([Anton Popov](https://github.com/CurtizJ)).
 * If data in the pretty format is displayed in the terminal, and a subsequent block has the same column widths, it can continue from the previous block, glue it to the previous block by moving the cursor up. This closes [#79333](https://github.com/ClickHouse/ClickHouse/issues/79333). The feature is controlled by the new setting, `output_format_pretty_glue_chunks`. [#79339](https://github.com/ClickHouse/ClickHouse/pull/79339) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
 * Extend the `isIPAddressInRange` function to `String`, `IPv4`, `IPv6`, `Nullable(String)`, `Nullable(IPv4)`, and `Nullable(IPv6)` data types. [#78364](https://github.com/ClickHouse/ClickHouse/pull/78364) ([YjyJeff](https://github.com/YjyJeff)).

base/poco/NetSSL_OpenSSL/include/Poco/Net/SecureSocketImpl.h

@@ -236,21 +236,16 @@ namespace Net
         /// to be able to re-use it again.
 
     private:
-        using MutexT = Poco::FastMutex;
-        using LockT = MutexT::ScopedLock;
-        using UnLockT = Poco::ScopedLockWithUnlock<MutexT>;
-
         SecureSocketImpl(const SecureSocketImpl &);
         SecureSocketImpl & operator=(const SecureSocketImpl &);
 
         mutable std::recursive_mutex _mutex;
-        std::atomic<SSL *> _pSSL;
+        SSL * _pSSL; // GUARDED_BY _mutex
         Poco::AutoPtr<SocketImpl> _pSocket;
         Context::Ptr _pContext;
         bool _needHandshake;
         std::string _peerHostName;
         Session::Ptr _pSession;
-        mutable MutexT _ssl_mutex;
 
         friend class SecureStreamSocketImpl;
 

base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp

@@ -103,8 +103,6 @@ void SecureSocketImpl::acceptSSL()
 	std::lock_guard<std::recursive_mutex> lock(_mutex);
 	poco_assert (!_pSSL);
 
-	LockT l(_ssl_mutex);
-
 	BIO* pBIO = BIO_new(BIO_s_socket());
 	if (!pBIO) throw SSLException("Cannot create BIO object");
 	BIO_set_fd(pBIO, static_cast<int>(_pSocket->sockfd()), BIO_NOCLOSE);
@@ -171,8 +169,6 @@ void SecureSocketImpl::connectSSL(bool performHandshake)
 	poco_assert (!_pSSL);
 	poco_assert (_pSocket->initialized());
 
-	LockT l(_ssl_mutex);
-
 	BIO* pBIO = BIO_new(BIO_s_socket());
 	if (!pBIO) throw SSLException("Cannot create SSL BIO object");
 	BIO_set_fd(pBIO, static_cast<int>(_pSocket->sockfd()), BIO_NOCLOSE);
@@ -250,8 +246,6 @@ void SecureSocketImpl::shutdown()
 	std::lock_guard<std::recursive_mutex> lock(_mutex);
 	if (_pSSL)
 	{
-        UnLockT l(_ssl_mutex);
-
         // Don't shut down the socket more than once.
         int shutdownState = SSL_get_shutdown(_pSSL);
         bool shutdownSent = (shutdownState & SSL_SENT_SHUTDOWN) == SSL_SENT_SHUTDOWN;
@@ -266,7 +260,6 @@ void SecureSocketImpl::shutdown()
 			// done with it.
 			int rc = SSL_shutdown(_pSSL);
 			if (rc < 0) handleError(rc);
-			l.unlock();
 			if (_pSocket->getBlocking())
 			{
 				_pSocket->shutdown();
@@ -297,9 +290,6 @@ int SecureSocketImpl::sendBytes(const void* buffer, int length, int flags)
 	poco_check_ptr (_pSSL);
 
 	int rc;
-
-	LockT l(_ssl_mutex);
-
 	if (_needHandshake)
 	{
 		rc = completeHandshake();
@@ -341,8 +331,6 @@ int SecureSocketImpl::receiveBytes(void* buffer, int length, int flags)
 	poco_assert (_pSocket->initialized());
 	poco_check_ptr (_pSSL);
 
-	LockT l(_ssl_mutex);
-
 	/// Special case: just check that we can read from socket
 	if ((flags & MSG_DONTWAIT) && (flags & MSG_PEEK))
 		return _pSocket->receiveBytes(buffer, length, flags);
@@ -380,8 +368,6 @@ int SecureSocketImpl::available() const
 	std::lock_guard<std::recursive_mutex> lock(_mutex);
 	poco_check_ptr (_pSSL);
 
-	LockT l(_ssl_mutex);
-
 	return SSL_pending(_pSSL);
 }
 
@@ -478,20 +464,10 @@ bool SecureSocketImpl::isLocalHost(const std::string& hostName)
 X509* SecureSocketImpl::peerCertificate() const
 {
 	std::lock_guard<std::recursive_mutex> lock(_mutex);
-    LockT l(_ssl_mutex);
-
-	X509* pCert = nullptr;
-
 	if (_pSSL)
-	{
-		pCert = ::SSL_get_peer_certificate(_pSSL);
-
-		if (X509_V_OK != SSL_get_verify_result(_pSSL))
-			throw CertificateValidationException("SecureSocketImpl::peerCertificate(): "
-				"Certificate verification error " + Utility::getLastError());
-	}
-
-	return pCert;
+		return SSL_get1_peer_certificate(_pSSL);
+	else
+		return 0;
 }
 
 Poco::Timespan SecureSocketImpl::getMaxTimeoutOrLimit()
@@ -632,8 +608,6 @@ void SecureSocketImpl::reset()
 	close();
 	if (_pSSL)
 	{
-		LockT l(_ssl_mutex);
-
 		SSL_free(_pSSL);
 		_pSSL = nullptr;
 	}
@@ -678,12 +652,9 @@ bool SecureSocketImpl::sessionWasReused()
 {
 	std::lock_guard<std::recursive_mutex> lock(_mutex);
 	if (_pSSL)
-	{
-		LockT l(_ssl_mutex);
-		return ::SSL_session_reused(_pSSL) != 0;
-	}
-
-	return false;
+		return SSL_session_reused(_pSSL) != 0;
+	else
+		return false;
 }
 
 void SecureSocketImpl::setBlocking(bool flag)

ci/docker/integration/integration-test/requirements.txt

@@ -18,8 +18,8 @@ pycurl==7.45.3
 PyJWT==2.3.0
 pyparsing==2.4.7
 SecretStorage==3.3.1
-setuptools==59.6.0
+setuptools==78.1.1
 six==1.16.0
 wadllib==1.3.6
-wheel==0.37.1
+wheel==0.46.1
 zipp==1.0.0

ci/jobs/performance_tests.py

@@ -165,6 +165,7 @@ def run_test(
                 --profile-seconds 10 \
                 {test_file}",
             verbose=True,
+            strip=False,
         )
         duration = sw.duration
         if res != 0:

ci/jobs/scripts/check_style/aspell-ignore/en/aspell-dict.txt

@@ -1431,6 +1431,7 @@ bitmaskToArray
 bitmaskToList
 bitov
 blake
+blobpath
 blockNumber
 blockSerializedSize
 blockSize

ci/jobs/scripts/workflow_hooks/trusted.py

@@ -7,7 +7,6 @@
     e.lower()
     for e in [
         "amosbird",
-        "azat",
         "den-crane",  # Documentation contributor
         "taiyang-li",
         "ucasFL",  # Amos Bird's friend

ci/praktika/utils.py

@@ -182,7 +182,7 @@ def get_output(cls, command, strict=False, verbose=False):
         return res.stdout.strip()
 
     @classmethod
-    def get_res_stdout_stderr(cls, command, verbose=True):
+    def get_res_stdout_stderr(cls, command, verbose=True, strip=True):
         if verbose:
             print(f"Run command [{command}]")
         res = subprocess.run(
@@ -192,7 +192,10 @@ def get_res_stdout_stderr(cls, command, verbose=True):
             stderr=subprocess.PIPE,
             text=True,
         )
-        return res.returncode, res.stdout.strip(), res.stderr.strip()
+        if strip:
+            return res.returncode, res.stdout.strip(), res.stderr.strip()
+        else:
+            return res.returncode, res.stdout, res.stderr
 
     @classmethod
     def check(

contrib/CMakeLists.txt

@@ -122,7 +122,6 @@ add_contrib (aws-cmake
     aws-c-mqtt
     aws-c-s3
     aws-c-sdkutils
-    aws-s2n-tls
     aws-checksums
     aws-crt-cpp
     aws-cmake