Merge remote-tracking branch 'blessed/master' into dependabot/pip/docker/test/integration/base/setuptools-78.1.1

Author: Algunenano

CHANGELOG.md

@@ -41,7 +41,7 @@
 * Gives the possibility to truncate specific tables from a database, filtered with the `LIKE` keyword. [#78597](https://github.com/ClickHouse/ClickHouse/pull/78597) ([Yarik Briukhovetskyi](https://github.com/yariks5s)).
 * Support `_part_starting_offset` virtual column in `MergeTree`-family tables. This column represents the cumulative row count of all preceding parts, calculated at query time based on the current part list. The cumulative values are retained throughout query execution and remain effective even after part pruning. Related internal logic has been refactored to support this behavior. [#79417](https://github.com/ClickHouse/ClickHouse/pull/79417) ([Amos Bird](https://github.com/amosbird)).
 * Add functions `divideOrNull`,`moduloOrNull`, `intDivOrNull`,`positiveModuloOrNull` to return NULL when right argument is zero. [#78276](https://github.com/ClickHouse/ClickHouse/pull/78276) ([kevinyhzou](https://github.com/KevinyhZou)).
-* TODO: WTF is that? Explain it in a way your gradma will understand. Support for Iceberg partition pruning bucket transform. [#79262](https://github.com/ClickHouse/ClickHouse/pull/79262) ([Daniil Ivanik](https://github.com/divanik)).
+* Add [`icebergHash`](https://iceberg.apache.org/spec/#appendix-b-32-bit-hash-requirements) and [`icebergBucketTransform`](https://iceberg.apache.org/spec/#bucket-transform-details) functions. Support data files pruning in `Iceberg` tables partitioned with [`bucket transfom`](https://iceberg.apache.org/spec/#partitioning). [#79262](https://github.com/ClickHouse/ClickHouse/pull/79262) ([Daniil Ivanik](https://github.com/divanik)).
 
 #### Experimental Feature
 * Hive metastore catalog for Iceberg datalake. [#77677](https://github.com/ClickHouse/ClickHouse/pull/77677) ([scanhex12](https://github.com/scanhex12)).

docs/en/operations/settings/constraints-on-settings.md

@@ -46,6 +46,14 @@ Constraints are defined as follows:
         <max>upper_boundary</max>
         <changeable_in_readonly/>
       </setting_name_5>
+      <setting_name_6>
+        <min>lower_boundary</min>
+        <max>upper_boundary</max>
+        <disallowed>value1</disallowed>
+        <disallowed>value2</disallowed>
+        <disallowed>value3</disallowed>
+        <changeable_in_readonly/>
+      </setting_name_6>
     </constraints>
   </user_name>
 </profiles>
@@ -59,12 +67,16 @@ setting remains unchanged.
 There are a few types of constraints supported in ClickHouse:
 - `min`
 - `max`
+- `disallowed`
 - `readonly` (with alias `const`)
 - `changeable_in_readonly`
 
 The `min` and `max` constraints specify upper and lower boundaries for a numeric 
 setting and can be used in combination with each other. 
 
+The `disallowed` constraint can be used to specify specific value(s) which should not
+be allowed for a specific setting.
+
 The `readonly` or `const` constraint specifies that the user cannot change the 
 corresponding setting at all. 
 

docs/en/sql-reference/table-functions/s3Cluster.md

@@ -17,7 +17,7 @@ Allows processing files from [Amazon S3](https://aws.amazon.com/s3/) and Google
 **Syntax**
 
 ```sql
-s3Cluster(cluster_name, url [, NOSIGN | access_key_id, secret_access_key, [session_token]] [,format] [,structure] [,compression_method],[,headers])
+s3Cluster(cluster_name, url[, NOSIGN | access_key_id, secret_access_key,[session_token]][, format][, structure][, compression_method][, headers][, extra_credentials])
 s3Cluster(cluster_name, named_collection[, option=value [,..]])
 ```
 
@@ -31,7 +31,8 @@ s3Cluster(cluster_name, named_collection[, option=value [,..]])
 - `format` — The [format](/sql-reference/formats) of the file.
 - `structure` — Structure of the table. Format `'column1_name column1_type, column2_name column2_type, ...'`.
 - `compression_method` — Parameter is optional. Supported values: `none`, `gzip` or `gz`, `brotli` or `br`, `xz` or `LZMA`, `zstd` or `zst`. By default, it will autodetect compression method by file extension.
-- `headers` - Parameter is optional. Allows headers to be passed in the S3 request. Pass in the format `headers(key=value)` e.g. `headers('x-amz-request-payer' = 'requester')`. See [here](/sql-reference/table-functions/s3#accessing-requester-pays-buckets) for example of use.
+- `headers` - Optional. Allows headers to be passed in the S3 request. Pass in the format `headers(key=value)` e.g. `headers('x-amz-request-payer' = 'requester')`. See [here](/sql-reference/table-functions/s3#accessing-requester-pays-buckets) for an example.
+- `extra_credentials` - Optional. `roleARN` can be passed via this parameter. See [here](/cloud/security/secure-s3#access-your-s3-bucket-with-the-clickhouseaccess-role) for an example.
 
 Arguments can also be passed using [named collections](operations/named-collections.md). In this case `url`, `access_key_id`, `secret_access_key`, `format`, `structure`, `compression_method` work in the same way, and some extra parameters are supported:
 

programs/keeper/keeper_main.cpp

@@ -219,5 +219,7 @@ int main(int argc_, char ** argv_)
     dumpCoverage();
 #endif
 
+    DB::OpenSSLInitializer::cleanup();
+
     return exit_code;
 }

programs/main.cpp

@@ -307,5 +307,7 @@ int main(int argc_, char ** argv_)
     dumpCoverage();
 #endif
 
+    DB::OpenSSLInitializer::cleanup();
+
     return exit_code;
 }

src/Access/SettingsConstraints.cpp

@@ -77,7 +77,7 @@ void SettingsConstraints::clear()
     settings_alias_cache.clear();
 }
 
-void SettingsConstraints::set(const String & full_name, const Field & min_value, const Field & max_value, SettingConstraintWritability writability)
+void SettingsConstraints::set(const String & full_name, const Field & min_value, const Field & max_value, const std::vector<Field> & disallowed_values, SettingConstraintWritability writability)
 {
     std::string resolved_name{resolveSettingName(full_name)};
 
@@ -90,23 +90,30 @@ void SettingsConstraints::set(const String & full_name, const Field & min_value,
         constraint.min_value = settingCastValueUtil(resolved_name, min_value);
     if (!max_value.isNull())
         constraint.max_value = settingCastValueUtil(resolved_name, max_value);
+    if (!disallowed_values.empty())
+    {
+        for (const auto & allowed_value : disallowed_values)
+            constraint.disallowed_values.emplace_back(settingCastValueUtil(resolved_name, allowed_value));
+    }
     constraint.writability = writability;
 }
 
-void SettingsConstraints::get(const Settings & current_settings, std::string_view short_name, Field & min_value, Field & max_value, SettingConstraintWritability & writability) const
+void SettingsConstraints::get(const Settings & current_settings, std::string_view short_name, Field & min_value, Field & max_value, std::vector<Field> & disallowed_values, SettingConstraintWritability & writability) const
 {
     // NOTE: for `Settings` short name is equal to full name
     auto checker = getChecker(current_settings, short_name);
     min_value = checker.constraint.min_value;
     max_value = checker.constraint.max_value;
+    disallowed_values = checker.constraint.disallowed_values;
     writability = checker.constraint.writability;
 }
 
-void SettingsConstraints::get(const MergeTreeSettings &, std::string_view short_name, Field & min_value, Field & max_value, SettingConstraintWritability & writability) const
+void SettingsConstraints::get(const MergeTreeSettings &, std::string_view short_name, Field & min_value, Field & max_value, std::vector<Field> & disallowed_values, SettingConstraintWritability & writability) const
 {
     auto checker = getMergeTreeChecker(short_name);
     min_value = checker.constraint.min_value;
     max_value = checker.constraint.max_value;
+    disallowed_values = checker.constraint.disallowed_values;
     writability = checker.constraint.writability;
 }
 
@@ -128,6 +135,8 @@ void SettingsConstraints::merge(const SettingsConstraints & other)
                 constraint.min_value = other_constraint.min_value;
             if (!other_constraint.max_value.isNull())
                 constraint.max_value = other_constraint.max_value;
+            if (!other_constraint.disallowed_values.empty())
+                constraint.disallowed_values = other_constraint.disallowed_values;
             if (other_constraint.writability == SettingConstraintWritability::CONST)
                 constraint.writability = SettingConstraintWritability::CONST; // NOTE: In this mode <readonly/> flag cannot be overridden to be false
         }
@@ -170,6 +179,10 @@ void SettingsConstraints::check(const Settings & current_settings, const Setting
             check(current_settings, value, source);
         }
 
+        /// Don't check disallowed_values here in the profile elements because they make constrains more restrictive
+        /// and don't allow to bypass constraints from config by creating a user with custom constraints. The check
+        /// for disallowed values are instead implemented in SettingsConstraints::Checker::check.
+
         SettingConstraintWritability new_value = SettingConstraintWritability::WRITABLE;
         SettingConstraintWritability old_value = SettingConstraintWritability::WRITABLE;
 
@@ -336,6 +349,20 @@ bool SettingsConstraints::Checker::check(SettingChange & change,
         }
     };
 
+    auto equals_or_cannot_compare = [=](const Field & left, const Field & right)
+    {
+        if (reaction == THROW_ON_VIOLATION)
+            return accurateEquals(left, right);
+        try
+        {
+            return accurateEquals(left, right);
+        }
+        catch (...)
+        {
+            return true;
+        }
+    };
+
 
     if (constraint.writability == SettingConstraintWritability::CONST)
     {
@@ -346,6 +373,7 @@ bool SettingsConstraints::Checker::check(SettingChange & change,
 
     const auto & min_value = constraint.min_value;
     const auto & max_value = constraint.max_value;
+    const auto & disallowed_values = constraint.disallowed_values;
 
     if (!min_value.isNull() && !max_value.isNull() && less_or_cannot_compare(max_value, min_value))
     {
@@ -379,6 +407,14 @@ bool SettingsConstraints::Checker::check(SettingChange & change,
         change.value = max_value;
     }
 
+    for (const auto & value : disallowed_values)
+    {
+        bool equals = equals_or_cannot_compare(value, new_value);
+        if (equals)
+            throw Exception(ErrorCodes::SETTING_CONSTRAINT_VIOLATION, "Setting {} shouldn't be {}",
+                setting_name, applyVisitor(FieldVisitorToString(), value));
+    }
+
     if (!getSettingSourceRestrictions(setting_name).isSourceAllowed(source))
     {
         if (reaction == THROW_ON_VIOLATION)
@@ -460,7 +496,7 @@ SettingsConstraints::Checker SettingsConstraints::getMergeTreeChecker(std::strin
 
 bool SettingsConstraints::Constraint::operator==(const Constraint & other) const
 {
-    return writability == other.writability && min_value == other.min_value && max_value == other.max_value;
+    return writability == other.writability && min_value == other.min_value && max_value == other.max_value && disallowed_values == other.disallowed_values;
 }
 
 bool operator ==(const SettingsConstraints & left, const SettingsConstraints & right)

src/Access/SettingsConstraints.h

@@ -71,9 +71,9 @@ class SettingsConstraints
     void clear();
     bool empty() const { return constraints.empty(); }
 
-    void set(const String & full_name, const Field & min_value, const Field & max_value, SettingConstraintWritability writability);
-    void get(const Settings & current_settings, std::string_view short_name, Field & min_value, Field & max_value, SettingConstraintWritability & writability) const;
-    void get(const MergeTreeSettings & current_settings, std::string_view short_name, Field & min_value, Field & max_value, SettingConstraintWritability & writability) const;
+    void set(const String & full_name, const Field & min_value, const Field & max_value, const std::vector<Field> & allowed_values, SettingConstraintWritability writability);
+    void get(const Settings & current_settings, std::string_view short_name, Field & min_value, Field & max_value, std::vector<Field> & allowed_values, SettingConstraintWritability & writability) const;
+    void get(const MergeTreeSettings & current_settings, std::string_view short_name, Field & min_value, Field & max_value, std::vector<Field> & allowed_values, SettingConstraintWritability & writability) const;
 
     void merge(const SettingsConstraints & other);
 
@@ -107,6 +107,7 @@ class SettingsConstraints
         SettingConstraintWritability writability = SettingConstraintWritability::WRITABLE;
         Field min_value{};
         Field max_value{};
+        std::vector<Field> disallowed_values{};
 
         bool operator ==(const Constraint & other) const;
         bool operator !=(const Constraint & other) const { return !(*this == other); }

src/Access/SettingsProfileElement.cpp

@@ -65,19 +65,22 @@ void SettingsProfileElement::init(const ASTSettingsProfileElement & ast, const A
         min_value = ast.min_value;
         max_value = ast.max_value;
         writability = ast.writability;
+        disallowed_values = ast.disallowed_values;
 
         if (value)
             value = Settings::castValueUtil(setting_name, *value);
         if (min_value)
             min_value = Settings::castValueUtil(setting_name, *min_value);
         if (max_value)
             max_value = Settings::castValueUtil(setting_name, *max_value);
+        for (auto & allowed_value : disallowed_values)
+            value = Settings::castValueUtil(setting_name, allowed_value);
     }
 }
 
 bool SettingsProfileElement::isConstraint() const
 {
-    return this->writability || this->min_value || this->max_value;
+    return this->writability || this->min_value || this->max_value || !this->disallowed_values.empty();
 }
 
 std::shared_ptr<ASTSettingsProfileElement> SettingsProfileElement::toAST() const
@@ -92,6 +95,7 @@ std::shared_ptr<ASTSettingsProfileElement> SettingsProfileElement::toAST() const
     ast->value = value;
     ast->min_value = min_value;
     ast->max_value = max_value;
+    ast->disallowed_values = disallowed_values;
     ast->writability = writability;
 
     return ast;
@@ -113,6 +117,7 @@ std::shared_ptr<ASTSettingsProfileElement> SettingsProfileElement::toASTWithName
     ast->value = value;
     ast->min_value = min_value;
     ast->max_value = max_value;
+    ast->disallowed_values = disallowed_values;
     ast->writability = writability;
 
     return ast;
@@ -277,6 +282,7 @@ SettingsConstraints SettingsProfileElements::toSettingsConstraints(const AccessC
                 elem.setting_name,
                 elem.min_value ? *elem.min_value : Field{},
                 elem.max_value ? *elem.max_value : Field{},
+                elem.disallowed_values,
                 elem.writability ? *elem.writability : SettingConstraintWritability::WRITABLE);
     return res;
 }
@@ -360,6 +366,8 @@ void SettingsProfileElements::normalize()
                     first_element.min_value = element.min_value;
                 if (element.max_value)
                     first_element.max_value = element.max_value;
+                if (!element.disallowed_values.empty())
+                    first_element.disallowed_values = element.disallowed_values;
                 if (element.writability)
                     first_element.writability = element.writability;
                 element.setting_name.clear();
@@ -484,6 +492,7 @@ void SettingsProfileElements::applyChanges(const AlterSettingsProfileElements &
         new_element.value = modify.value;
         new_element.min_value = modify.min_value;
         new_element.max_value = modify.max_value;
+        new_element.disallowed_values = modify.disallowed_values;
         new_element.writability = modify.writability;
         push_back(new_element); /// normalizeProfileElements() will merge this new element with the previous elements.
     };

src/Access/SettingsProfileElement.h

@@ -28,9 +28,10 @@ struct SettingsProfileElement
     std::optional<Field> value;
     std::optional<Field> min_value;
     std::optional<Field> max_value;
+    std::vector<Field> disallowed_values;
     std::optional<SettingConstraintWritability> writability;
 
-    auto toTuple() const { return std::tie(parent_profile, setting_name, value, min_value, max_value, writability); }
+    auto toTuple() const { return std::tie(parent_profile, setting_name, value, min_value, max_value, disallowed_values, writability); }
     friend bool operator==(const SettingsProfileElement & lhs, const SettingsProfileElement & rhs) { return lhs.toTuple() == rhs.toTuple(); }
     friend bool operator!=(const SettingsProfileElement & lhs, const SettingsProfileElement & rhs) { return !(lhs == rhs); }
     friend bool operator <(const SettingsProfileElement & lhs, const SettingsProfileElement & rhs) { return lhs.toTuple() < rhs.toTuple(); }
@@ -46,7 +47,7 @@ struct SettingsProfileElement
     std::shared_ptr<ASTSettingsProfileElement> toAST() const;
     std::shared_ptr<ASTSettingsProfileElement> toASTWithNames(const AccessControl & access_control) const;
 
-    bool empty() const { return !parent_profile && (setting_name.empty() || (!value && !min_value && !max_value && !writability)); }
+    bool empty() const { return !parent_profile && (setting_name.empty() || (!value && !min_value && !max_value && disallowed_values.empty() && !writability)); }
 
     bool isConstraint() const;
 

src/Access/UsersConfigAccessStorage.cpp

@@ -699,6 +699,19 @@ namespace
                     profile_element.min_value = settingStringToValueUtil(setting_name, config.getString(path_to_name + "." + constraint_type));
                 else if (constraint_type == "max")
                     profile_element.max_value = settingStringToValueUtil(setting_name, config.getString(path_to_name + "." + constraint_type));
+                /// When the xml config is parsed, the first constraint_type is parsed as `disallowed` and the subsequent ones are parsed as
+                /// disallowed[1], disallowed[2] and so on. So, both `disallowed` and `disallowed[` should be considered as valid constraint types.
+                /// Example:
+                ///   <constraints>
+                ///     <max_execution_time>
+                ///         <max>50</max>
+                ///         <disallowed>3</disallowed>
+                ///         <disallowed>4</disallowed>
+                ///         <disallowed>5</disallowed>
+                ///     </max_execution_time>
+                /// </constraints>
+                else if (constraint_type == "disallowed" || constraint_type.starts_with("disallowed["))
+                    profile_element.disallowed_values.push_back(settingStringToValueUtil(setting_name, config.getString(path_to_name + "." + constraint_type)));
                 else if (constraint_type == "readonly" || constraint_type == "const")
                 {
                     writability_count++;