Merge branch 'main' into ch-audit-splunk-integration-guide

Author: lio-p

clickhouseapi.js

@@ -52,7 +52,7 @@ function generateDocusaurusMarkdown(spec, groupedEndpoints, prefix) {
 
       markdownContent += `| Method | Path |\n`
       markdownContent += `| :----- | :--- |\n`
-      markdownContent += `| ${method.toUpperCase()} | ${path} |\n\n`
+      markdownContent += `| ${method.toUpperCase()} | \`${path}\` |\n\n`
 
       markdownContent += `### Request\n\n`;
 

copyClickhouseRepoDocs.sh

@@ -1,7 +1,8 @@
-#! ./bin/bash
+#! /bin/bash
 
 SCRIPT_NAME=$(basename "$0")
 
+rm -rf ClickHouse
 echo "[$SCRIPT_NAME] Start tasks for copying docs from ClickHouse repo"
 
 # Clone ClickHouse repo

docs/en/cloud/bestpractices/avoidoptimizefinal.md

@@ -2,10 +2,32 @@
 slug: /en/cloud/bestpractices/avoid-optimize-final
 sidebar_label: Avoid Optimize Final
 title: Avoid Optimize Final
+keywords: ['OPTIMIZE TABLE', 'FINAL', 'unscheduled merge']
 ---
 
-Using the [`OPTIMIZE TABLE ... FINAL`](/docs/en/sql-reference/statements/optimize/) query will initiate an unscheduled merge of data parts for the specific table into one data part. During this process, ClickHouse reads all the data parts, uncompresses, merges, compresses them into a single part, and then rewrites back into object store, causing huge CPU and IO consumption.
+Using the [`OPTIMIZE TABLE ... FINAL`](/docs/en/sql-reference/statements/optimize/) query initiates an unscheduled merge of data parts for a specific table into one single data part. 
+During this process, ClickHouse performs the following steps:
+
+- Data parts are read.
+- The parts get uncompressed.
+- The parts get merged.
+- They are compressed into a single part.
+- The part is then written back into the object store.
+
+The operations described above are resource intensive, consuming significant CPU and disk I/O.
+It is important to note that using this optimization will force a rewrite of a part, 
+even if merging to a single part has already occurred.
+
+Additionally, use of the `OPTIMIZE TABLE ... FINAL` query may disregard 
+setting [`max_bytes_to_merge_at_max_space_in_pool`](https://clickhouse.com/docs/en/operations/settings/merge-tree-settings#max-bytes-to-merge-at-max-space-in-pool) which controls the maximum size of parts
+that ClickHouse will typically merge by itself in the background.
+
+The [`max_bytes_to_merge_at_max_space_in_pool`](https://clickhouse.com/docs/en/operations/settings/merge-tree-settings#max-bytes-to-merge-at-max-space-in-pool) setting is by default set to 150 GB. 
+When running `OPTIMIZE TABLE ... FINAL`, 
+the steps outlined above will be performed resulting in a single part after merge. 
+This remaining single part could exceed the 150 GB specified by the default of this setting. 
+This is another important consideration and reason why you should avoid use of this statement, 
+since merging a large number of 150 GB parts into a single part could require a significant amount of time and/or memory.
 
-Note that this optimization rewrites the one part even if they are already merged into a single part. Also, it is important to note the scope of a "single part" - this indicates that the value of the setting [`max_bytes_to_merge_at_max_space_in_pool`](https://clickhouse.com/docs/en/operations/settings/merge-tree-settings#max-bytes-to-merge-at-max-space-in-pool) will be ignored. For example, [`max_bytes_to_merge_at_max_space_in_pool`](https://clickhouse.com/docs/en/operations/settings/merge-tree-settings#max-bytes-to-merge-at-max-space-in-pool) is by default set to 150 GB. When running OPTIMIZE TABLE ... FINAL, the remaining single part could exceed even this size. This is another important consideration and reason not to generally use this command, since merging a large number of 150 GB parts into a single part could require a significant amount of time and/or memory.
 
 

docs/en/cloud/reference/changelog.md

@@ -957,7 +957,7 @@ Adds support for a subset of features in ClickHouse 23.1, for example:
 - New functions, including `age()`, `quantileInterpolatedWeighted()`, `quantilesInterpolatedWeighted()`
 - Ability to use structure from insertion table in `generateRandom` without arguments
 - Improved database creation and rename logic that allows the reuse of previous names
-- See the 23.1 release [webinar slides](https://presentations.clickhouse.com/release_23.1/#cover) and [23.1 release changelog](/docs/en/whats-new/changelog/index.md/#clickhouse-release-231) for more details
+- See the 23.1 release [webinar slides](https://presentations.clickhouse.com/release_23.1/#cover) and [23.1 release changelog](/docs/en/whats-new/changelog/index.md#clickhouse-release-231) for more details
 
 ### Integrations changes
 - [Kafka-Connect](/docs/en/integrations/data-ingestion/kafka/index.md): Added support for Amazon MSK

docs/en/cloud/reference/warehouses.md

@@ -157,6 +157,9 @@ settings distributed_ddl_task_timeout=0
 6. **The original service should be new enough, or migrated**
 Unfortunately, not all existing services can share their storage with other services. During the last year, we released a few features that the service needs to support (like the Shared Merge Tree engine), so old services will mostly not be able to share their data with other services. This does not depend on ClickHouse version. The good news is that we can migrate the old service to the new engine, so it can support creating additional services. If you have a service for which you cannot enable compute-compute separation, please contact support to assist with the migration.
 
+7. **Single-node secondary services can be unavailable for up to 1 hour during upgrades**
+When creating a database service, you can select the number of replicas. When creating a secondary service, you can select to create a single-node service, which means that there will be no high availability for this particular service. Currently, when performing an upgrade of such a service, a usual rolling upgrade can not be performed, which means that the single-node service will be unavailable during the upgrade. Though usually an upgrade takes only a few minutes, in some cases, if there are long-running queries, it can take up to one hour. The single-node service will be unavailable during this time. Consider creating at least two nodes service if this is not acceptable - in this case, there will be no downtime at all. We are working on removing this limitation.
+
 ## Pricing
 
 Extra services created during the private preview are billed as usual. Compute prices are the same for all services in a warehouse (primary and secondary). Storage is billed only once - it is included in the first (original) service.

docs/en/data-compression/compression-modes.md

@@ -41,7 +41,7 @@ From [facebook benchmarks](https://facebook.github.io/zstd/#benchmarks):
 | mode            | byte    | Compression mode                                 |
 | compressed_data | binary  | Block of compressed data                         |
 
-![compression block diagram](../native-protocol/images/ch_compression_block.drawio.svg)
+![compression block diagram](./images/ch_compression_block.png)
 
 Header is (raw_size + data_size + mode), raw size consists of len(header + compressed_data).
 

docs/en/data-compression/images/ch_compression_block.png

@@ -448,7 +448,7 @@ GROUP BY
 
 Here, we create a Null table, `pypi_v2,` to receive the rows that will be used to build our materialized view. Note how we limit the schema to only the columns we need. Our materialized view performs an aggregation over rows inserted into this table (one block at a time), sending the results to our target table, `pypi_downloads_per_day`.
 
-::note
+:::note
 We have used `pypi_downloads_per_day` as our target table here. For additional resiliency, users could create a duplicate table, `pypi_downloads_per_day_v2`, and use this as the target table of the view, as shown in previous examples. On completion of the insert, partitions in `pypi_downloads_per_day_v2` could, in turn, be moved to `pypi_downloads_per_day.` This would allow recovery in the case our insert fails due to memory issues or server interruptions i.e. we just truncate `pypi_downloads_per_day_v2`, tune settings, and retry. 
 :::
 

docs/en/data-modeling/backfilling.md

@@ -63,12 +63,10 @@ Each node has corresponding children and the overall tree represents the overall
 
 ## Analyzer
 
-<BetaBadge />
-
-ClickHouse currently has two architectures for the Analyzer. You can use the old architecture by setting: `allow_experimental_analyzer=0`. If you want to use the new architecture, you should set `allow_experimental_analyzer=1`. We are going to describe only the new architecture here, given the old one is going to be deprecated once the new analyzer is generally available.
+ClickHouse currently has two architectures for the Analyzer. You can use the old architecture by setting: `enable_analyzer=0`. The new architecture is enabled by default. We are going to describe only the new architecture here, given the old one is going to be deprecated once the new analyzer is generally available.
 
 :::note
-The new analyzer is in Beta. The new architecture should provide us with a better framework to improve ClickHouse's performance. However, given it is a fundamental component of the query processing steps, it also might have a negative impact on some queries. After moving to the new analyzer, you may see performance degradation, queries failing, or queries giving you an unexpected result. You can revert back to the old analyzer by changing the `allow_experimental_analyzer` setting at the query or user level. Please report any issues in GitHub.
+The new architecture should provide us with a better framework to improve ClickHouse's performance. However, given it is a fundamental component of the query processing steps, it also might have a negative impact on some queries and there are [known incompatibilities](/docs/en/operations/analyzer#known-incompatibilities). You can revert back to the old analyzer by changing the `enable_analyzer` setting at the query or user level. 
 :::
 
 The analyzer is an important step of the query execution. It takes an AST and transforms it into a query tree. The main benefit of a query tree over an AST is that a lot of the components will be resolved, like the storage for instance. We also know from which table to read, aliases are also resolved, and the tree knows the different data types used. With all these benefits, the analyzer can apply optimizations. The way these optimizations work is via “passes”. Every pass is going to look for different optimizations. You can see all the passes [here](https://github.com/ClickHouse/ClickHouse/blob/76578ebf92af3be917cd2e0e17fea2965716d958/src/Analyzer/QueryTreePassManager.cpp#L249), let’s see it in practice with our previous query:

docs/en/guides/developer/understanding-query-execution-with-the-analyzer.md

@@ -89,7 +89,7 @@ With asynchronous inserts, data is inserted into a buffer first and then written
 <br />
 
 <img src={require('./images/postgres-inserts.png').default}    
-     class="image"
+     className="image"
      alt="NEEDS ALT"
      style={{width: '600px'}} 
 />