merge main into branch

Author: Blargian

.github/workflows/badges-notify.yml

@@ -0,0 +1,57 @@
+name: Weekly Badge Monitor
+
+on:
+  schedule:
+    # Run every Monday at 10:00 AM Amsterdam time (UTC+1/+2)
+    # Using 8:00 UTC to account for Amsterdam time (adjusts for DST)
+    - cron: '0 8 * * 1'
+  workflow_dispatch: # Allow manual triggering
+
+jobs:
+  monitor-badges:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Make script executable
+        run: chmod +x scripts/badger.sh
+
+      - name: Run badge finder script
+        id: find_badges
+        run: |
+          # Run the script and capture output
+          SCRIPT_OUTPUT=$(scripts/badger.sh)
+          
+          # Escape the output for JSON and store in GitHub output
+          # Replace newlines with \n and escape quotes
+          ESCAPED_OUTPUT=$(echo "$SCRIPT_OUTPUT" | sed 's/\\/\\\\/g' | sed 's/"/\\"/g' | sed ':a;N;$!ba;s/\n/\\n/g')
+          
+          echo "badges_output<<EOF" >> $GITHUB_OUTPUT
+          echo "$ESCAPED_OUTPUT" >> $GITHUB_OUTPUT
+          echo "EOF" >> $GITHUB_OUTPUT
+
+      - name: Send webhook
+        env:
+          WEBHOOK_URL: ${{ secrets.DOCS_BADGER }}
+          BADGES_OUTPUT: ${{ steps.find_badges.outputs.badges_output }}
+        run: |
+          # Create JSON payload
+          JSON_PAYLOAD=$(cat <<EOF
+          {
+            "badges_list": "$BADGES_OUTPUT"
+          }
+          EOF
+          )
+          
+          # Send webhook
+          curl -X POST \
+            -H "Content-Type: application/json" \
+            -d "$JSON_PAYLOAD" \
+            "$WEBHOOK_URL"
+
+      - name: Log completion
+        run: |
+          echo "Badge monitoring completed successfully"
+          echo "Webhook sent to configured endpoint"

.gitignore

@@ -55,6 +55,15 @@ static/knowledgebase_toc.json
 .frontmatter-validation-failed
 logs/
 
+# Landing page files with autogenerated table of contents
+docs/cloud/reference/release-notes-index.md
+docs/cloud/manage/api/api-reference-index.md
+docs/getting-started/index.md
+docs/data-modeling/projections/index.md
+docs/cloud/manage/jan2025_faq/index.md
+docs/chdb/guides/index.md
+docs/use-cases/AI_ML/index.md
+
 .vscode
 .aspell.en.prepl
 *.md.bak
@@ -68,3 +77,5 @@ docs/cloud/reference/release-notes-index.md
 docs/whats-new/changelog/index.md
 docs/cloud/manage/api/api-reference-index.md
 docs/getting-started/index.md
+docs/data-modeling/projections/index.md
+docs/integrations/data-ingestion/clickpipes/kafka/index.md

README.md

@@ -1,8 +1,8 @@
 <div align=center>
 
-![Website](https://img.shields.io/website?up_message=AVAILABLE&down_message=DOWN&url=https%3A%2F%2Fclickhouse.com%2Fdocs&style=for-the-badge)
+[![Website](https://img.shields.io/website?up_message=AVAILABLE&down_message=DOWN&url=https%3A%2F%2Fclickhouse.com%2Fdocs&style=for-the-badge)](https://clickhouse.com)
 [![CC BY-NC-SA 4.0 License](https://img.shields.io/badge/license-CC-blueviolet?style=for-the-badge)](http://creativecommons.org/licenses/by-nc-sa/4.0/)
-![Checks](https://img.shields.io/github/actions/workflow/status/clickhouse/clickhouse-docs/debug.yml?style=for-the-badge&label=Checks)
+[![Checks](https://img.shields.io/github/actions/workflow/status/clickhouse/clickhouse-docs/debug.yml?style=for-the-badge&label=Checks)](https://github.com/ClickHouse/clickhouse-docs/actions)
 
 <picture align=center>
     <source media="(prefers-color-scheme: dark)" srcset="https://github.com/ClickHouse/clickhouse-docs/assets/9611008/4ef9c104-2d3f-4646-b186-507358d2fe28">
@@ -18,19 +18,19 @@
 
 ClickHouse is blazing fast, but understanding ClickHouse and using it effectively is a journey. The documentation is your source for gaining the knowledge you need to be successful with your ClickHouse projects and applications. [Head over to clickhouse.com/docs to learn more →](https://clickhouse.com/)
 
-## Table of contents {#table-of-contents}
+## Table of contents
 
 - [About this repo](#about-this-repo)
 - [Run locally](#run-locally)
 - [Contributing](#contributing)
 - [Issues](#issues)
 - [License](#license)
 
-## About this repo {#about-this-repo}
+## About this repo
 
 This repository manages the documentation for [ClickHouse](https://clickhouse.com/docs). The content is built with [Docusaurus](https://docusaurus.io/) and hosted on [Vercel](https://vercel.com). Documentation content is written in Markdown and is held in the `/docs` directory.
 
-## Run locally {#run-locally}
+## Run locally
 
 You can run a copy of this website locally within a few steps. Some folks find this useful when contributing so they can see precisely what their changes will look like on the production site.
 
@@ -116,23 +116,19 @@ To check spelling and markdown is correct locally run:
 yarn check-style
 ```
 
-### Notes {#notes}
+### Notes
 
 Here are some things to keep in mind when building a local copy of the ClickHouse docs site.
 
-#### Build-time {#build-time}
-
-Due to the complex structure of this repo, the docs site can take some time to build locally. As a benchmark, it takes ~3 minutes to build on an M1 Macbook with 8GB RAM.
-
-#### Redirects {#redirects}
+#### Redirects
 
 Due to how the local server is built, redirects will not work. For example, visiting `clickhouse.com/docs` on the production site will lead you to `clickhouse.com/docs/intro`. However, on a local copy of the site, you will see a 404 page if you try to visit `localhost:8000/docs`.
 
-## Contributing {#contributing}
+## Contributing
 
 Want to help out? Contributions are always welcome! If you want to help out but aren't sure where to start, check out the [issues board](https://github.com/clickhouse/clickhouse-docs/issues).
 
-### Pull requests {#pull-requests}
+### Pull requests
 
 Please assign any pull request (PR) against an issue; this helps the docs team track who is working on what and what each PR is meant to address. If there isn't an issue for the specific _thing_ you want to work on, quickly create one and comment so that it can be assigned to you. One of the repository maintainers will add you as an assignee.
 
@@ -171,39 +167,39 @@ Have you noticed a typo or found some wonky formatting? For small contributions
 
 1. Each page in Clickhouse.com/docs has an **Edit this page** link at the top:
 
-    ![The ClickHouse Docs website with the edit button highlighted.](./static/images/contribute/readme-edit-this-page.png)
+    ![The ClickHouse Docs website with the edit button highlighted.](./static/images/contributing/readme-edit-this-page.png)
 
     Click this button to edit this page in GitHub.
 
 1. Once you're in GitHub, click the pencil icon to edit this page:
 
-    ![README Pencil Icon](./static/images/contribute/readme-pencil-icon.png)
+    ![README Pencil Icon](./static/images/contributing/readme-pencil-icon.png)
 
 1. GitHub will _fork_ the repository for you. This creates a copy of the `clickhouse-docs` repository on your personal GitHub account.
 1. Make your changes in the textbox. Once you're done, click **Commit changes**:
 
-    ![README Commit Changes](./static/images/contribute/readme-commit-changes.png)
+    ![README Commit Changes](./static/images/contributing/readme-commit-changes.png)
 
 1. In the **Propose changes** popup, enter a descriptive title to explain the changes you just made. Keep this title to 10 words or less. If your changes are fairly complex and need further explanation, enter your comments into the **Extended description** field.
 1. Make sure **Create a new branch** is selected, and click **Propose changes**:
 
-    ![README Propose Changes](./static/images/contribute/readme-propose-changes.png)
+    ![README Propose Changes](./static/images/contributing/readme-propose-changes.png)
 
 1. A new page should open with a new pull request. Double-check that the title and description are accurate.
 1. If you've spoken to someone on the docs team about your changes, tag them into the **Reviewers** section:
 
-    ![README Create Pull Request](./static/images/contribute/readme-create-pull-request.png)
+    ![README Create Pull Request](./static/images/contributing/readme-create-pull-request.png)
 
     If you haven't mentioned your changes to anyone yet, leave the **Reviewers** section blank.
 
 1. Click **Create pull request**.
 
 At this point, your pull request will be handed over to the docs team, who will review it and suggest or make changes where necessary.
 
-## Issues {#issues}
+## Issues
 
 Found a problem with the Clickhouse docs site? [Please raise an issue](https://github.com/clickhouse/clickhouse-docs/issues/new). Be as specific and descriptive as possible; screenshots help!
 
-## License {#license}
+## License
 
 This work is licensed under a [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License](http://creativecommons.org/licenses/by-nc-sa/4.0/).

contribute/autogenerated-documentation-from-source.md

@@ -1,15 +1,18 @@
 # Generating documentation from source code
 
 A [script in our docs repo](https://github.com/ClickHouse/clickhouse-docs/blob/main/scripts/settings/autogenerate-settings.sh) 
-extracts setting names, descriptions, default values, etc. from ClickHouse's source code.
+is used to run SQL queries which generate markdown documentation from the system tables in ClickHouse and insert it 
+into the appropriate pages before build time of the docs.
 
 ## Session settings
 
 Documentation for session settings is autogenerated from source file [`src/Core/Settings.cpp`](https://github.com/ClickHouse/ClickHouse/blob/master/src/Core/Settings.cpp).
+[SQL](../scripts/settings/session-settings.sql)
 
 ## Format settings
 
 Documentation for session settings is autogenerated from source file [`src/Core/FormatFactorySettings.h`](https://github.com/ClickHouse/ClickHouse/blob/master/src/Core/FormatFactorySettings.h).
+[SQL](../scripts/settings/format-settings.sql)
 
 ## Server settings
 
@@ -43,14 +46,33 @@ If it is, then please edit the setting description in the source code documentat
 in [`ServerSettings.cpp`](https://github.com/ClickHouse/ClickHouse/blob/master/src/Core/ServerSettings.cpp) 
 or else edit `_server_settings_outside_source.md`.**
 
+[SQL](../scripts/settings/global-server-settings.sql)
+
 ## MergeTree settings
 
 Documentation for MergeTree settings is autogenerated from [MergeTreeSettings.cpp](https://github.com/ClickHouse/ClickHouse/blob/master/src/Storages/MergeTree/MergeTreeSettings.cpp)
 
+[SQL](../scripts/settings/mergetree-settings.sql)
+
 ## System tables
 
-TO DO. ETA April.
+TO DO
 
 ## Functions
 
-TO DO. ETA April/May.
+Documentation for functions is autogenerated from the documentation in system tables (`system.functions`) which is 
+registered along with the function registration in the C++ code.
+
+To add a new page:
+- add SQL to generate the markdown from system tables
+- update `autogenerate-settings.sh` in the section which specifies which markdown file contents to
+  copy to which file in between the `<!--AUTOGENERATED_START-->` and `<!--AUTOGENERATED_END-->`
+
+The following pages are autogenerated from the source code for functions:
+- [Arithmetic](/sql-reference/functions/arithmetic-functions) ([SQL](../scripts/settings/arithmetic-functions.sql))
+- [Arrays](/sql-reference/functions/array-functions) ([SQL](../scripts/settings/array-functions.sql))
+- [Bit](/sql-reference/functions/bit-functions) ([SQL](../scripts/settings/bit-functions.sql))
+- [Bitmap](/sql-reference/functions/bitmap-functions) ([SQL](../scripts/settings/bitmap-functions.sql))
+- [Comparison](/sql-reference/functions/comparison-functions) ([SQL](../scripts/settings/comparison-functions.sql))
+- [Conditional](/sql-reference/functions/conditional-functions) ([SQL](../scripts/settings/conditional-functions.sql))
+

docs/about-us/beta-and-experimental-features.md

@@ -22,7 +22,7 @@ The sections below explicitly describe the properties of **Beta** and **Experime
 - Possibly enabled in ClickHouse Cloud
 - The ClickHouse team supports beta features
 
-The following features are considered Beta in ClickHouse Cloud and are available for use in ClickHouse Cloud Services, even though they may be currently under a ClickHouse SETTING named ```allow_experimental_*```:
+You can find below the features considered Beta in ClickHouse Cloud and are available for use in your ClickHouse Cloud Services.
 
 Note: please be sure to be using a current version of the ClickHouse [compatibility](/operations/settings/settings#compatibility) setting to be using a recently introduced feature.
 

docs/about-us/cloud.md

@@ -22,7 +22,28 @@ Some of the benefits of using ClickHouse Cloud are described below:
 - **Total cost of ownership**: Best price / performance ratio and low administrative overhead.
 - **Broad ecosystem**: Bring your favorite data connectors, visualization tools, SQL and language clients with you.
 
-### What version of ClickHouse does ClickHouse Cloud use? {#what-version-of-clickhouse-does-clickhouse-cloud-use}
+<!--
+## OSS vs ClickHouse Cloud comparison {#oss-vs-clickhouse-cloud}
+
+| Feature                        | Benefits                                                                                                                                                                                                                                                                                                   | OSS ClickHouse  | ClickHouse Cloud  |
+|--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|-------------------|
+| **Deployment modes**           | ClickHouse provides flexibility to self-manage with open-source or deploy in the cloud. Use ClickHouse Local for local files without a server or chDB to embed ClickHouse directly into your application.                                                                                                  | ✅               | ✅                 |
+| **Storage**                    | As an open-source and cloud-hosted product, ClickHouse can be deployed in both shared-disk and shared-nothing architectures.                                                                                                                                                                               | ✅               | ✅                 |
+| **Monitoring and alerting**    | Monitoring and alerting about the status of your services is critical to ensuring optimal performance and a proactive approach to detect and triage potential issues.                                                                                                                                      | ✅               | ✅                 |
+| **ClickPipes**                 | ClickPipes is ClickHouse's managed ingestion pipeline that allows you to seamlessly connect your external data sources like databases, APIs, and streaming services into ClickHouse Cloud, eliminating the need for managing pipelines, custom jobs, or ETL processes. It supports workloads of all sizes. | ❌               | ✅                 |
+| **Pre-built integrations**     | ClickHouse provides pre-built integrations that connect ClickHouse to popular tools and services such as data lakes, SQL and language clients, visualization libraries, and more.                                                                                                                          | ❌               | ✅                 |
+| **SQL console**                | The SQL console offers a fast, intuitive way to connect, explore, and query ClickHouse databases, featuring a slick caption, query interface, data import tools, visualizations, collaboration features, and GenAI-powered SQL assistance.                                                                 | ❌               | ✅                 |
+| **Compliance**                 | ClickHouse Cloud compliance includes CCPA, EU-US DPF, GDPR, HIPAA, ISO 27001, ISO 27001 SoA, PCI DSS, SOC2. ClickHouse Cloud's security, availability, processing integrity, and confidentiality processes are all independently audited. Details: trust.clickhouse.com.                                   | ❌               | ✅                 |
+| **Enterprise-grade security**  | Support for advanced security features such as SSO, multi-factor authentication, role-based access control (RBAC), private and secure connections with support for Private Link and Private Service Connect, IP filtering, customer-managed encryption keys (CMEK), and more.                              | ❌               | ✅                 |
+| **Scaling and optimization**   | Seamlessly scales up or down based on workload, supporting both horizontal and vertical scaling. With automated backups, replication, and high availability, ClickHouse, it provides users with optimal resource allocation.                                                                               | ❌               | ✅                 |
+| **Support services**           | Our best-in-class support services and open-source community resources provide coverage for whichever deployment model you choose.                                                                                                                                                                         | ❌               | ✅                 |
+| **Database upgrades**          | Regular database upgrades are essential to establish a strong security posture and access the latest features and performance improvements.                                                                                                                                                                | ❌               | ✅                 |
+| **Backups**                    | Backups and restore functionality ensures data durability and supports graceful recovery in the event of outages or other disruptions.                                                                                                                                                                     | ❌               | ✅                 |
+| **Compute-compute separation** | Users can scale compute resources independently of storage, so teams and workloads can share the same storage and maintain dedicated compute resources. This ensures that the performance of one workload doesn't interfere with another, enhancing flexibility, performance, and cost-efficiency.         | ❌               | ✅                 |
+| **Managed services**           | With a cloud-managed service, teams can focus on business outcomes and accelerate time-to-market without having to worry about the operational overhead of sizing, setup, and maintenance of ClickHouse.                                                                                                   | ❌               | ✅                 |
+-->
+
+## What version of ClickHouse does ClickHouse Cloud use? {#what-version-of-clickhouse-does-clickhouse-cloud-use}
 
 Clickhouse Cloud continuously upgrades your service to a newer version. After publishing a core database version in the open source, we do additional validation in our cloud staging environment, which typically takes 6-8 weeks before rolling out to production. The rollout is phased out by cloud service provider, service type, and region.
 

docs/best-practices/using_data_skipping_indices.md

@@ -140,7 +140,7 @@ LIMIT 1
 A simple analysis shows that `ViewCount` is correlated with the `CreationDate` (a primary key) as one might expect - the longer a post exists, the more time it has to be viewed.
 
 ```sql
-SELECT toDate(CreationDate) as day, avg(ViewCount) as view_count FROM stackoverflow.posts WHERE day > '2009-01-01'  GROUP BY day
+SELECT toDate(CreationDate) AS day, avg(ViewCount) AS view_count FROM stackoverflow.posts WHERE day > '2009-01-01'  GROUP BY day
 ```
 
 This therefore makes a logical choice for a data skipping index. Given the numeric type, a min_max index makes sense. We add an index using the following `ALTER TABLE` commands - first adding it, then "materializing it".