diff --git a/contribute/CODE_OF_CONDUCT.md b/contribute/CODE_OF_CONDUCT.md index c272c84b9a9..624178b76d4 100644 --- a/contribute/CODE_OF_CONDUCT.md +++ b/contribute/CODE_OF_CONDUCT.md @@ -1,4 +1,4 @@ -# ClickHouse Community Code of Conduct +# ClickHouse community code of conduct This code of conduct governs ClickHouse Community discussions, channels, and events. ## Introduction {#introduction} @@ -7,13 +7,13 @@ This code of conduct governs ClickHouse Community discussions, channels, and eve * Our goal is to maintain a safe and friendly community for everyone, regardless of experience, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, religion, nationality, or other defining characteristic. * This code and related procedures apply to unacceptable behavior occurring in all community venues, including behavior outside the scope of community activities — online and in-person — as well as in all one-on-one communications, and anywhere such behavior has the potential to adversely affect the safety and well-being of community members. -## Expected Behavior {#expected-behavior} +## Expected behavior {#expected-behavior} * Be welcoming. * Be kind and patient. * Look out for each other. * Be careful and aware of your wording and tone. -## Unacceptable Behavior {#unacceptable-behavior} +## Unacceptable behavior {#unacceptable-behavior} * Conduct or speech which might be considered sexist, racist, homophobic, transphobic, ableist or otherwise discriminatory or offensive in nature. * Do not use unwelcome, suggestive, derogatory or inappropriate nicknames or terms. * Do not show disrespect towards others. (Jokes, innuendo, dismissive attitudes.) diff --git a/contribute/contrib-writing-guide.md b/contribute/contrib-writing-guide.md index d329bd482db..95b8294c7d3 100644 --- a/contribute/contrib-writing-guide.md +++ b/contribute/contrib-writing-guide.md @@ -31,7 +31,7 @@ Our build process is a little different because part of our docs are in the [Cli The documentation is built with Docusaurus, which requires Node.js. We recommend version 18. Install [Node.js](https://nodejs.org/en/download/). -### Installing npm and yarn {#installing-npm-and-yarn} +### Installing NPM and yarn {#installing-npm-and-yarn} We use [yarn classic](https://classic.yarnpkg.com/lang/en/) to build the ClickHouse docs. @@ -146,12 +146,12 @@ We have seen more and more 404s as the size of the docs grow. It seems that these are related to the use of `../` in the path to the markdown files. Please write links to other doc pages as: -#### link to another doc {#link-to-another-doc} +#### Link to another doc {#link-to-another-doc} ```md title="foo" [async_insert](/operations/settings/settings.md) ``` -#### link to an anchor within another doc {#link-to-an-anchor-within-another-doc} +#### Link to an anchor within another doc {#link-to-an-anchor-within-another-doc} ```md [async_insert](/operations/settings/settings.md/#async-insert) @@ -252,7 +252,7 @@ Some docs contain information for both Cloud and Self-managed ClickHouse, and so Note: in the examples below you may notice that the H1 and H2 headers are outside of the tabs, the right-side "What's on this page" nav is built from the H1,2,3 headers in the main body of the markdown file, headings within the tabs are not currently included. -### Differences between Cloud and Self-managed {#differences-between-cloud-and-self-managed} +### Differences between Cloud and self-managed {#differences-between-cloud-and-self-managed} Backup and restore are very different across Cloud and Self-managed as the target for the backup does not need to be specified, etc. The backup docs are an example where there is no content reused, so everything goes into one tab or the other. This is how the tabs are constructed: diff --git a/contribute/style-guide.md b/contribute/style-guide.md index d9e5b69191a..2215668854b 100644 --- a/contribute/style-guide.md +++ b/contribute/style-guide.md @@ -22,7 +22,7 @@ doc_type: 'reference' (or 'guide' or 'changelog') ### Associated markdown rule or CI check -#### front-matter validation +#### front-matter validation There is a custom Docusaurus plugin which runs on build that makes the following checks on front-matter: @@ -249,7 +249,7 @@ imported by that page. Find the snippet location from the import statement at th Docusaurus also has a built-in broken anchor checker. Unfortunately it sometimes can give false positives. -### linking to spans +### Linking to spans Sometimes you want to link to something other than a header. It is logical to use a span for this purpose. For instance if you want to link to an image. @@ -335,7 +335,7 @@ which may need versioned documentation, we use the following custom The `ClientVersionDropdown` component supports two APIs: -#### API 1: Inline Content (Recommended) +#### API 1: inline content (recommended) This approach keeps all content in the main file, which allows Docusaurus to properly generate the table of contents (TOC) for all versions. @@ -375,7 +375,7 @@ Your v0.7.x content here... - The TOC will show only headers from the currently selected version - The component will display the first version as 'selected' by default -#### API 2: External Snippets (Legacy) +#### API 2: external snippets (legacy) This approach uses separate snippet files for each version. Note that this method has limitations with TOC generation. @@ -458,7 +458,7 @@ in your component configuration. For example: - URL: `docs/api?v=v08` - Matches: `version: 'v0.8+'` in your dropdown configuration -#### Conventions That Work +#### Conventions that work - **Simple Version Strings**: Parameters like `?v=v08`, `?v=v07` work by - matching against stripped versions of your configured version names. @@ -471,7 +471,7 @@ in your component configuration. For example: - **Preserving Other Parameters**: Other URL parameters are preserved when switching versions. -#### What Won't Work +#### What won't work - **Partial Matches**: `?v=8` won't match `v0.8` with the default implementation. @@ -481,7 +481,7 @@ require more sophisticated matching logic. (Reach out to the docs team if requir - **Non-Standard Formats**: Version formats not accounted for in the matching logic might fail. -#### Best Practices +#### Best practices 1. Keep version strings in consistent formats for predictable results. diff --git a/docs/_snippets/_direct_observability_integration_options.md b/docs/_snippets/_direct_observability_integration_options.md index 3e9ba34d9c5..f47619d7846 100644 --- a/docs/_snippets/_direct_observability_integration_options.md +++ b/docs/_snippets/_direct_observability_integration_options.md @@ -7,7 +7,7 @@ import NativeAdvancedDashboard from '@site/static/images/cloud/manage/monitoring The ClickHouse data source plugin for Grafana enables visualization and exploration of data directly from ClickHouse using system tables. This approach works well for monitoring performance and creating custom dashboards for detailed system analysis. For plugin installation and configuration details, see the ClickHouse [data source plugin](/integrations/grafana). For a complete monitoring setup using the Prometheus-Grafana mix-in with pre-built dashboards and alerting rules, see [Monitor ClickHouse with the new Prometheus-Grafana mix-in](https://clickhouse.com/blog/monitor-with-new-prometheus-grafana-mix-in). -### Direct Datadog Integration {#direct-datadog} +### Direct Datadog integration {#direct-datadog} Datadog offers a Clickhouse Monitoring plugin for its agent which queries system tables directly. This integration provides comprehensive database monitoring with cluster awareness through clusterAllReplicas functionality. :::note diff --git a/docs/_snippets/_users-and-roles-common.md b/docs/_snippets/_users-and-roles-common.md index 0d2d3fbdf00..698a891443d 100644 --- a/docs/_snippets/_users-and-roles-common.md +++ b/docs/_snippets/_users-and-roles-common.md @@ -183,7 +183,7 @@ Roles are used to define groups of users for certain privileges instead of manag -##### Log into the clickhouse client using the `clickhouse_admin` user {#login-admin-user} +##### Log into the ClickHouse client using the `clickhouse_admin` user {#login-admin-user} ```bash clickhouse-client --user clickhouse_admin --password password diff --git a/docs/about-us/distinctive-features.md b/docs/about-us/distinctive-features.md index c90b05bd416..90d48fcdba3 100644 --- a/docs/about-us/distinctive-features.md +++ b/docs/about-us/distinctive-features.md @@ -88,7 +88,7 @@ ClickHouse uses asynchronous multi-master replication. After being written to an For more information, see the section [Data replication](../engines/table-engines/mergetree-family/replication.md). -## Role-Based Access Control {#role-based-access-control} +## Role-based access control {#role-based-access-control} ClickHouse implements user account management using SQL queries and allows for [role-based access control configuration](/guides/sre/user-management/index.md) similar to what can be found in the ANSI SQL standard and popular relational database management systems. diff --git a/docs/best-practices/index.md b/docs/best-practices/index.md index 4fdaa772493..43a2ebf41d7 100644 --- a/docs/best-practices/index.md +++ b/docs/best-practices/index.md @@ -9,7 +9,7 @@ doc_type: 'landing-page' import TableOfContents from '@site/docs/best-practices/_snippets/_table_of_contents.md'; -# Best Practices in ClickHouse {#best-practices-in-clickhouse} +# Best practices in ClickHouse {#best-practices-in-clickhouse} This section provides the best practices you will want to follow to get the most out of ClickHouse. diff --git a/docs/best-practices/minimize_optimize_joins.md b/docs/best-practices/minimize_optimize_joins.md index c3bec49726e..4af134b2f7c 100644 --- a/docs/best-practices/minimize_optimize_joins.md +++ b/docs/best-practices/minimize_optimize_joins.md @@ -25,7 +25,7 @@ Not all data needs to be denormalized — focus on the attributes that are frequ For a full guide on denormalizing data in ClickHouse see [here](/data-modeling/denormalization). -## When JOINs are required {#when-joins-are-required} +## When joins are required {#when-joins-are-required} When JOINs are required, ensure you're using **at least version 24.12 and preferably the latest version**, as JOIN performance continues to improve with each new release. As of ClickHouse 24.12, the query planner now automatically places the smaller table on the right side of the join for optimal performance — a task that previously had to be done manually. Even more enhancements are coming soon, including more aggressive filter pushdown and automatic re-ordering of multiple joins. @@ -42,7 +42,7 @@ Follow these best practices to improve JOIN performance: When using dictionaries for JOINs in ClickHouse, it's important to understand that dictionaries, by design, do not allow duplicate keys. During data loading, any duplicate keys are silently deduplicated—only the last loaded value for a given key is retained. This behavior makes dictionaries ideal for one-to-one or many-to-one relationships where only the latest or authoritative value is needed. However, using a dictionary for a one-to-many or many-to-many relationship (e.g. joining roles to actors where an actor can have multiple roles) will result in silent data loss, as all but one of the matching rows will be discarded. As a result, dictionaries are not suitable for scenarios requiring full relational fidelity across multiple matches. ::: -## Choosing the correct JOIN Algorithm {#choosing-the-right-join-algorithm} +## Choosing the correct join algorithm {#choosing-the-right-join-algorithm} ClickHouse supports several JOIN algorithms that trade off between speed and memory: diff --git a/docs/best-practices/sizing-and-hardware-recommendations.md b/docs/best-practices/sizing-and-hardware-recommendations.md index 03fe792c5bd..e1cfbf5d16e 100644 --- a/docs/best-practices/sizing-and-hardware-recommendations.md +++ b/docs/best-practices/sizing-and-hardware-recommendations.md @@ -59,7 +59,7 @@ For data warehousing workloads and ad-hoc analytical queries, we recommend the [ There is no standard CPU utilization target for ClickHouse. Utilize a tool such as [iostat](https://linux.die.net/man/1/iostat) to measure average CPU usage, and accordingly adjust the size of your servers to manage unexpected traffic spikes. However, for analytical or data warehousing use cases with ad-hoc queries, you should target 10-20% CPU utilization. -### How many CPU cores should I use? {#how-many-cpu-cores-should-i-use} +### How many CPU cores should i use? {#how-many-cpu-cores-should-i-use} The number of CPUs you should use depends on your workload. However, we generally recommend the following memory-to-CPU-core ratios based on your CPU type: @@ -164,7 +164,7 @@ To provide guidance (not recommendations), the following are example configurati -### Fortune 500 Telecom Operator for a logging use case {#fortune-500-telecom-operator-for-a-logging-use-case} +### Fortune 500 telecom operator for a logging use case {#fortune-500-telecom-operator-for-a-logging-use-case} diff --git a/docs/chdb/api/python.md b/docs/chdb/api/python.md index 99a7c15240c..bb31e99b4ea 100644 --- a/docs/chdb/api/python.md +++ b/docs/chdb/api/python.md @@ -7,9 +7,9 @@ keywords: ['chdb', 'embedded', 'clickhouse-lite', 'python', 'api', 'reference'] doc_type: 'reference' --- -# Python API Reference +# Python API reference -## Core Query Functions {#core-query-functions} +## Core query functions {#core-query-functions} ### `chdb.query` {#chdb-query} @@ -235,7 +235,7 @@ chdb.to_df(r) 0 1 hello ``` -## Connection and Session Management {#connection-session-management} +## Connection and session management {#connection-session-management} The following Session Functions are available: @@ -331,7 +331,7 @@ Creating a new connection will close any existing connection. - [`Connection`](#chdb-state-sqlitelike-connection) - Database connection class - [`Cursor`](#chdb-state-sqlitelike-cursor) - Database cursor for DB-API 2.0 operations -## Exception Handling {#chdb-exceptions} +## Exception handling {#chdb-exceptions} ### **class** `chdb.ChdbError` {#chdb_chdbError} @@ -698,7 +698,7 @@ id,name - [`send_query()`](#chdb-session-session-send_query) - For streaming query execution - [`sql`](#chdb-session-session-sql) - Alias for this method -## State Management {#chdb-state-management} +## State management {#chdb-state-management} ### `chdb.state.connect` {#chdb_state_connect} @@ -1534,7 +1534,7 @@ text object dtype: object ``` -## DataFrame Integration {#dataframe-integration} +## DataFrame integration {#dataframe-integration} ### **class** `chdb.dataframe.Table` {#chdb-dataframe-table} @@ -1544,7 +1544,7 @@ Bases: class chdb.dataframe.Table(*args: Any, **kwargs: Any) ``` -## Database API (DBAPI) 2.0 Interface {#database-api-interface} +## Database API (DBAPI) 2.0 interface {#database-api-interface} chDB provides a Python DB-API 2.0 compatible interface for database connectivity, allowing you to use chDB with tools and frameworks that expect standard database interfaces. @@ -1558,7 +1558,7 @@ The chDB DB-API 2.0 interface includes: --- -### Core Functions {#core-functions} +### Core functions {#core-functions} The Database API (DBAPI) 2.0 Interface implements the following core functions: @@ -1635,7 +1635,7 @@ chdb.dbapi.Binary(x) --- -### Connection Class {#connection-class} +### Connection class {#connection-class} #### **class** `chdb.dbapi.connections.Connection(path=None)` {#chdb-dbapi-connections-connection} @@ -1923,7 +1923,7 @@ transactions. Provided for DB-API 2.0 compliance. --- -### Cursor Class {#cursor-class} +### Cursor class {#cursor-class} #### **class** `chdb.dbapi.cursors.Cursor` {#chdb-dbapi-cursors-cursor} @@ -2351,7 +2351,7 @@ chDB automatically handles output sizing internally. --- -### Error Classes {#error-classes} +### Error classes {#error-classes} Exception classes for chdb database operations. @@ -2756,7 +2756,7 @@ This follows the Python DB API 2.0 specification for warning exceptions. --- -### Module Constants {#module-constants} +### Module constants {#module-constants} #### `chdb.dbapi.apilevel = '2.0'` {#apilevel} @@ -2817,7 +2817,7 @@ errors defaults to ‘strict’. --- -### Type Constants {#type-constants} +### Type constants {#type-constants} #### `chdb.dbapi.STRING = frozenset({247, 253, 254})` {#string-type} @@ -3099,7 +3099,7 @@ with dbapi.connect("test.chdb") as conn: - Parameter binding syntax follows format style: `%s` ::: -## User-Defined Functions (UDF) {#user-defined-functions} +## User-defined functions (UDF) {#user-defined-functions} User-defined functions module for chDB. @@ -3347,9 +3347,9 @@ chdb.utils.infer_data_types`(column_data: Dict[str, List[Any]], n_rows: int = 10 |---------------|----------------------------------------------------------------------------| | `List[tuple]` | A list of tuples, each containing a column name and its inferred data type | -## Abstract Base Classes {#abstract-base-classes} +## Abstract base classes {#abstract-base-classes} -### **class** `chdb.rwabc.PyReader`(data: Any)` {#pyreader} +### **class** `chdb.rwabc.PyReader`(data: any)` {#pyreader} Bases: `ABC` @@ -3422,7 +3422,7 @@ abstractmethod write(col_names: List[str], columns: List[List[Any]]) → None | `col_names` | `List[str]` | List of column names that are being written | | `columns` | `List[List[Any]]` | List of columns data, each column is represented by a list | -## Exception Handling {#exception-handling} +## Exception handling {#exception-handling} ### **class** `chdb.ChdbError` {#chdberror} @@ -3469,7 +3469,7 @@ You should catch this exception when handling potentially failing queries to provide appropriate error handling in your application. ::: -## Version Information {#version-information} +## Version information {#version-information} ### `chdb.chdb_version = ('3', '6', '0')` {#chdb-version} diff --git a/docs/chdb/getting-started.md b/docs/chdb/getting-started.md index 875f13ff221..93d0f05afdd 100644 --- a/docs/chdb/getting-started.md +++ b/docs/chdb/getting-started.md @@ -340,7 +340,7 @@ We could write the following code: df["likeDislikeRatio"] = df["likeCount"] / df["dislikeCount"] ``` -## Querying a Pandas dataframe {#querying-a-pandas-dataframe} +## Querying a Pandas DataFrame {#querying-a-pandas-dataframe} We can then query that DataFrame from chDB: diff --git a/docs/chdb/guides/query-remote-clickhouse.md b/docs/chdb/guides/query-remote-clickhouse.md index ca5706e46dc..f08815377bd 100644 --- a/docs/chdb/guides/query-remote-clickhouse.md +++ b/docs/chdb/guides/query-remote-clickhouse.md @@ -128,7 +128,7 @@ sklearn_df.sort_values(by=["x"], ascending=False).head(n=10) 2383 2024-09-23 1777554 ``` -## Merging Pandas DataFrames {#merging-pandas-dataframes} +## Merging Pandas dataframes {#merging-pandas-dataframes} We now have two DataFrames, which we can merge together based on date (which is the `x` column) like this: @@ -166,7 +166,7 @@ df.head(n=5) 4 2018-03-02 5 23842 0.000210 ``` -## Querying Pandas DataFrames {#querying-pandas-dataframes} +## Querying Pandas dataframes {#querying-pandas-dataframes} Next, let's say we want to find the dates with the best and worst ratios. We can go back to chDB and compute those values: diff --git a/docs/chdb/guides/querying-pandas.md b/docs/chdb/guides/querying-pandas.md index 28ec1619571..efc08655133 100644 --- a/docs/chdb/guides/querying-pandas.md +++ b/docs/chdb/guides/querying-pandas.md @@ -155,7 +155,7 @@ match_id 3943077 Name: 0, dtype: object ``` -## Querying Pandas DataFrames {#querying-pandas-dataframes} +## Querying Pandas dataframes {#querying-pandas-dataframes} Next, let's see how to query these DataFrames using chDB. We'll import the library: @@ -279,7 +279,7 @@ LIMIT 10 9 Carlos Eccehomo Cuesta Figueroa 50 ``` -## Joining Pandas DataFrames {#joining-pandas-dataframes} +## Joining Pandas dataframes {#joining-pandas-dataframes} We can also join DataFrames together in a query. For example, to get an overview of the match, we could write the following query: diff --git a/docs/chdb/index.md b/docs/chdb/index.md index e7b1d6efa3b..3555fe3efd8 100644 --- a/docs/chdb/index.md +++ b/docs/chdb/index.md @@ -34,7 +34,7 @@ chDB has the following language bindings: * [Bun](install/bun.md) * [C and C++](install/c.md) -## How do I get started? {#how-do-i-get-started} +## How do i get started? {#how-do-i-get-started} * If you're using [Go](install/go.md), [Rust](install/rust.md), [NodeJS](install/nodejs.md), [Bun](install/bun.md) or [C and C++](install/c.md), take a look at the corresponding language pages. * If you're using Python, see the [getting started developer guide](getting-started.md) or the [chDB on-demand course](https://learn.clickhouse.com/user_catalog_class/show/1901178). There are also guides showing how to do common tasks like: diff --git a/docs/chdb/install/bun.md b/docs/chdb/install/bun.md index 0e9aeea9eaf..f228bfbbfbb 100644 --- a/docs/chdb/install/bun.md +++ b/docs/chdb/install/bun.md @@ -13,7 +13,7 @@ chDB-bun provides experimental FFI (Foreign Function Interface) bindings for chD ## Installation {#installation} -### Step 1: Install system dependencies {#install-system-dependencies} +### Step 1: install system dependencies {#install-system-dependencies} First, install the required system dependencies: @@ -27,7 +27,7 @@ curl -sL https://lib.chdb.io | bash You'll need either `gcc` or `clang` installed on your system: -### Step 2: Install chDB-bun {#install-chdb-bun} +### Step 2: install chdb-bun {#install-chdb-bun} ```bash # Install from the GitHub repository diff --git a/docs/chdb/install/c.md b/docs/chdb/install/c.md index f61fc917170..1c6f5b2de8a 100644 --- a/docs/chdb/install/c.md +++ b/docs/chdb/install/c.md @@ -7,13 +7,13 @@ keywords: ['chdb', 'c', 'cpp', 'embedded', 'clickhouse', 'sql', 'olap', 'api'] doc_type: 'guide' --- -# chDB for C and C++ +# chDB for c and c++ chDB provides a native C/C++ API for embedding ClickHouse functionality directly into your applications. The API supports both simple queries and advanced features like persistent connections and streaming query results. ## Installation {#installation} -### Step 1: Install libchdb {#install-libchdb} +### Step 1: install libchdb {#install-libchdb} Install the chDB library on your system: @@ -21,7 +21,7 @@ Install the chDB library on your system: curl -sL https://lib.chdb.io | bash ``` -### Step 2: Include headers {#include-headers} +### Step 2: include headers {#include-headers} Include the chDB header in your project: @@ -29,7 +29,7 @@ Include the chDB header in your project: #include ``` -### Step 3: Link library {#link-library} +### Step 3: link library {#link-library} Compile and link your application with chDB: @@ -41,7 +41,7 @@ gcc -o myapp myapp.c -lchdb g++ -o myapp myapp.cpp -lchdb ``` -## C Examples {#c-examples} +## C examples {#c-examples} ### Basic connection and queries {#basic-connection-queries} diff --git a/docs/chdb/install/go.md b/docs/chdb/install/go.md index b029fdbeaa5..5d59028f922 100644 --- a/docs/chdb/install/go.md +++ b/docs/chdb/install/go.md @@ -13,7 +13,7 @@ chDB-go provides Go bindings for chDB, enabling you to run ClickHouse queries di ## Installation {#installation} -### Step 1: Install libchdb {#install-libchdb} +### Step 1: install libchdb {#install-libchdb} First, install the chDB library: @@ -21,7 +21,7 @@ First, install the chDB library: curl -sL https://lib.chdb.io | bash ``` -### Step 2: Install chdb-go {#install-chdb-go} +### Step 2: install chdb-go {#install-chdb-go} Install the Go package: @@ -52,7 +52,7 @@ chDB-go includes a CLI for quick queries: ./chdb-go --path /tmp/chdb ``` -### Go Library - quick start {#quick-start} +### Go library - quick start {#quick-start} #### Stateless queries {#stateless-queries} diff --git a/docs/chdb/install/nodejs.md b/docs/chdb/install/nodejs.md index 1976a669287..c5fc3882255 100644 --- a/docs/chdb/install/nodejs.md +++ b/docs/chdb/install/nodejs.md @@ -45,7 +45,7 @@ const systemInfo = query("SELECT * FROM system.functions LIMIT 5", "Pretty"); console.log("System functions:", systemInfo); ``` -### Session-Based queries {#session-based-queries} +### Session-based queries {#session-based-queries} ```javascript const { Session } = require("chdb"); diff --git a/docs/chdb/install/python.md b/docs/chdb/install/python.md index 1300f1556e7..e47437522fb 100644 --- a/docs/chdb/install/python.md +++ b/docs/chdb/install/python.md @@ -32,7 +32,7 @@ python3 -m chdb "SELECT 1, 'abc'" Pretty python3 -m chdb "SELECT version()" JSON ``` -### Basic python usage {#basic-python-usage} +### Basic Python usage {#basic-python-usage} ```python import chdb @@ -279,7 +279,7 @@ result = sess.query(""" See also: [test_stateful.py](https://github.com/chdb-io/chdb/blob/main/tests/test_stateful.py). -### Python DB-API 2.0 interface {#python-db-api-20} +### Python db-api 2.0 interface {#python-db-api-20} Standard database interface for compatibility with existing Python applications: @@ -508,7 +508,7 @@ sess.close() ### Python table engine {#python-table-engine} -#### Query Pandas DataFrames {#query-pandas-dataframes} +#### Query Pandas dataframes {#query-pandas-dataframes} ```python import chdb @@ -563,7 +563,7 @@ window_result = chdb.query(""" print(window_result) ``` -#### Custom data sources with PyReader {#custom-data-sources-pyreader} +#### Custom data sources with pyreader {#custom-data-sources-pyreader} Implement custom data readers for specialized data sources: @@ -686,12 +686,12 @@ chDB consistently outperforms other embedded engines: ```python import chdb -# 1. Use appropriate output formats +# 1. use appropriate output formats df_result = chdb.query("SELECT * FROM large_table", "DataFrame") # For analysis arrow_result = chdb.query("SELECT * FROM large_table", "Arrow") # For interop native_result = chdb.query("SELECT * FROM large_table", "Native") # For chDB-to-chDB -# 2. Optimize queries with settings +# 2. optimize queries with settings fast_result = chdb.query(""" SELECT customer_id, sum(amount) FROM sales @@ -702,7 +702,7 @@ fast_result = chdb.query(""" use_uncompressed_cache = 1 """, "DataFrame") -# 3. Leverage streaming for large datasets +# 3. leverage streaming for large datasets from chdb import session sess = session.Session() @@ -739,7 +739,7 @@ with sess.send_query("SELECT customer_id, sum(amount) as total FROM large_sales print(f"Final result: {processed_rows} customers processed, total amount: {total_amount}") -# Stream to external systems (e.g., Delta Lake) +# Stream to external systems (e.g., delta lake) stream = sess.send_query("SELECT * FROM large_sales LIMIT 1000000", "Arrow") batch_reader = stream.record_batch(rows_per_batch=50000) @@ -747,7 +747,7 @@ batch_reader = stream.record_batch(rows_per_batch=50000) for batch in batch_reader: print(f"Processing batch with {batch.num_rows} rows...") # Transform or export each batch - # df_batch = batch.to_pandas() + # Df_batch = batch.to_pandas() # process_batch(df_batch) stream.close() diff --git a/docs/chdb/install/rust.md b/docs/chdb/install/rust.md index b34bed661d3..027cd803a41 100644 --- a/docs/chdb/install/rust.md +++ b/docs/chdb/install/rust.md @@ -51,7 +51,7 @@ fn main() -> Result<(), Box> { } ``` -### Stateful usage (Sessions) {#stateful-usage-sessions} +### Stateful usage (sessions) {#stateful-usage-sessions} For queries requiring persistent state like databases and tables: diff --git a/docs/cloud/features/02_integrations.md b/docs/cloud/features/02_integrations.md index cc3a4fed100..029d8633bbc 100644 --- a/docs/cloud/features/02_integrations.md +++ b/docs/cloud/features/02_integrations.md @@ -45,7 +45,7 @@ ClickPipes can be used for long-term streaming needs or one-time data loading jo | Google Cloud Storage | |Object Storage| Stable | Configure ClickPipes to ingest large volumes of data from object storage. | | DigitalOcean Spaces | | Object Storage | Stable | Configure ClickPipes to ingest large volumes of data from object storage. | Azure Blob Storage | | Object Storage | Private Beta | Configure ClickPipes to ingest large volumes of data from object storage. -| [Amazon Kinesis](/integrations/clickpipes/kinesis) | |Streaming| Stable | Configure ClickPipes and start ingesting streaming data from Amazon Kinesis into ClickHouse cloud. | +| [Amazon Kinesis](/integrations/clickpipes/kinesis) | |Streaming| Stable | Configure ClickPipes and start ingesting streaming data from Amazon Kinesis into ClickHouse Cloud. | | [Postgres](/integrations/clickpipes/postgres) | |DBMS| Stable | Configure ClickPipes and start ingesting data from Postgres into ClickHouse Cloud. | | [MySQL](/integrations/clickpipes/mysql) | |DBMS| Private Beta | Configure ClickPipes and start ingesting data from MySQL into ClickHouse Cloud. | | [MongoDB](/integrations/clickpipes/mongodb) | |DBMS| Private Preview | Configure ClickPipes and start ingesting data from MongoDB into ClickHouse Cloud. | diff --git a/docs/cloud/features/03_sql_console_features/01_sql-console.md b/docs/cloud/features/03_sql_console_features/01_sql-console.md index b071ab74325..b4688d4553c 100644 --- a/docs/cloud/features/03_sql_console_features/01_sql-console.md +++ b/docs/cloud/features/03_sql_console_features/01_sql-console.md @@ -43,7 +43,7 @@ import update_query_name from '@site/static/images/cloud/sqlconsole/update-query import update_subtitle_etc from '@site/static/images/cloud/sqlconsole/update-subtitle-etc.png'; import adjust_axis_scale from '@site/static/images/cloud/sqlconsole/adjust-axis-scale.png'; -# SQL Console +# SQL console SQL console is the fastest and easiest way to explore and query your databases in ClickHouse Cloud. You can use the SQL console to: diff --git a/docs/cloud/features/03_sql_console_features/02_query-insights.md b/docs/cloud/features/03_sql_console_features/02_query-insights.md index d1f95976720..68b3a4893e9 100644 --- a/docs/cloud/features/03_sql_console_features/02_query-insights.md +++ b/docs/cloud/features/03_sql_console_features/02_query-insights.md @@ -14,7 +14,7 @@ import insights_recent from '@site/static/images/cloud/sqlconsole/insights_recen import insights_drilldown from '@site/static/images/cloud/sqlconsole/insights_drilldown.png'; import insights_query_info from '@site/static/images/cloud/sqlconsole/insights_query_info.png'; -# Query Insights +# Query insights The **Query Insights** feature makes ClickHouse's built-in query log easier to use through various visualizations and tables. ClickHouse's `system.query_log` table is a key source of information for query optimization, debugging, and monitoring overall cluster health and performance. diff --git a/docs/cloud/features/04_infrastructure/deployment-options.md b/docs/cloud/features/04_infrastructure/deployment-options.md index 07cda6c3a3d..1e0befeebdf 100644 --- a/docs/cloud/features/04_infrastructure/deployment-options.md +++ b/docs/cloud/features/04_infrastructure/deployment-options.md @@ -6,7 +6,7 @@ keywords: ['bring yor own cloud', 'byoc', 'private', 'government', 'self-deploye doc_type: 'reference' --- -# ClickHouse Deployment Options +# ClickHouse deployment options ClickHouse provides a range of deployment options to cater to diverse customer requirements, offering varying degrees of control, compliance, and operational overhead. This document outlines the distinct deployment types available, enabling users to select the optimal solution that aligns with their specific architectural preferences, regulatory obligations, and resource management strategies. @@ -20,7 +20,7 @@ It offers consumption-based pricing, and automatic scaling, ensuring reliable an Learn more about [ClickHouse Cloud](/getting-started/quick-start/cloud). -## Bring Your Own Cloud {#byoc} +## Bring Your Own Cloud (BYOC) {#byoc} ClickHouse Bring Your Own Cloud (BYOC) allows organizations to deploy and manage ClickHouse within their own cloud environment while leveraging a managed service layer. This option bridges the gap between the fully managed experience of ClickHouse Cloud and the complete control of self-managed deployments. With ClickHouse BYOC, users retain control over their data, infrastructure, and security policies, meeting specific compliance and regulatory requirements, while offloading operational tasks like patching, monitoring, and scaling to the ClickHouse. This model offers the flexibility of a private cloud deployment with the benefits of a managed service, making it suitable for large-scale deployments at enterprises with stringent security, governance, and data residency needs. @@ -32,7 +32,7 @@ ClickHouse Private is a self-deployed version of ClickHouse, leveraging the same Learn more about [ClickHouse Private](/cloud/infrastructure/clickhouse-private). -## ClickHouse Government {#clickhouse-government} +## ClickHouse government {#clickhouse-government} ClickHouse Government is a self-deployed version of ClickHouse designed to meet the unique and rigorous demands of government agencies and public sector organizations that need isolated and accredited environments. This deployment option provides a highly secure, compliant, and isolated environment, focusing on FIPS 140-3 compliance utilizing OpenSSL, additional system hardening, and vulnerability management. It leverages the robust capabilities of ClickHouse Cloud while integrating specialized features and configurations to address the specific operational and security requirements of governmental entities. With ClickHouse Government, agencies can achieve high-performance analytics on sensitive data within a controlled and accredited infrastructure, backed by expert support tailored to public sector needs. diff --git a/docs/cloud/features/04_infrastructure/replica-aware-routing.md b/docs/cloud/features/04_infrastructure/replica-aware-routing.md index 425a4296301..fb6b65e4592 100644 --- a/docs/cloud/features/04_infrastructure/replica-aware-routing.md +++ b/docs/cloud/features/04_infrastructure/replica-aware-routing.md @@ -26,7 +26,7 @@ When Envoy receives a hostname that matches such a pattern, it will compute the Note the original hostname will still use `LEAST_CONNECTION` load balancing, which is the default routing algorithm. -## Limitations of Replica-aware routing {#limitations-of-replica-aware-routing} +## Limitations of replica-aware routing {#limitations-of-replica-aware-routing} ### Replica-aware routing does not guarantee isolation {#replica-aware-routing-does-not-guarantee-isolation} diff --git a/docs/cloud/features/04_infrastructure/shared-merge-tree.md b/docs/cloud/features/04_infrastructure/shared-merge-tree.md index 3b5e400d84c..264e66cdfe2 100644 --- a/docs/cloud/features/04_infrastructure/shared-merge-tree.md +++ b/docs/cloud/features/04_infrastructure/shared-merge-tree.md @@ -11,7 +11,7 @@ import shared_merge_tree from '@site/static/images/cloud/reference/shared-merge- import shared_merge_tree_2 from '@site/static/images/cloud/reference/shared-merge-tree-2.png'; import Image from '@theme/IdealImage'; -# SharedMergeTree table engine +# Sharedmergetree table engine The SharedMergeTree table engine family is a cloud-native replacement of the ReplicatedMergeTree engines that is optimized to work on top of shared storage (e.g. Amazon S3, Google Cloud Storage, MinIO, Azure Blob Storage). There is a SharedMergeTree analog for every specific MergeTree engine type, i.e. ReplacingSharedMergeTree replaces ReplacingReplicatedMergeTree. @@ -45,7 +45,7 @@ This table serves as the alternative to `system.replication_queue` for SharedMer This table is the alternative to `system.replicated_fetches` SharedMergeTree. It contains information about current in-progress fetches of primary keys and checksums into memory. -## Enabling SharedMergeTree {#enabling-sharedmergetree} +## Enabling sharedmergetree {#enabling-sharedmergetree} `SharedMergeTree` is enabled by default. diff --git a/docs/cloud/features/05_admin_features/api/api-overview.md b/docs/cloud/features/05_admin_features/api/api-overview.md index cd6269e39c7..a5e2974249f 100644 --- a/docs/cloud/features/05_admin_features/api/api-overview.md +++ b/docs/cloud/features/05_admin_features/api/api-overview.md @@ -19,7 +19,7 @@ organization, and more. [Learn how to create your first API key and start using the ClickHouse Cloud API.](/cloud/manage/openapi) -## Swagger (OpenAPI) Endpoint and UI {#swagger-openapi-endpoint-and-ui} +## Swagger (OpenAPI) endpoint and UI {#swagger-openapi-endpoint-and-ui} The ClickHouse Cloud API is built on the open-source [OpenAPI specification](https://www.openapis.org/) to allow for predictable client-side consumption. If you need to programmatically @@ -58,7 +58,7 @@ If your organization has been migrated to one of the [new pricing plans](https:/ You will now also be able to specify the `num_replicas` field as a property of the service resource. ::: -## Terraform and OpenAPI New Pricing: Replica Settings Explained {#terraform-and-openapi-new-pricing---replica-settings-explained} +## Terraform and OpenAPI new pricing: replica settings explained {#terraform-and-openapi-new-pricing---replica-settings-explained} The number of replicas each service will be created with defaults to 3 for the Scale and Enterprise tiers, while it defaults to 1 for the Basic tier. For the Scale and the Enterprise tiers it is possible to adjust it by passing a `numReplicas` field in the service creation request. diff --git a/docs/cloud/features/05_admin_features/api/postman.md b/docs/cloud/features/05_admin_features/api/postman.md index 1fd9a9d142e..86963b5d568 100644 --- a/docs/cloud/features/05_admin_features/api/postman.md +++ b/docs/cloud/features/05_admin_features/api/postman.md @@ -81,7 +81,7 @@ The Postman Application is available for use within a web browser or can be down * [Variables](https://learning.postman.com/docs/sending-requests/variables/) enable the storage and reuse of values in Postman allowing for easier API testing. -#### Set the organization ID and Service ID {#set-the-organization-id-and-service-id} +#### Set the organization id and service id {#set-the-organization-id-and-service-id} * Within the "Collection", click the "Variable" tab in the middle pane (The Base URL will have been set by the earlier API import): * Below `baseURL` click the open field "Add new value", and Substitute your organization ID and service ID: @@ -90,7 +90,7 @@ The Postman Application is available for use within a web browser or can be down ## Test the ClickHouse Cloud API functionalities {#test-the-clickhouse-cloud-api-functionalities} -### Test "GET list of available organizations" {#test-get-list-of-available-organizations} +### Test "get list of available organizations" {#test-get-list-of-available-organizations} * Under the "OpenAPI spec for ClickHouse Cloud", expand the folder > V1 > organizations * Click "GET list of available organizations" and press the blue "Send" button on the right: @@ -101,7 +101,7 @@ The Postman Application is available for use within a web browser or can be down -### Test "GET organizational details" {#test-get-organizational-details} +### Test "get organizational details" {#test-get-organizational-details} * Under the `organizationid` folder, navigate to "GET organizational details": * In the middle frame menu under Params an `organizationid` is required. @@ -118,7 +118,7 @@ The Postman Application is available for use within a web browser or can be down * The returned results should deliver your organization details with "status": 200. (If you receive a "status" 400 with no organization information your configuration is not correct). -### Test "GET service details" {#test-get-service-details} +### Test "get service details" {#test-get-service-details} * Click "GET service details" * Edit the Values for `organizationid` and `serviceid` with `{{orgid}}` and `{{serviceid}}` respectively. diff --git a/docs/cloud/features/06_security.md b/docs/cloud/features/06_security.md index 49c4e2b716d..f869265b23f 100644 --- a/docs/cloud/features/06_security.md +++ b/docs/cloud/features/06_security.md @@ -7,81 +7,81 @@ doc_type: 'reference' keywords: ['security', 'cloud security', 'access control', 'compliance', 'data protection'] --- -# ClickHouse Cloud Security +# ClickHouse Cloud security This document details the security options and best practices available for ClickHouse organization and service protection. ClickHouse is dedicated to providing secure analytical database solutions; therefore, safeguarding data and service integrity is a priority. The information herein covers various methods designed to assist users in securing their ClickHouse environments. -## Cloud Console Authentication {#cloud-console-auth} +## Cloud console authentication {#cloud-console-auth} -### Password Authentication {#password-auth} +### Password authentication {#password-auth} ClickHouse Cloud console passwords are configured to NIST 800-63B standards with a minimum of 12 characters and 3 of 4 complexity requirements: upper case characters, lower case characters, numbers and/or special characters. Learn more about [password authentication](/cloud/security/manage-my-account#email-and-password). -### Social Single Sign-On (SSO) {#social-sso} +### Social single sign-on (SSO) {#social-sso} ClickHouse Cloud supports Google or Microsoft social authentication for single sign-on (SSO). Learn more about [social SSO](/cloud/security/manage-my-account#social-sso). -### Multi-Factor Authentication {#mfa} +### Multi-factor authentication {#mfa} Users using email and password or social SSO may also configure multi-factor authentication utilizing an authenticator app such as Authy or Google Authenticator. Learn more about [multi-factor authentication](/cloud/security/manage-my-account/#mfa). -### Security Assertion Markup Language (SAML) Authentication {#saml-auth} +### Security assertion markup language (SAML) authentication {#saml-auth} Enterprise customers may configure SAML authentication. Learn more about [SAML authentication](/cloud/security/saml-setup). -### API Authentication {#api-auth} +### API authentication {#api-auth} Customers may configure API keys for use with OpenAPI, Terraform and Query API endpoints. Learn more about [API authentication](/cloud/manage/openapi). -## Database Authentication {#database-auth} +## Database authentication {#database-auth} -### Database Password Authentication {#db-password-auth} +### Database password authentication {#db-password-auth} ClickHouse database user passwords are configured to NIST 800-63B standards with a minimum of 12 characters and complexity requirements: upper case characters, lower case characters, numbers and/or special characters. Learn more about [database password authentication](/cloud/security/manage-database-users#database-user-id--password). -### Secure Shell (SSH) Database Authentication {#ssh-auth} +### Secure shell (ssh) database authentication {#ssh-auth} ClickHouse database users may be configured to use SSH authentication. Learn more about [SSH authentication](/cloud/security/manage-database-users#database-ssh). -## Access Control {#access-control} +## Access control {#access-control} -### Console Role-Based Access Control (RBAC) {#console-rbac} +### Console role-based access control (RBAC) {#console-rbac} ClickHouse Cloud supports role assignment for organization, service and database permissions. Database permissions using this method are supported in SQL console only. Learn more about [console RBAC](/cloud/security/console-roles). -### Database User Grants {#database-user-grants} +### Database user grants {#database-user-grants} ClickHouse databases support granular permission management and role-based access via user grants. Learn more about [database user grants](/cloud/security/manage-database-users#database-permissions). -## Network Security {#network-security} +## Network security {#network-security} -### IP Filters {#ip-filters} +### Ip filters {#ip-filters} Configure IP filters to limit inbound connections to your ClickHouse service. Learn more about [IP filters](/cloud/security/setting-ip-filters). -### Private Connectivity {#private-connectivity} +### Private connectivity {#private-connectivity} Connect to your ClickHouse clusters from AWS, GCP or Azure using private connectivity. @@ -89,39 +89,39 @@ Learn more about [private connectivity](/cloud/security/connectivity/private-net ## Encryption {#encryption} -### Storage Level Encryption {#storage-encryption} +### Storage level encryption {#storage-encryption} ClickHouse Cloud encrypts data at rest by default using cloud provider-managed AES 256 keys. Learn more about [storage encryption](/cloud/security/cmek#storage-encryption). -### Transparent Data Encryption {#tde} +### Transparent data encryption {#tde} In addition to storage encryption, ClickHouse Cloud Enterprise customers may enable database level transparent data encryption for additional protection. Learn more about [transparent data encryption](/cloud/security/cmek#transparent-data-encryption-tde). -### Customer Managed Encryption Keys {#cmek} +### Customer managed encryption keys {#cmek} ClickHouse Cloud Enterprise customers may use their own key for database level encryption. Learn more about [customer managed encryption keys](/cloud/security/cmek#customer-managed-encryption-keys-cmek). -## Auditing and Logging {#auditing-logging} +## Auditing and logging {#auditing-logging} -### Console Audit Log {#console-audit-log} +### Console audit log {#console-audit-log} Activities within the console are logged. Logs are available for review and export. Learn more about [console audit logs](/cloud/security/audit-logging/console-audit-log). -### Database Audit Logs {#database-audit-logs} +### Database audit logs {#database-audit-logs} Activities within the database are logged. Logs are available for review and export. Learn more about [database audit logs](/cloud/security/audit-logging/database-audit-log). -### BYOC Security Playbook {#byoc-security-playbook} +### BYOC security playbook {#byoc-security-playbook} Sample detection queries for security teams managing ClickHouse BYOC instances. @@ -129,19 +129,19 @@ Learn more about the [BYOC security playbook](/cloud/security/audit-logging/byoc ## Compliance {#compliance} -### Security and Compliance Reports {#compliance-reports} +### Security and compliance reports {#compliance-reports} ClickHouse maintains a strong security and compliance program. Check back periodically for new third party audit reports. Learn more about [security and compliance reports](/cloud/security/compliance-overview). -### HIPAA Compliant Services {#hipaa-compliance} +### HIPAA compliant services {#hipaa-compliance} ClickHouse Cloud Enterprise customers may deploy services housing protected health information (PHI) to HIPAA compliant regions after signing a Business Associate Agreement (BAA). Learn more about [HIPAA compliance](/cloud/security/compliance/hipaa-onboarding). -### PCI Compliant Services {#pci-compliance} +### PCI compliant services {#pci-compliance} ClickHouse Cloud Enterprise customers may deploy services housing credit card information to PCI compliant regions. diff --git a/docs/cloud/features/07_monitoring/prometheus.md b/docs/cloud/features/07_monitoring/prometheus.md index 8ca2dc305eb..5c3832a8fa8 100644 --- a/docs/cloud/features/07_monitoring/prometheus.md +++ b/docs/cloud/features/07_monitoring/prometheus.md @@ -15,7 +15,7 @@ import prometheus_grafana_metrics_explorer from '@site/static/images/integration import prometheus_datadog from '@site/static/images/integrations/prometheus-datadog.png'; import Image from '@theme/IdealImage'; -# Prometheus Integration +# Prometheus integration The feature supports integrating [Prometheus](https://prometheus.io/) to monitor ClickHouse Cloud services. Access to Prometheus metrics is exposed via the [ClickHouse Cloud API](/cloud/manage/api/api-overview) endpoint that allows users to securely connect and export metrics into their Prometheus metrics collector. These metrics can be integrated with dashboards e.g., Grafana, Datadog for visualization. @@ -194,7 +194,7 @@ Users have two primary ways to integrate with Grafana: We provide instructions on using these options below, focusing on the details specific to the ClickHouse Cloud Prometheus Endpoint. -### Grafana Cloud with metrics endpoint {#grafana-cloud-with-metrics-endpoint} +### Grafana cloud with metrics endpoint {#grafana-cloud-with-metrics-endpoint} - Login to your Grafana Cloud account - Add a new connection by selecting the **Metrics Endpoint** @@ -213,7 +213,7 @@ Once configured, you should see the metrics in the drop-down that you can select -### Grafana Cloud with Alloy {#grafana-cloud-with-alloy} +### Grafana cloud with alloy {#grafana-cloud-with-alloy} If you are using Grafana Cloud, Alloy can be installed by navigating to the Alloy menu in Grafana and following the onscreen instructions: @@ -257,7 +257,7 @@ prometheus.remote_write "metrics_service" { Note the `honor_labels` configuration parameter needs to be set to `true` for the instance label to be properly populated. -### Grafana self-managed with Alloy {#grafana-self-managed-with-alloy} +### Grafana Self-managed with alloy {#grafana-self-managed-with-alloy} Self-managed users of Grafana can find the instructions for installing the Alloy agent [here](https://grafana.com/docs/alloy/latest/get-started/install/). We assume users have configured Alloy to send Prometheus metrics to their desired destination. The `prometheus.scrape` component below causes Alloy to scrape the ClickHouse Cloud Endpoint. We assume `prometheus.remote_write` receives the scraped metrics. Adjust the `forward_to key` to the target destination if this does not exist. diff --git a/docs/cloud/features/08_backups.md b/docs/cloud/features/08_backups.md index 5e34fd20ced..f76d040a7cf 100644 --- a/docs/cloud/features/08_backups.md +++ b/docs/cloud/features/08_backups.md @@ -53,7 +53,7 @@ In some rare scenarios, the backup scheduler will not respect the **Start Time** See ["Configure backup schedules"](/cloud/manage/backups/configurable-backups) for steps to configure your backups. -## Bring Your Own Bucket (BYOB) Backups {#byob} +## Bring your own bucket (BYOB) backups {#byob} @@ -81,7 +81,7 @@ To export backups to your own cloud account, you have two options: -##### Via Cloud Console UI {#via-ui} +##### Via cloud console UI {#via-ui} External backups can be [configured in the UI](/cloud/manage/backups/backup-restore-via-ui). By default, backups will then be taken daily (as specified in the [default backup policy](/cloud/features/backups#default-backup-policy)). diff --git a/docs/cloud/guides/SQL_console/query-endpoints.md b/docs/cloud/guides/SQL_console/query-endpoints.md index 32d50d29b61..cb6e6c0cc23 100644 --- a/docs/cloud/guides/SQL_console/query-endpoints.md +++ b/docs/cloud/guides/SQL_console/query-endpoints.md @@ -356,7 +356,7 @@ fetch( -### Request with query variables and version 2 on JSONCompactEachRow format {#request-with-query-variables-and-version-2-on-jsoncompacteachrow-format} +### Request with query variables and version 2 on jsoncompacteachrow format {#request-with-query-variables-and-version-2-on-jsoncompacteachrow-format} **Query API Endpoint SQL:** diff --git a/docs/cloud/guides/backups/03_bring_your_own_backup/01_export-backups-to-own-cloud-account.md b/docs/cloud/guides/backups/03_bring_your_own_backup/01_export-backups-to-own-cloud-account.md index c231e80a76e..3ba635430b4 100644 --- a/docs/cloud/guides/backups/03_bring_your_own_backup/01_export-backups-to-own-cloud-account.md +++ b/docs/cloud/guides/backups/03_bring_your_own_backup/01_export-backups-to-own-cloud-account.md @@ -51,7 +51,7 @@ In order to use role based authentication, please follow the Secure s3 [setup](h 2. Azure container name in the storage account. 3. Azure Blob within the container. -### Google Cloud Storage (GCS) {#google-cloud-storage-gcs} +### Google cloud storage (gcs) {#google-cloud-storage-gcs} 1. GCS endpoint, in the format: @@ -61,9 +61,9 @@ In order to use role based authentication, please follow the Secure s3 [setup](h 2. Access HMAC key and HMAC secret.
-# Backup / Restore +# Backup / restore -## Backup / Restore to AWS S3 Bucket {#backup--restore-to-aws-s3-bucket} +## Backup / restore to AWS S3 bucket {#backup--restore-to-aws-s3-bucket} ### Take a DB backup {#take-a-db-backup} @@ -99,7 +99,7 @@ FROM S3('https://testchbackups.s3.amazonaws.com/backups/', '', '', ' -#### 2. Steps to follow in ClickHouse Cloud {#cloud-steps} +#### 2. steps to follow in ClickHouse Cloud {#cloud-steps} Follow the steps below in the ClickHouse Cloud console to configure the external bucket: @@ -123,7 +123,7 @@ On the Settings page, click on Set up external backup: -##### Configure AWS IAM Role ARN and S3 bucket details {#configure-aws-iam-role-arn-and-s3-bucket-details} +##### Configure AWS iam role arn and S3 bucket details {#configure-aws-iam-role-arn-and-s3-bucket-details} On the next screen provide the AWS IAM Role ARN you just created and the S3 bucket URL in the following format: @@ -139,7 +139,7 @@ External Backups will now happen in your bucket on the default schedule. Alternatively, you can configure the backup schedule from the “Settings” page. If configured differently, the custom schedule is used to write backups to your bucket and the default schedule (backups every 24 hours) is used for backups in -the ClickHouse cloud owned bucket. +the ClickHouse Cloud owned bucket. ##### View backups stored in your bucket {#view-backups-stored-in-your-bucket} @@ -160,7 +160,7 @@ Follow the steps below to restore backups from AWS: Create a new service to restore the backup to. -##### Add service ARN {#add-service-arn} +##### Add service arn {#add-service-arn} Add the newly created service’s ARN (from the service settings page in Clickhouse Cloud console) to the trust policy for the IAM role. This is the same as the @@ -207,7 +207,7 @@ Follow the steps below to take backups to GCP: Create a storage bucket in your GCP account to export backups to. -##### Generate an HMAC Key and Secret {#generate-an-hmac-key-and-secret} +##### Generate an hmac key and secret {#generate-an-hmac-key-and-secret} Generate an HMAC Key and Secret, which is required for password-based authentication. Follow the steps below to generate the keys: @@ -242,7 +242,7 @@ On the `Settings` page, click on `Change external backup` -##### Configure GCP HMAC Key and Secret {#gcp-configure-gcp-hmac-key-and-secret} +##### Configure GCP hmac key and secret {#gcp-configure-gcp-hmac-key-and-secret} In the popup dialogue, provide the GCP bucket path, HMAC key and Secret created in the previous section. @@ -258,7 +258,7 @@ External Backups will now happen in your bucket on the default schedule. Alternatively, you can configure the backup schedule from the `Settings` page. If configured differently, the custom schedule is used to write backups to your bucket and the default schedule (backups every 24 hours) is used for backups in -ClickHouse cloud owned bucket. +ClickHouse Cloud owned bucket. ##### View backups stored in your bucket {#gcp-view-backups-stored-in-your-bucket} @@ -358,7 +358,7 @@ Click on `Save External Bucket` to save the settings External Backups will now happen in your bucket on the default schedule. Alternatively, you can configure the backup schedule from the “Settings” page. If configured differently, the custom schedule is used to write backups to your bucket and the default schedule -(backups every 24 hours) is used for backups in ClickHouse cloud owned bucket. +(backups every 24 hours) is used for backups in ClickHouse Cloud owned bucket. ##### View backups stored in your bucket {#azure-view-backups-stored-in-your-bucket} diff --git a/docs/cloud/guides/backups/03_bring_your_own_backup/03_backup_restore_using_commands.md b/docs/cloud/guides/backups/03_bring_your_own_backup/03_backup_restore_using_commands.md index 2a2b896c738..3f15cb61b3e 100644 --- a/docs/cloud/guides/backups/03_bring_your_own_backup/03_backup_restore_using_commands.md +++ b/docs/cloud/guides/backups/03_bring_your_own_backup/03_backup_restore_using_commands.md @@ -44,7 +44,7 @@ You will need the following details to export/restore backups to your own CSP st -## Backup / Restore specific DB {#backup_restore_db} +## Backup / restore specific DB {#backup_restore_db} Here we show the backup and restore of a *single* database. See the [backup command summary](/operations/backup/overview#command-summary) for full backup and restore commands. @@ -83,7 +83,7 @@ FROM S3( -### Google Cloud Storage (GCS) {#google-cloud-storage} +### Google cloud storage (gcs) {#google-cloud-storage} @@ -116,7 +116,7 @@ FROM S3( -### Azure Blob Storage {#azure-blob-storage} +### Azure blob storage {#azure-blob-storage} @@ -148,7 +148,7 @@ FROM AzureBlobStorage( -## Backup / Restore entire service {#backup_restore_entire_service} +## Backup / restore entire service {#backup_restore_entire_service} For backing up the entire service, use the commands below. This backup will contain all user data and system data for created entities, settings profiles, role policies, quotas, and functions. diff --git a/docs/cloud/guides/best_practices/index.md b/docs/cloud/guides/best_practices/index.md index 86f6a53df8b..52b3c93a7a3 100644 --- a/docs/cloud/guides/best_practices/index.md +++ b/docs/cloud/guides/best_practices/index.md @@ -9,7 +9,7 @@ doc_type: 'landing-page' import TableOfContents from '@site/docs/best-practices/_snippets/_table_of_contents.md'; -# Best Practices in ClickHouse Cloud {#best-practices-in-clickhouse-cloud} +# Best practices in ClickHouse Cloud {#best-practices-in-clickhouse-cloud} This section provides best practices you will want to follow to get the most out of ClickHouse Cloud. diff --git a/docs/cloud/guides/data_sources/01_cloud-endpoints-api.md b/docs/cloud/guides/data_sources/01_cloud-endpoints-api.md index 4df7b512dfd..3259fbc1783 100644 --- a/docs/cloud/guides/data_sources/01_cloud-endpoints-api.md +++ b/docs/cloud/guides/data_sources/01_cloud-endpoints-api.md @@ -11,7 +11,7 @@ import Image from '@theme/IdealImage'; import aws_rds_mysql from '@site/static/images/_snippets/aws-rds-mysql.png'; import gcp_authorized_network from '@site/static/images/_snippets/gcp-authorized-network.png'; -## Static IPs API {#static-ips-api} +## Static ips API {#static-ips-api} If you need to fetch the list of static IPs, you can use the following ClickHouse Cloud API endpoint: [`https://api.clickhouse.cloud/static-ips.json`](https://api.clickhouse.cloud/static-ips.json). This API provides the endpoints for ClickHouse Cloud services, such as ingress/egress IPs and S3 endpoints per region and cloud. @@ -40,7 +40,7 @@ For example, to allow access from a ClickHouse Cloud service hosted on AWS in th ... ``` -For example, an AWS RDS instance running in `us-east-2` that needs to connect to a ClickHouse cloud service should have the following Inbound security group rules: +For example, an AWS RDS instance running in `us-east-2` that needs to connect to a ClickHouse Cloud service should have the following Inbound security group rules: diff --git a/docs/cloud/guides/data_sources/02_accessing-s3-data-securely.md b/docs/cloud/guides/data_sources/02_accessing-s3-data-securely.md index e90da00c552..519dc42e402 100644 --- a/docs/cloud/guides/data_sources/02_accessing-s3-data-securely.md +++ b/docs/cloud/guides/data_sources/02_accessing-s3-data-securely.md @@ -24,9 +24,9 @@ This approach allows customers to manage all access to their S3 buckets in a sin ## Setup {#setup} -### Obtaining the ClickHouse service IAM role ARN {#obtaining-the-clickhouse-service-iam-role-arn} +### Obtaining the ClickHouse service iam role arn {#obtaining-the-clickhouse-service-iam-role-arn} -1 - Login to your ClickHouse cloud account. +1 - Login to your ClickHouse Cloud account. 2 - Select the ClickHouse service you want to create the integration @@ -38,9 +38,9 @@ This approach allows customers to manage all access to their S3 buckets in a sin -### Setting up IAM assume role {#setting-up-iam-assume-role} +### Setting up iam assume role {#setting-up-iam-assume-role} -#### Option 1: Deploying with CloudFormation stack {#option-1-deploying-with-cloudformation-stack} +#### Option 1: deploying with cloudformation stack {#option-1-deploying-with-cloudformation-stack} 1 - Login to your AWS Account in the web browser with an IAM user that has permission to create & manage IAM role. @@ -72,7 +72,7 @@ This approach allows customers to manage all access to their S3 buckets in a sin -#### Option 2: Manually create IAM role {#option-2-manually-create-iam-role} +#### Option 2: manually create iam role {#option-2-manually-create-iam-role} 1 - Login to your AWS Account in the web browser with an IAM user that has permission to create & manage IAM role. @@ -129,7 +129,7 @@ IAM policy (Please replace `{BUCKET_NAME}` with your bucket name): 4 - Copy the new **IAM Role Arn** after creation. This is what needed to access your S3 bucket. -## Access your S3 bucket with the ClickHouseAccess role {#access-your-s3-bucket-with-the-clickhouseaccess-role} +## Access your S3 bucket with the clickhouseaccess role {#access-your-s3-bucket-with-the-clickhouseaccess-role} ClickHouse Cloud has a new feature that allows you to specify `extra_credentials` as part of the S3 table function. Below is an example of how to run a query using the newly created role copied from above. diff --git a/docs/cloud/guides/infrastructure/01_deployment_options/03_clickhouse-government.md b/docs/cloud/guides/infrastructure/01_deployment_options/03_clickhouse-government.md index 7e21939a23f..2d146200942 100644 --- a/docs/cloud/guides/infrastructure/01_deployment_options/03_clickhouse-government.md +++ b/docs/cloud/guides/infrastructure/01_deployment_options/03_clickhouse-government.md @@ -62,7 +62,7 @@ Customers can initiate onboarding by reaching out to [us](https://clickhouse.com This section is intended to provide an overview of the resources required to deploy ClickHouse Government. Specific deployment guides are provided as part of onboarding. Instance/server types and sizes depend on the use case. -### ClickHouse Government on AWS {#clickhouse-government-aws} +### ClickHouse government on AWS {#clickhouse-government-aws} Required resources: - [ECR](https://docs.aws.amazon.com/ecr/) to receive the images and Helm charts diff --git a/docs/cloud/guides/infrastructure/01_deployment_options/byoc/03_onboarding/01_aws.md b/docs/cloud/guides/infrastructure/01_deployment_options/byoc/03_onboarding/01_aws.md index af160cc13c7..2a6e86b278c 100644 --- a/docs/cloud/guides/infrastructure/01_deployment_options/byoc/03_onboarding/01_aws.md +++ b/docs/cloud/guides/infrastructure/01_deployment_options/byoc/03_onboarding/01_aws.md @@ -30,11 +30,11 @@ With this account and the initial organization admin email, you can contact Clic The initial BYOC setup can be performed using either a CloudFormation template or a Terraform module. Both approaches create the same IAM role, enabling BYOC controllers from ClickHouse Cloud to manage your infrastructure. Note that S3, VPC, and compute resources required for running ClickHouse are not included in this initial setup. -#### CloudFormation Template {#cloudformation-template} +#### Cloudformation template {#cloudformation-template} [BYOC CloudFormation template](https://s3.us-east-2.amazonaws.com/clickhouse-public-resources.clickhouse.cloud/cf-templates/byoc.yaml) -#### Terraform Module {#terraform-module} +#### Terraform module {#terraform-module} [BYOC Terraform module](https://s3.us-east-2.amazonaws.com/clickhouse-public-resources.clickhouse.cloud/tf/byoc.tar.gz) @@ -93,14 +93,14 @@ Create a support ticket with the following information: * The Private Subnet IDs you've allocated for ClickHouse * The availability zones these subnets are in -### Optional: Setup VPC Peering {#optional-setup-vpc-peering} +### Optional: setup VPC peering {#optional-setup-vpc-peering} To create or delete VPC peering for ClickHouse BYOC, follow the steps: -#### Step 1: Enable private load balancer for ClickHouse BYOC {#step-1-enable-private-load-balancer-for-clickhouse-byoc} +#### Step 1: enable private load balancer for ClickHouse BYOC {#step-1-enable-private-load-balancer-for-clickhouse-byoc} Contact ClickHouse Support to enable Private Load Balancer. -#### Step 2 Create a peering connection {#step-2-create-a-peering-connection} +#### Step 2 create a peering connection {#step-2-create-a-peering-connection} 1. Navigate to the VPC Dashboard in ClickHouse BYOC account. 2. Select Peering Connections. 3. Click Create Peering Connection @@ -114,7 +114,7 @@ Contact ClickHouse Support to enable Private Load Balancer.
-#### Step 3 Accept the peering connection request {#step-3-accept-the-peering-connection-request} +#### Step 3 accept the peering connection request {#step-3-accept-the-peering-connection-request} Go to the peering account, in the (VPC -> Peering connections -> Actions -> Accept request) page customer can approve this VPC peering request.
@@ -123,7 +123,7 @@ Go to the peering account, in the (VPC -> Peering connections -> Actions -> Acce
-#### Step 4 Add destination to ClickHouse VPC route tables {#step-4-add-destination-to-clickhouse-vpc-route-tables} +#### Step 4 add destination to ClickHouse VPC route tables {#step-4-add-destination-to-clickhouse-vpc-route-tables} In ClickHouse BYOC account, 1. Select Route Tables in the VPC Dashboard. 2. Search for the ClickHouse VPC ID. Edit each route table attached to the private subnets. @@ -138,7 +138,7 @@ In ClickHouse BYOC account,
-#### Step 5 Add destination to the target VPC route tables {#step-5-add-destination-to-the-target-vpc-route-tables} +#### Step 5 add destination to the target VPC route tables {#step-5-add-destination-to-the-target-vpc-route-tables} In the peering AWS account, 1. Select Route Tables in the VPC Dashboard. 2. Search for the target VPC ID. @@ -153,7 +153,7 @@ In the peering AWS account,
-#### Step 6: Edit security group to allow peered VPC access {#step-6-edit-security-group-to-allow-peered-vpc-access} +#### Step 6: edit security group to allow peered VPC access {#step-6-edit-security-group-to-allow-peered-vpc-access} In the ClickHouse BYOC account, you need to update the Security Group settings to allow traffic from your peered VPC. Please contact ClickHouse Support to request the addition of inbound rules that include the CIDR ranges of your peered VPC. --- @@ -175,9 +175,9 @@ While we aim for seamless upgrades (e.g., rolling upgrades and restarts), some, Maintenance windows do not apply to security and vulnerability fixes. These are handled as off-cycle upgrades, with timely communication to coordinate a suitable time and minimize operational impact. ::: -## CloudFormation IAM roles {#cloudformation-iam-roles} +## Cloudformation iam roles {#cloudformation-iam-roles} -### Bootstrap IAM role {#bootstrap-iam-role} +### Bootstrap iam role {#bootstrap-iam-role} The bootstrap IAM role has the following permissions: @@ -187,7 +187,7 @@ The bootstrap IAM role has the following permissions: - **IAM operations (e.g., `iam:CreatePolicy`)**: Needed for controllers to create additional roles (see the next section for details). - **EKS operations**: Limited to resources with names starting with the `clickhouse-cloud` prefix. -### Additional IAM roles created by the controller {#additional-iam-roles-created-by-the-controller} +### Additional iam roles created by the controller {#additional-iam-roles-created-by-the-controller} In addition to the `ClickHouseManagementRole` created via CloudFormation, the controller will create several additional roles. diff --git a/docs/cloud/guides/infrastructure/01_deployment_options/byoc/05_observability/01_aws.md b/docs/cloud/guides/infrastructure/01_deployment_options/byoc/05_observability/01_aws.md index 5b578f1b68c..207f2113b9e 100644 --- a/docs/cloud/guides/infrastructure/01_deployment_options/byoc/05_observability/01_aws.md +++ b/docs/cloud/guides/infrastructure/01_deployment_options/byoc/05_observability/01_aws.md @@ -47,7 +47,7 @@ Contact ClickHouse support to enable the Private Load balancer and ask for the U https://prometheus-internal...aws.clickhouse-byoc.com/query ``` -#### Prometheus Integration {#prometheus-integration} +#### Prometheus integration {#prometheus-integration} diff --git a/docs/cloud/guides/production-readiness.md b/docs/cloud/guides/production-readiness.md index d811f9e9e6f..2dc85332749 100644 --- a/docs/cloud/guides/production-readiness.md +++ b/docs/cloud/guides/production-readiness.md @@ -7,7 +7,7 @@ keywords: ['production readiness', 'enterprise', 'saml', 'sso', 'terraform', 'mo doc_type: 'guide' --- -# ClickHouse Cloud Production Readiness Guide {#production-readiness} +# ClickHouse Cloud production readiness guide {#production-readiness} For organizations who have completed the quick start guide and have an active service with data flowing @@ -97,7 +97,7 @@ Learn more about [Cloud Access Management](/cloud/security/cloud_access_manageme Managing ClickHouse Cloud through infrastructure-as-code practices and API automation provides consistency, version control, and repeatability for your deployment configuration. -### Terraform Provider {#terraform-provider} +### Terraform provider {#terraform-provider} Configure the ClickHouse Terraform provider with API keys created in the ClickHouse Cloud console: diff --git a/docs/cloud/guides/security/01_cloud_access_management/01_manage-my-account.md b/docs/cloud/guides/security/01_cloud_access_management/01_manage-my-account.md index f1995a9488b..9a4f8bb93fd 100644 --- a/docs/cloud/guides/security/01_cloud_access_management/01_manage-my-account.md +++ b/docs/cloud/guides/security/01_cloud_access_management/01_manage-my-account.md @@ -37,7 +37,7 @@ If your company uses Google Workspace or Microsoft 365, you can leverage your cu Users using SAML SSO are automatically added by their identity provider upon sign in. ClickHouse Cloud users with the Organization Admin role may [manage roles](/cloud/security/manage-cloud-users) assigned to SAML users and enforce SAML as the only authentication method. -## Manage multi-factor authentication (MFA) {#mfa} +## Manage multi-factor authentication (mfa) {#mfa} Users with email + password or social authentication can further secure their account using multi-factor authentication (MFA). To set up MFA: @@ -82,7 +82,7 @@ If you forgot your password, follow these steps for self-service recovery: 7. Click Back to sign in 8. Sign in normally with your new password -### MFA self-service recovery {#mfa-self-serivce-recovery} +### Mfa self-service recovery {#mfa-self-serivce-recovery} If you lost your MFA device or deleted your token, follow these steps to recover and create a new token: 1. Go to https://console.clickhouse.cloud @@ -103,7 +103,7 @@ If you lost your MFA device or deleted your token, follow these steps to recover 16. Click the box next to `I have safely recorded this code` and click Continue 17. Follow the process above to set up a new MFA factor -### Lost MFA and recovery code {#lost-mfa-and-recovery-code} +### Lost mfa and recovery code {#lost-mfa-and-recovery-code} If you lost your MFA device AND recovery code or you lost your MFA device and never obtained a recovery code, follow these steps to request a reset: diff --git a/docs/cloud/guides/security/01_cloud_access_management/02_manage-cloud-users.md b/docs/cloud/guides/security/01_cloud_access_management/02_manage-cloud-users.md index a07534c8eba..ac352b5c0f6 100644 --- a/docs/cloud/guides/security/01_cloud_access_management/02_manage-cloud-users.md +++ b/docs/cloud/guides/security/01_cloud_access_management/02_manage-cloud-users.md @@ -45,7 +45,7 @@ If your organization is configured for [SAML SSO](/cloud/security/saml-setup) fo 2. When the user logs in to ClickHouse Cloud they will automatically be assigned the `Member` role which may only log in and has no other access 3. Follow the instructions in the `Manage user role assignments` below to grant permissions -### Enforcing SAML-only authentication {#enforce-saml} +### Enforcing saml-only authentication {#enforce-saml} Once you have at least one SAML user in the organization assigned to the Organization Admin role, remove users with other authentication methods from the organization to enforce SAML only authentication for the organization. diff --git a/docs/cloud/guides/security/01_cloud_access_management/03_manage-sql-console-role-assignments.md b/docs/cloud/guides/security/01_cloud_access_management/03_manage-sql-console-role-assignments.md index 1b3d52494f6..b1ed2494901 100644 --- a/docs/cloud/guides/security/01_cloud_access_management/03_manage-sql-console-role-assignments.md +++ b/docs/cloud/guides/security/01_cloud_access_management/03_manage-sql-console-role-assignments.md @@ -40,7 +40,7 @@ Under the "Security" section, find the "SQL console access" area: -### Update the settings for Service Admin {#update-settings-for-service-admin} +### Update the settings for service admin {#update-settings-for-service-admin} Select the drop-down menu for Service Admin to change the access control settings for Service Admin roles: @@ -54,7 +54,7 @@ You can choose from the following roles: | `Read only` | | `Full access` | -### Update the settings for Service Read Only {#update-settings-for-service-read-only} +### Update the settings for service read only {#update-settings-for-service-read-only} Select the drop-down menu for Service Read Only to change the access control settings for Service Read Only roles: diff --git a/docs/cloud/guides/security/01_cloud_access_management/04_manage-database-users.md b/docs/cloud/guides/security/01_cloud_access_management/04_manage-database-users.md index e7be10aad54..0e86f48d6d4 100644 --- a/docs/cloud/guides/security/01_cloud_access_management/04_manage-database-users.md +++ b/docs/cloud/guides/security/01_cloud_access_management/04_manage-database-users.md @@ -52,7 +52,7 @@ The user will be assigned the role associated with their email address whenever ## Database authentication {#database-authentication} -### Database user ID and password {#database-user-id--password} +### Database user id and password {#database-user-id--password} Use the SHA256_hash method when [creating user accounts](/sql-reference/statements/create/user.md) to secure passwords. ClickHouse database passwords must contain a minimum of 12 characters and meet complexity requirements: upper case characters, lower case characters, numbers and/or special characters. @@ -65,7 +65,7 @@ such as [this one](https://tools.keycdn.com/sha256-online-generator) before prov CREATE USER userName IDENTIFIED WITH sha256_hash BY 'hash'; ``` -### Database user with secure shell (SSH) authentication {#database-ssh} +### Database user with secure shell (ssh) authentication {#database-ssh} To set up SSH authentication for a ClickHouse Cloud database user. @@ -139,7 +139,7 @@ FROM system.role_grants LEFT OUTER JOIN system.grants ON role_grants.granted_rol WHERE role_grants.user_name is null; ``` -#### Associate grant list to Console users with access to SQL console {#associate-grant-list-to-console-users-with-access-to-sql-console} +#### Associate grant list to console users with access to SQL console {#associate-grant-list-to-console-users-with-access-to-sql-console} Associate this list with Console users that have access to SQL console. diff --git a/docs/cloud/guides/security/01_cloud_access_management/04_saml-sso-setup.md b/docs/cloud/guides/security/01_cloud_access_management/04_saml-sso-setup.md index 0d111ebd490..ebdef6e7a79 100644 --- a/docs/cloud/guides/security/01_cloud_access_management/04_saml-sso-setup.md +++ b/docs/cloud/guides/security/01_cloud_access_management/04_saml-sso-setup.md @@ -120,7 +120,7 @@ We recommend setting up a **direct link to your organization** in addition to yo -### Configure Okta SAML {#configure-okta-saml} +### Configure okta SAML {#configure-okta-saml} You will configure two App Integrations in Okta for each ClickHouse organization: one SAML app and one bookmark to house your direct link. @@ -307,7 +307,7 @@ Azure (Microsoft) SAML may also be referred to as Azure Active Directory (AD) or -### Configure Duo SAML {#configure-duo-saml} +### Configure duo SAML {#configure-duo-saml}
Create a Generic SAML Service Provider for Duo @@ -356,7 +356,7 @@ Security is our top priority when it comes to authentication. For this reason, w - **We do not automatically link SSO and non-SSO accounts.** You may see multiple accounts for your users in your ClickHouse user list even if they are using the same email address. -## Troubleshooting Common Issues {#troubleshooting-common-issues} +## Troubleshooting common issues {#troubleshooting-common-issues} | Error | Cause | Solution | |:------|:------|:---------| diff --git a/docs/cloud/guides/security/02_connectivity/01_setting-ip-filters.md b/docs/cloud/guides/security/02_connectivity/01_setting-ip-filters.md index febb905b9fc..77311908608 100644 --- a/docs/cloud/guides/security/02_connectivity/01_setting-ip-filters.md +++ b/docs/cloud/guides/security/02_connectivity/01_setting-ip-filters.md @@ -11,7 +11,7 @@ import Image from '@theme/IdealImage'; import ip_filtering_after_provisioning from '@site/static/images/cloud/security/ip-filtering-after-provisioning.png'; import ip_filter_add_single_ip from '@site/static/images/cloud/security/ip-filter-add-single-ip.png'; -## Setting IP filters {#setting-ip-filters} +## Setting ip filters {#setting-ip-filters} IP access lists filter traffic to ClickHouse services or API keys by specifying which source addresses are permitted to connect. These lists are configurable for each service and each API key. Lists can be configured during service or API key creation, or afterward. @@ -25,7 +25,7 @@ Before you begin, collect the IP addresses or ranges that should be added to the Classless Inter-domain Routing (CIDR) notation, allows you to specify IP address ranges smaller than the traditional Class A, B, or C (8, 6, or 24) subnet mask sizes. [ARIN](https://account.arin.net/public/cidrCalculator) and several other organizations provide CIDR calculators if you need one, and if you would like more information on CIDR notation, please see the [Classless Inter-domain Routing (CIDR)](https://www.rfc-editor.org/rfc/rfc4632.html) RFC. -## Create or modify an IP access list {#create-or-modify-an-ip-access-list} +## Create or modify an ip access list {#create-or-modify-an-ip-access-list} :::note Applicable only to connections outside of PrivateLink IP access lists only apply to connections from the public internet, outside of [PrivateLink](/cloud/security/connectivity/private-networking). diff --git a/docs/cloud/guides/security/02_connectivity/private_networking/02_aws-privatelink.md b/docs/cloud/guides/security/02_connectivity/private_networking/02_aws-privatelink.md index 24edf7df786..5efe966b44e 100644 --- a/docs/cloud/guides/security/02_connectivity/private_networking/02_aws-privatelink.md +++ b/docs/cloud/guides/security/02_connectivity/private_networking/02_aws-privatelink.md @@ -18,7 +18,7 @@ import pe_remove_private_endpoint from '@site/static/images/cloud/security/pe-re import aws_private_link_pe_filters from '@site/static/images/cloud/security/aws-privatelink-pe-filters.png'; import aws_private_link_ped_nsname from '@site/static/images/cloud/security/aws-privatelink-pe-dns-name.png'; -# AWS PrivateLink +# AWS privatelink @@ -83,7 +83,7 @@ Before you get started you will need: Follow these steps to connect your ClickHouse Cloud services via AWS PrivateLink. -### Obtain endpoint "Service name" {#obtain-endpoint-service-info} +### Obtain endpoint "service name" {#obtain-endpoint-service-info} #### Option 1: ClickHouse Cloud console {#option-1-clickhouse-cloud-console} @@ -169,7 +169,7 @@ After creating the VPC Endpoint, make a note of the `Endpoint ID` value; you'll -#### Option 2: AWS CloudFormation {#option-2-aws-cloudformation} +#### Option 2: AWS cloudformation {#option-2-aws-cloudformation} Next, you need to create a VPC Endpoint using `Service name`console or `endpointServiceId`API you got from [Obtain Endpoint "Service name" ](#obtain-endpoint-service-info) step. Make sure to use correct subnet IDs, security groups, and VPC ID. @@ -223,7 +223,7 @@ There are various ways to configure DNS. Please set up DNS according to your spe You need to point "DNS name", taken from [Obtain Endpoint "Service name" ](#obtain-endpoint-service-info) step, to AWS Endpoint network interfaces. This ensures that services/components within your VPC/Network can resolve it properly. -### Add "Endpoint ID" to ClickHouse service allow list {#add-endpoint-id-to-services-allow-list} +### Add "endpoint id" to ClickHouse service allow list {#add-endpoint-id-to-services-allow-list} #### Option 1: ClickHouse Cloud console {#option-1-clickhouse-cloud-console-2} @@ -292,7 +292,7 @@ curl --silent --user "${KEY_ID:?}:${KEY_SECRET:?}" \ -d @pl_config.json | jq ``` -### Accessing an instance using PrivateLink {#accessing-an-instance-using-privatelink} +### Accessing an instance using privatelink {#accessing-an-instance-using-privatelink} Each service with Private Link enabled has a public and private endpoint. In order to connect using Private Link, you need to use a private endpoint which will be `privateDnsHostname`API or `DNS Name`console taken from [Obtain Endpoint "Service name"](#obtain-endpoint-service-info). @@ -336,7 +336,7 @@ In this example connection via value of `privateDnsHostname` host name will be r ## Troubleshooting {#troubleshooting} -### Multiple PrivateLinks in one region {#multiple-privatelinks-in-one-region} +### Multiple privatelinks in one region {#multiple-privatelinks-in-one-region} In most cases, you only need to create a single endpoint service for each VPC. This endpoint can route requests from the VPC to multiple ClickHouse Cloud services. Please refer [here](#considerations) @@ -347,7 +347,7 @@ Please refer [here](#considerations) - Please verify `inbound` rules on security group attached to Endpoint and allow ClickHouse ports. - Please verify `outbound` rules on security group attached to VM which is used to connectivity test and allow connections to ClickHouse ports. -### Private Hostname: Not found address of host {#private-hostname-not-found-address-of-host} +### Private hostname: not found address of host {#private-hostname-not-found-address-of-host} - Please check your DNS configuration diff --git a/docs/cloud/guides/security/02_connectivity/private_networking/03_gcp-private-service-connect.md b/docs/cloud/guides/security/02_connectivity/private_networking/03_gcp-private-service-connect.md index 80fe1c73e4a..250f8245600 100644 --- a/docs/cloud/guides/security/02_connectivity/private_networking/03_gcp-private-service-connect.md +++ b/docs/cloud/guides/security/02_connectivity/private_networking/03_gcp-private-service-connect.md @@ -21,7 +21,7 @@ import gcp_pe_remove_private_endpoint from '@site/static/images/cloud/security/g import gcp_privatelink_pe_filters from '@site/static/images/cloud/security/gcp-privatelink-pe-filters.png'; import gcp_privatelink_pe_dns from '@site/static/images/cloud/security/gcp-privatelink-pe-dns.png'; -# Private Service Connect {#private-service-connect} +# Private service connect {#private-service-connect} @@ -91,7 +91,7 @@ jq ".result[] | select (.region==\"${REGION:?}\" and .provider==\"${PROVIDER:?}\ - You can [create a new key](/cloud/manage/openapi) or use an existing one. ::: -## Obtain GCP service attachment and DNS name for Private Service Connect {#obtain-gcp-service-attachment-and-dns-name-for-private-service-connect} +## Obtain GCP service attachment and DNS name for private service connect {#obtain-gcp-service-attachment-and-dns-name-for-private-service-connect} ### Option 1: ClickHouse Cloud console {#option-1-clickhouse-cloud-console} @@ -133,7 +133,7 @@ In this section, we're going to create a service endpoint. First up, we're going to create a Private Service Connection. -#### Option 1: Using Google Cloud console {#option-1-using-google-cloud-console} +#### Option 1: using Google cloud console {#option-1-using-google-cloud-console} In the Google Cloud console, navigate to **Network services -> Private Service Connect**. @@ -157,7 +157,7 @@ The **Status** column will change from **Pending** to **Accepted** once the conn Copy ***PSC Connection ID***, we are going to use it as ***Endpoint ID*** in the next steps. -#### Option 2: Using Terraform {#option-2-using-terraform} +#### Option 2: using Terraform {#option-2-using-terraform} ```json provider "google" { @@ -217,7 +217,7 @@ There are various ways to configure DNS. Please set up DNS according to your spe You need to point "DNS name", taken from [Obtain GCP service attachment for Private Service Connect](#obtain-gcp-service-attachment-and-dns-name-for-private-service-connect) step, to GCP Private Service Connect endpoint IP address. This ensures that services/components within your VPC/Network can resolve it properly. -## Add Endpoint ID to ClickHouse Cloud organization {#add-endpoint-id-to-clickhouse-cloud-organization} +## Add endpoint id to ClickHouse Cloud organization {#add-endpoint-id-to-clickhouse-cloud-organization} ### Option 1: ClickHouse Cloud console {#option-1-clickhouse-cloud-console-1} @@ -276,7 +276,7 @@ Add/remove Private Endpoint to an organization: curl --silent --user "${KEY_ID:?}:${KEY_SECRET:?}" -X PATCH -H "Content-Type: application/json" "https://api.clickhouse.cloud/v1/organizations/${ORG_ID:?}" -d @pl_config_org.json ``` -## Add "Endpoint ID" to ClickHouse service allow list {#add-endpoint-id-to-services-allow-list} +## Add "endpoint id" to ClickHouse service allow list {#add-endpoint-id-to-services-allow-list} You need to add an Endpoint ID to the allow-list for each instance that should be available using Private Service Connect. @@ -330,7 +330,7 @@ EOF curl --silent --user "${KEY_ID:?}:${KEY_SECRET:?}" -X PATCH -H "Content-Type: application/json" "https://api.clickhouse.cloud/v1/organizations/${ORG_ID:?}/services/${INSTANCE_ID:?}" -d @pl_config.json | jq ``` -## Accessing instance using Private Service Connect {#accessing-instance-using-private-service-connect} +## Accessing instance using private service connect {#accessing-instance-using-private-service-connect} Each service with Private Link enabled has a public and private endpoint. In order to connect using Private Link, you need to use a private endpoint which will be `privateDnsHostname` taken from [Obtain GCP service attachment for Private Service Connect](#obtain-gcp-service-attachment-and-dns-name-for-private-service-connect). diff --git a/docs/cloud/guides/security/02_connectivity/private_networking/04_azure-privatelink.md b/docs/cloud/guides/security/02_connectivity/private_networking/04_azure-privatelink.md index 8ac50cb28f6..fbfc9fea35f 100644 --- a/docs/cloud/guides/security/02_connectivity/private_networking/04_azure-privatelink.md +++ b/docs/cloud/guides/security/02_connectivity/private_networking/04_azure-privatelink.md @@ -27,7 +27,7 @@ import azure_pe_remove_private_endpoint from '@site/static/images/cloud/security import azure_privatelink_pe_filter from '@site/static/images/cloud/security/azure-privatelink-pe-filter.png'; import azure_privatelink_pe_dns from '@site/static/images/cloud/security/azure-privatelink-pe-dns.png'; -# Azure Private Link +# Azure private link @@ -59,7 +59,7 @@ If you already have Private Link configured for other services in your ClickHous Find Terraform examples at the ClickHouse [Terraform Provider repository](https://github.com/ClickHouse/terraform-provider-clickhouse/tree/main/examples/). -## Obtain Azure connection alias for Private Link {#obtain-azure-connection-alias-for-private-link} +## Obtain Azure connection alias for private link {#obtain-azure-connection-alias-for-private-link} ### Option 1: ClickHouse Cloud console {#option-1-clickhouse-cloud-console} @@ -116,7 +116,7 @@ For any issues related to Azure configuration tasks, contact Azure Support direc In this section, we're going to create a Private Endpoint in Azure. You can use either the Azure Portal or Terraform. -### Option 1: Using Azure Portal to create a private endpoint in Azure {#option-1-using-azure-portal-to-create-a-private-endpoint-in-azure} +### Option 1: using Azure portal to create a private endpoint in Azure {#option-1-using-azure-portal-to-create-a-private-endpoint-in-azure} In the Azure Portal, open **Private Link Center → Private Endpoints**. @@ -185,7 +185,7 @@ Open the network interface associated with Private Endpoint and copy the **Priva -### Option 2: Using Terraform to create a private endpoint in Azure {#option-2-using-terraform-to-create-a-private-endpoint-in-azure} +### Option 2: using Terraform to create a private endpoint in Azure {#option-2-using-terraform-to-create-a-private-endpoint-in-azure} Use the template below to use Terraform to create a Private Endpoint: @@ -204,7 +204,7 @@ resource "azurerm_private_endpoint" "example_clickhouse_cloud" { } ``` -### Obtaining the Private Endpoint Resource ID {#obtaining-private-endpoint-resourceid} +### Obtaining the private endpoint resource id {#obtaining-private-endpoint-resourceid} In order to use Private Link, you need to add the Private Endpoint connection Resource ID to your service allow list. @@ -222,11 +222,11 @@ You can still use the resourceGUID for backward compatibility. Find the `resourc -## Setting up DNS for Private Link {#setting-up-dns-for-private-link} +## Setting up DNS for private link {#setting-up-dns-for-private-link} You will need to create a Private DNS zone (`${location_code}.privatelink.azure.clickhouse.cloud`) and attach it to your VNet to access resources via Private Link. -### Create Private DNS zone {#create-private-dns-zone} +### Create private DNS zone {#create-private-dns-zone} **Option 1: Using Azure portal** @@ -301,7 +301,7 @@ Name: xxxxxxxxxx.westus3.privatelink.azure.clickhouse.cloud Address: 10.0.0.4 ``` -## Add the Private Endpoint Resource ID to your ClickHouse Cloud organization {#add-the-private-endpoint-id-to-your-clickhouse-cloud-organization} +## Add the private endpoint resource id to your ClickHouse Cloud organization {#add-the-private-endpoint-id-to-your-clickhouse-cloud-organization} ### Option 1: ClickHouse Cloud console {#option-1-clickhouse-cloud-console-1} @@ -369,7 +369,7 @@ After adding or removing a Private Endpoint, run the following command to apply curl --silent --user "${KEY_ID:?}:${KEY_SECRET:?}" -X PATCH -H "Content-Type: application/json" "https://api.clickhouse.cloud/v1/organizations/${ORG_ID:?}" -d @pl_config_org.json ``` -## Add the Private Endpoint Resource ID to your service(s) allow list {#add-private-endpoint-id-to-services-allow-list} +## Add the private endpoint resource id to your service(s) allow list {#add-private-endpoint-id-to-services-allow-list} By default, a ClickHouse Cloud service is not available over a Private Link connection even if the Private Link connection is approved and established. You need to explicitly add the Private Endpoint Resource ID for each service that should be available using Private Link. @@ -432,7 +432,7 @@ After adding or removing a Private Endpoint to the services allow list, run the curl --silent --user "${KEY_ID:?}:${KEY_SECRET:?}" -X PATCH -H "Content-Type: application/json" "https://api.clickhouse.cloud/v1/organizations/${ORG_ID:?}/services/${INSTANCE_ID:?}" -d @pl_config.json | jq ``` -## Access your ClickHouse Cloud service using Private Link {#access-your-clickhouse-cloud-service-using-private-link} +## Access your ClickHouse Cloud service using private link {#access-your-clickhouse-cloud-service-using-private-link} Each service with Private Link enabled has a public and private endpoint. In order to connect using Private Link, you need to use a private endpoint which will be `privateDnsHostname`API or `DNS name`console taken from [Obtain Azure connection alias for Private Link](#obtain-azure-connection-alias-for-private-link). @@ -497,7 +497,7 @@ Address: 10.0.0.4 Most likely, the Private Endpoint Resource ID was not added to the service allow-list. Revisit the [_Add Private Endpoint Resource ID to your services allow-list_ step](#add-private-endpoint-id-to-services-allow-list). -### Private Endpoint is in pending state {#private-endpoint-is-in-pending-state} +### Private endpoint is in pending state {#private-endpoint-is-in-pending-state} Most likely, the Private Endpoint Resource ID was not added to the service allow-list. Revisit the [_Add Private Endpoint Resource ID to your services allow-list_ step](#add-private-endpoint-id-to-services-allow-list). diff --git a/docs/cloud/guides/security/04_cmek.md b/docs/cloud/guides/security/04_cmek.md index 50002d7b164..063481071ff 100644 --- a/docs/cloud/guides/security/04_cmek.md +++ b/docs/cloud/guides/security/04_cmek.md @@ -28,7 +28,7 @@ Data at rest is encrypted by default using cloud provider-managed AES 256 keys. Enhanced encryption is currently available in AWS and GCP services. Azure is coming soon. -### Transparent Data Encryption (TDE) {#transparent-data-encryption-tde} +### Transparent data encryption (TDE) {#transparent-data-encryption-tde} TDE must be enabled on service creation. Existing services cannot be encrypted after creation. Once TDE is enabled, it cannot be disabled. All data in the service will remain encrypted. If you want to disable TDE after it has been enabled, you must create a new service and migrate your data there. @@ -38,7 +38,7 @@ TDE must be enabled on service creation. Existing services cannot be encrypted a 4. Click the drop-down for Enterprise features and toggle Enable Transparent Data Encryption (TDE) 5. Click Create service -### Customer Managed Encryption Keys (CMEK) {#customer-managed-encryption-keys-cmek} +### Customer managed encryption keys (CMEK) {#customer-managed-encryption-keys-cmek} :::warning Deleting a KMS key used to encrypt a ClickHouse Cloud service will cause your ClickHouse service to be stopped and its data will be unretrievable, along with existing backups. To prevent accidental data loss when rotating keys you may wish to maintain old KMS keys for a period of time prior to deletion. diff --git a/docs/cloud/guides/security/05_audit_logging/03_byoc-security-playbook.md b/docs/cloud/guides/security/05_audit_logging/03_byoc-security-playbook.md index 534d80eb700..ebff58e6129 100644 --- a/docs/cloud/guides/security/05_audit_logging/03_byoc-security-playbook.md +++ b/docs/cloud/guides/security/05_audit_logging/03_byoc-security-playbook.md @@ -30,7 +30,7 @@ ClickHouse utilizes pre-created roles to enable system functions. This section a If an incident may be the result of a compromised role, review activities in CloudTrail and CloudWatch related to the ClickHouse IAM roles and actions. Refer to the [CloudFormation](/cloud/reference/byoc/onboarding/aws#cloudformation-iam-roles) stack or Terraform module provided as part of setup for a list of IAM roles. -## Unauthorized access to EKS cluster {#unauthorized-access-eks-cluster} +## Unauthorized access to eks cluster {#unauthorized-access-eks-cluster} ClickHouse BYOC runs inside EKS. This section assumes the customer is using CloudTrail and CloudWatch in AWS and has access to the logs. diff --git a/docs/cloud/guides/security/06_compliance/hipaa-onboarding.md b/docs/cloud/guides/security/06_compliance/hipaa-onboarding.md index e2cc563c12a..6f14621de4f 100644 --- a/docs/cloud/guides/security/06_compliance/hipaa-onboarding.md +++ b/docs/cloud/guides/security/06_compliance/hipaa-onboarding.md @@ -28,7 +28,7 @@ This page describes the process for enabling deployment of HIPAA compliant servi -### Sign up for Enterprise services {#sign-up-for-enterprise} +### Sign up for enterprise services {#sign-up-for-enterprise} 1. Select your organization name in the lower left corner of the console. 2. Click **Billing**. diff --git a/docs/cloud/guides/security/06_compliance/pci-onboarding.md b/docs/cloud/guides/security/06_compliance/pci-onboarding.md index e130471f096..a1c2ade0f05 100644 --- a/docs/cloud/guides/security/06_compliance/pci-onboarding.md +++ b/docs/cloud/guides/security/06_compliance/pci-onboarding.md @@ -25,7 +25,7 @@ This page describes the process for enabling deployment of PCI compliant service -### Sign up for Enterprise services {#sign-up-for-enterprise} +### Sign up for enterprise services {#sign-up-for-enterprise} 1. Select your organization name in the lower left corner of the console. 2. Click **Billing**. diff --git a/docs/cloud/onboard/01_discover/01_what_is.md b/docs/cloud/onboard/01_discover/01_what_is.md index 278b77891f1..8f318391991 100644 --- a/docs/cloud/onboard/01_discover/01_what_is.md +++ b/docs/cloud/onboard/01_discover/01_what_is.md @@ -28,7 +28,7 @@ ClickHouse Cloud offers several major benefits over the open-source version: - **Total cost of ownership**: Best price / performance ratio and low administrative overhead. - **Broad ecosystem**: Bring your favorite data connectors, visualization tools, SQL and language clients with you. -## Comparison of self-managed ClickHouse and ClickHouse Cloud {#oss-vs-clickhouse-cloud} +## Comparison of Self-managed ClickHouse and ClickHouse Cloud {#oss-vs-clickhouse-cloud} | Feature | Benefits | OSS ClickHouse | ClickHouse Cloud | |--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|-------------------| diff --git a/docs/cloud/onboard/01_discover/02_use_cases/01_real-time-analytics.md b/docs/cloud/onboard/01_discover/02_use_cases/01_real-time-analytics.md index abff368feac..4200d4ff37c 100644 --- a/docs/cloud/onboard/01_discover/02_use_cases/01_real-time-analytics.md +++ b/docs/cloud/onboard/01_discover/02_use_cases/01_real-time-analytics.md @@ -47,7 +47,7 @@ Organizations can discover insights about their customers by aggregating and analyzing events like this. This has traditionally been done using batch analytics, and in the next section, we’ll compare batch and real-time analytics. -## Real-Time analytics vs batch analytics {#real-time-analytics-vs-batch-analytics} +## Real-time analytics vs batch analytics {#real-time-analytics-vs-batch-analytics} The diagram below shows what a typical batch analytics system would look like from the perspective of an individual event: diff --git a/docs/cloud/onboard/01_discover/02_use_cases/02_observability.md b/docs/cloud/onboard/01_discover/02_use_cases/02_observability.md index 93d37bbdb8b..b6d3509d183 100644 --- a/docs/cloud/onboard/01_discover/02_use_cases/02_observability.md +++ b/docs/cloud/onboard/01_discover/02_use_cases/02_observability.md @@ -19,7 +19,7 @@ into a comprehensive approach to understanding system behavior. However, implementing effective observability isn't straightforward - it requires understanding technical concepts and organizational challenges. -## What is Observability? {#what-is-observability} +## What is observability? {#what-is-observability} Observability is understanding a system's internal state by examining its outputs. In software systems, this means understanding what's happening inside your @@ -101,7 +101,7 @@ helps maintain responsive services that keep customers engaged through comprehensive monitoring and quick issue resolution often gain an edge over competitors -### Cost Savings from faster incident response {#cost-savings-from-faster-incident-response} +### Cost savings from faster incident response {#cost-savings-from-faster-incident-response} One of the most immediate benefits of observability is reduced labor costs through faster detection and resolution of issues. This comes from: diff --git a/docs/cloud/onboard/01_discover/02_use_cases/03_data_warehousing.md b/docs/cloud/onboard/01_discover/02_use_cases/03_data_warehousing.md index b2460585065..10bc6afcc71 100644 --- a/docs/cloud/onboard/01_discover/02_use_cases/03_data_warehousing.md +++ b/docs/cloud/onboard/01_discover/02_use_cases/03_data_warehousing.md @@ -100,7 +100,7 @@ This combination delivers the performance characteristics of a specialized analytical database without sacrificing the core benefits of the lakehouse model, including component interchangeability, open formats, and unified data management. -## Hybrid architecture: The best of both worlds {#hybrid-architecture-the-best-of-both-worlds} +## Hybrid architecture: the best of both worlds {#hybrid-architecture-the-best-of-both-worlds} While ClickHouse excels at querying lakehouse components, its highly optimized storage engine offers an additional advantage. For use cases demanding ultra-low diff --git a/docs/cloud/onboard/01_discover/02_use_cases/04_machine_learning_and_genAI/02_agent_facing_analytics.md b/docs/cloud/onboard/01_discover/02_use_cases/04_machine_learning_and_genAI/02_agent_facing_analytics.md index 40e825a3c48..aeb9b7b73a0 100644 --- a/docs/cloud/onboard/01_discover/02_use_cases/04_machine_learning_and_genAI/02_agent_facing_analytics.md +++ b/docs/cloud/onboard/01_discover/02_use_cases/04_machine_learning_and_genAI/02_agent_facing_analytics.md @@ -34,7 +34,7 @@ and communicate. It's as if they've suddenly become more fluent in "human" aka. able to grasp requests and respond with relevant contextual information drawn from the model's training. -### AI agents superpowers: “Tools” {#tools} +### Ai agents superpowers: “tools” {#tools} These agents really shine through their access to “tools”. Tools enhance AI agents by giving them abilities to perform tasks. Rather than just being conversational @@ -84,7 +84,7 @@ to thirty minutes, a task that would traditionally take a human several hours. -## Real-time analytics for AI agents {#real-time-analytics-for-ai-agents} +## Real-time analytics for ai agents {#real-time-analytics-for-ai-agents} Let's take the case of an agentic AI assistant with access to a real-time analytics database containing the company's CRM data. When a user asks @@ -136,7 +136,7 @@ responses while offloading the heavy computational work. It positions the real-time analytics database as a preferred "context provider" for AI agents when it comes to insights. -## AI agents as an emerging user persona {#ai-agents-as-an-emerging-user-persona} +## Ai agents as an emerging user persona {#ai-agents-as-an-emerging-user-persona} A useful way to think about AI agents leveraging real-time analytics databases is to perceive them as a new category of users, or in product manager speak: diff --git a/docs/cloud/onboard/02_migrate/01_migration_guides/02_postgres/appendix.md b/docs/cloud/onboard/02_migrate/01_migration_guides/02_postgres/appendix.md index 785aac2172d..8645cebeab6 100644 --- a/docs/cloud/onboard/02_migrate/01_migration_guides/02_postgres/appendix.md +++ b/docs/cloud/onboard/02_migrate/01_migration_guides/02_postgres/appendix.md @@ -9,7 +9,7 @@ doc_type: 'reference' import postgresReplicas from '@site/static/images/integrations/data-ingestion/dbms/postgres-replicas.png'; import Image from '@theme/IdealImage'; -## Postgres vs ClickHouse: Equivalent and different concepts {#postgres-vs-clickhouse-equivalent-and-different-concepts} +## Postgres vs ClickHouse: equivalent and different concepts {#postgres-vs-clickhouse-equivalent-and-different-concepts} Users coming from OLTP systems who are used to ACID transactions should be aware that ClickHouse makes deliberate compromises in not fully providing these in exchange for performance. ClickHouse semantics can deliver high durability guarantees and high write throughput if well understood. We highlight some key concepts below that users should be familiar with prior to working with ClickHouse from Postgres. @@ -101,7 +101,7 @@ See [here](/cloud/reference/shared-merge-tree#consistency) for further details o > Use of sequential consistency will place a greater load on ClickHouse Keeper. The result can mean slower inserts and reads. SharedMergeTree, used in ClickHouse Cloud as the main table engine, sequential consistency [incurs less overhead and will scale better](/cloud/reference/shared-merge-tree#consistency). OSS users should use this approach cautiously and measure Keeper load. -## Transactional (ACID) support {#transactional-acid-support} +## Transactional (acid) support {#transactional-acid-support} Users migrating from PostgreSQL may be used to its robust support for ACID (Atomicity, Consistency, Isolation, Durability) properties, making it a reliable choice for transactional databases. Atomicity in PostgreSQL ensures that each transaction is treated as a single unit, which either completely succeeds or is entirely rolled back, preventing partial updates. Consistency is maintained by enforcing constraints, triggers, and rules that guarantee that all database transactions lead to a valid state. Isolation levels, from Read Committed to Serializable, are supported in PostgreSQL, allowing fine-tuned control over the visibility of changes made by concurrent transactions. Lastly, Durability is achieved through write-ahead logging (WAL), ensuring that once a transaction is committed, it remains so even in the event of a system failure. diff --git a/docs/cloud/onboard/02_migrate/01_migration_guides/02_postgres/migration_guide/03_migration_guide_part3.md b/docs/cloud/onboard/02_migrate/01_migration_guides/02_postgres/migration_guide/03_migration_guide_part3.md index be59bf3edea..34c394caabe 100644 --- a/docs/cloud/onboard/02_migrate/01_migration_guides/02_postgres/migration_guide/03_migration_guide_part3.md +++ b/docs/cloud/onboard/02_migrate/01_migration_guides/02_postgres/migration_guide/03_migration_guide_part3.md @@ -18,11 +18,11 @@ import Image from '@theme/IdealImage'; We recommend users migrating from Postgres read [the guide for modeling data in ClickHouse](/data-modeling/schema-design). This guide uses the same Stack Overflow dataset and explores multiple approaches using ClickHouse features. -## Primary (Ordering) Keys in ClickHouse {#primary-ordering-keys-in-clickhouse} +## Primary (ordering) keys in ClickHouse {#primary-ordering-keys-in-clickhouse} Users coming from OLTP databases often look for the equivalent concept in ClickHouse. On noticing that ClickHouse supports a `PRIMARY KEY` syntax, users might be tempted to define their table schema using the same keys as their source OLTP database. This is not appropriate. -### How are ClickHouse Primary keys different? {#how-are-clickhouse-primary-keys-different} +### How are ClickHouse primary keys different? {#how-are-clickhouse-primary-keys-different} To understand why using your OLTP primary key in ClickHouse is not appropriate, users should understand the basics of ClickHouse indexing. We use Postgres as an example comparison, but these general concepts apply to other OLTP databases. diff --git a/docs/cloud/onboard/02_migrate/01_migration_guides/03_bigquery/01_overview.md b/docs/cloud/onboard/02_migrate/01_migration_guides/03_bigquery/01_overview.md index dc0feb8301f..270ca84f739 100644 --- a/docs/cloud/onboard/02_migrate/01_migration_guides/03_bigquery/01_overview.md +++ b/docs/cloud/onboard/02_migrate/01_migration_guides/03_bigquery/01_overview.md @@ -11,7 +11,7 @@ doc_type: 'guide' import bigquery_1 from '@site/static/images/migrations/bigquery-1.png'; import Image from '@theme/IdealImage'; -# Comparing ClickHouse Cloud and BigQuery +# Comparing ClickHouse Cloud and bigquery ## Resource organization {#resource-organization} @@ -21,9 +21,9 @@ The way resources are organized in ClickHouse Cloud is similar to [BigQuery's re ### Organizations {#organizations} -Similar to BigQuery, organizations are the root nodes in the ClickHouse cloud resource hierarchy. The first user you set up in your ClickHouse Cloud account is automatically assigned to an organization owned by the user. The user may invite additional users to the organization. +Similar to BigQuery, organizations are the root nodes in the ClickHouse Cloud resource hierarchy. The first user you set up in your ClickHouse Cloud account is automatically assigned to an organization owned by the user. The user may invite additional users to the organization. -### BigQuery Projects vs ClickHouse Cloud Services {#bigquery-projects-vs-clickhouse-cloud-services} +### Bigquery projects vs ClickHouse Cloud services {#bigquery-projects-vs-clickhouse-cloud-services} Within organizations, you can create services loosely equivalent to BigQuery projects because stored data in ClickHouse Cloud is associated with a service. There are [several service types available](/cloud/manage/cloud-tiers) in ClickHouse Cloud. Each ClickHouse Cloud service is deployed in a specific region and includes: @@ -31,15 +31,15 @@ Within organizations, you can create services loosely equivalent to BigQuery pro 2. An object storage folder where the service stores all the data. 3. An endpoint (or multiple endpoints created via ClickHouse Cloud UI console) - a service URL that you use to connect to the service (for example, `https://dv2fzne24g.us-east-1.aws.clickhouse.cloud:8443`) -### BigQuery Datasets vs ClickHouse Cloud Databases {#bigquery-datasets-vs-clickhouse-cloud-databases} +### Bigquery datasets vs ClickHouse Cloud databases {#bigquery-datasets-vs-clickhouse-cloud-databases} ClickHouse logically groups tables into databases. Like BigQuery datasets, ClickHouse databases are logical containers that organize and control access to table data. -### BigQuery Folders {#bigquery-folders} +### Bigquery folders {#bigquery-folders} ClickHouse Cloud currently has no concept equivalent to BigQuery folders. -### BigQuery Slot reservations and Quotas {#bigquery-slot-reservations-and-quotas} +### Bigquery slot reservations and quotas {#bigquery-slot-reservations-and-quotas} Like BigQuery slot reservations, you can [configure vertical and horizontal autoscaling](/manage/scaling#configuring-vertical-auto-scaling) in ClickHouse Cloud. For vertical autoscaling, you can set the minimum and maximum size for the memory and CPU cores of the compute nodes for a service. The service will then scale as needed within those bounds. These settings are also available during the initial service creation flow. Each compute node in the service has the same size. You can change the number of compute nodes within a service with [horizontal scaling](/manage/scaling#manual-horizontal-scaling). @@ -80,13 +80,13 @@ When presented with multiple options for ClickHouse types, consider the actual r ## Query acceleration techniques {#query-acceleration-techniques} -### Primary and Foreign keys and Primary index {#primary-and-foreign-keys-and-primary-index} +### Primary and foreign keys and primary index {#primary-and-foreign-keys-and-primary-index} In BigQuery, a table can have [primary key and foreign key constraints](https://cloud.google.com/bigquery/docs/information-schema-table-constraints). Typically, primary and foreign keys are used in relational databases to ensure data integrity. A primary key value is normally unique for each row and is not `NULL`. Each foreign key value in a row must be present in the primary key column of the primary key table or be `NULL`. In BigQuery, these constraints are not enforced, but the query optimizer may use this information to optimize queries better. In ClickHouse, a table can also have a primary key. Like BigQuery, ClickHouse doesn't enforce uniqueness for a table's primary key column values. Unlike BigQuery, a table's data is stored on disk [ordered](/guides/best-practices/sparse-primary-indexes#optimal-compression-ratio-of-data-files) by the primary key column(s). The query optimizer utilizes this sort order to prevent resorting, to minimize memory usage for joins, and to enable short-circuiting for limit clauses. Unlike BigQuery, ClickHouse automatically creates [a (sparse) primary index](/guides/best-practices/sparse-primary-indexes#an-index-design-for-massive-data-scales) based on the primary key column values. This index is used to speed up all queries that contain filters on the primary key columns. ClickHouse currently doesn't support foreign key constraints. -## Secondary indexes (Only available in ClickHouse) {#secondary-indexes-only-available-in-clickhouse} +## Secondary indexes (only available in ClickHouse) {#secondary-indexes-only-available-in-clickhouse} In addition to the primary index created from the values of a table's primary key columns, ClickHouse allows you to create secondary indexes on columns other than those in the primary key. ClickHouse offers several types of secondary indexes, each suited to different types of queries: diff --git a/docs/cloud/onboard/02_migrate/01_migration_guides/03_bigquery/02_migrating-to-clickhouse-cloud.md b/docs/cloud/onboard/02_migrate/01_migration_guides/03_bigquery/02_migrating-to-clickhouse-cloud.md index 30693c1ba73..dc6361dca45 100644 --- a/docs/cloud/onboard/02_migrate/01_migration_guides/03_bigquery/02_migrating-to-clickhouse-cloud.md +++ b/docs/cloud/onboard/02_migrate/01_migration_guides/03_bigquery/02_migrating-to-clickhouse-cloud.md @@ -21,13 +21,13 @@ import bigquery_11 from '@site/static/images/migrations/bigquery-11.png'; import bigquery_12 from '@site/static/images/migrations/bigquery-12.png'; import Image from '@theme/IdealImage'; -## Why use ClickHouse Cloud over BigQuery? {#why-use-clickhouse-cloud-over-bigquery} +## Why use ClickHouse Cloud over bigquery? {#why-use-clickhouse-cloud-over-bigquery} TLDR: Because ClickHouse is faster, cheaper, and more powerful than BigQuery for modern data analytics: -## Loading data from BigQuery to ClickHouse Cloud {#loading-data-from-bigquery-to-clickhouse-cloud} +## Loading data from bigquery to ClickHouse Cloud {#loading-data-from-bigquery-to-clickhouse-cloud} ### Dataset {#dataset} @@ -44,7 +44,7 @@ Migrating data between BigQuery and ClickHouse Cloud falls into two primary work - **Initial bulk load with periodic updates** - An initial dataset must be migrated along with periodic updates at set intervals e.g. daily. Updates here are handled by resending rows that have changed - identified by either a column that can be used for comparisons (e.g., a date). Deletes are handled with a complete periodic reload of the dataset. - **Real time replication or CDC** - An initial dataset must be migrated. Changes to this dataset must be reflected in ClickHouse in near-real time with only a delay of several seconds acceptable. This is effectively a [Change Data Capture (CDC) process](https://en.wikipedia.org/wiki/Change_data_capture) where tables in BigQuery must be synchronized with ClickHouse i.e. inserts, updates and deletes in the BigQuery table must be applied to an equivalent table in ClickHouse. -#### Bulk loading via Google Cloud Storage (GCS) {#bulk-loading-via-google-cloud-storage-gcs} +#### Bulk loading via Google cloud storage (gcs) {#bulk-loading-via-google-cloud-storage-gcs} BigQuery supports exporting data to Google's object store (GCS). For our example data set: @@ -144,7 +144,7 @@ INSERT INTO stackoverflow.posts SELECT * FROM gcs( 'gs://clickhouse-public-datas We don't retain any nulls in our new schema. The above insert converts these implicitly to default values for their respective types - 0 for integers and an empty value for strings. ClickHouse also automatically converts any numerics to their target precision. -## How are ClickHouse Primary keys different? {#how-are-clickhouse-primary-keys-different} +## How are ClickHouse primary keys different? {#how-are-clickhouse-primary-keys-different} As described [here](/migrations/bigquery), like in BigQuery, ClickHouse doesn't enforce uniqueness for a table's primary key column values. @@ -394,7 +394,7 @@ We recommend using projections when: - A complete reordering of the data is required. While the expression in the projection can, in theory, use a `GROUP BY,` materialized views are more effective for maintaining aggregates. The query optimizer is also more likely to exploit projections that use a simple reordering, i.e., `SELECT * ORDER BY x`. Users can select a subset of columns in this expression to reduce storage footprint. - Users are comfortable with the associated increase in storage footprint and overhead of writing data twice. Test the impact on insertion speed and [evaluate the storage overhead](/data-compression/compression-in-clickhouse). -## Rewriting BigQuery queries in ClickHouse {#rewriting-bigquery-queries-in-clickhouse} +## Rewriting bigquery queries in ClickHouse {#rewriting-bigquery-queries-in-clickhouse} The following provides example queries comparing BigQuery to ClickHouse. This list aims to demonstrate how to exploit ClickHouse features to significantly simplify queries. The examples here use the full Stack Overflow dataset (up to April 2024). diff --git a/docs/cloud/onboard/02_migrate/01_migration_guides/03_bigquery/03_loading-data.md b/docs/cloud/onboard/02_migrate/01_migration_guides/03_bigquery/03_loading-data.md index 7c3b470aebc..09c5d6037c0 100644 --- a/docs/cloud/onboard/02_migrate/01_migration_guides/03_bigquery/03_loading-data.md +++ b/docs/cloud/onboard/02_migrate/01_migration_guides/03_bigquery/03_loading-data.md @@ -27,7 +27,7 @@ Exporting data from BigQuery to ClickHouse is dependent on the size of your data -## Export table data to GCS {#1-export-table-data-to-gcs} +## Export table data to gcs {#1-export-table-data-to-gcs} In this step, we utilize the [BigQuery SQL workspace](https://cloud.google.com/bigquery/docs/bigquery-web-ui) to execute our SQL commands. Below, we export a BigQuery table named `mytable` to a GCS bucket using the [`EXPORT DATA`](https://cloud.google.com/bigquery/docs/reference/standard-sql/other-statements) statement. @@ -63,7 +63,7 @@ This approach has a number of advantages: - Exports produce multiple files automatically, limiting each to a maximum of 1GB of table data. This is beneficial to ClickHouse since it allows imports to be parallelized. - Parquet, as a column-oriented format, represents a better interchange format since it is inherently compressed and faster for BigQuery to export and ClickHouse to query -## Importing data into ClickHouse from GCS {#2-importing-data-into-clickhouse-from-gcs} +## Importing data into ClickHouse from gcs {#2-importing-data-into-clickhouse-from-gcs} Once the export is complete, we can import this data into a ClickHouse table. You can use the [ClickHouse SQL console](/integrations/sql-clients/sql-console) or [`clickhouse-client`](/interfaces/cli) to execute the commands below. diff --git a/docs/cloud/onboard/02_migrate/01_migration_guides/04_snowflake/02_migration_guide.md b/docs/cloud/onboard/02_migrate/01_migration_guides/04_snowflake/02_migration_guide.md index cebdd3940e9..98221117949 100644 --- a/docs/cloud/onboard/02_migrate/01_migration_guides/04_snowflake/02_migration_guide.md +++ b/docs/cloud/onboard/02_migrate/01_migration_guides/04_snowflake/02_migration_guide.md @@ -11,7 +11,7 @@ doc_type: 'guide' import migrate_snowflake_clickhouse from '@site/static/images/migrations/migrate_snowflake_clickhouse.png'; import Image from '@theme/IdealImage'; -# Migrate from Snowflake to ClickHouse +# Migrate from snowflake to ClickHouse > This guide shows you how to migrate data from Snowflake to ClickHouse. @@ -22,7 +22,7 @@ of ClickHouse. -## Export data from Snowflake {#1-exporting-data-from-snowflake} +## Export data from snowflake {#1-exporting-data-from-snowflake} diff --git a/docs/cloud/onboard/02_migrate/01_migration_guides/06_redshift/01_overview.md b/docs/cloud/onboard/02_migrate/01_migration_guides/06_redshift/01_overview.md index e233884bc5c..d0196a68042 100644 --- a/docs/cloud/onboard/02_migrate/01_migration_guides/06_redshift/01_overview.md +++ b/docs/cloud/onboard/02_migrate/01_migration_guides/06_redshift/01_overview.md @@ -7,7 +7,7 @@ title: 'Comparing ClickHouse Cloud and Amazon Redshift' doc_type: 'guide' --- -# Amazon Redshift to ClickHouse migration +# Amazon redshift to ClickHouse migration > This document provides an introduction to migrating data from Amazon Redshift to ClickHouse. @@ -28,7 +28,7 @@ purpose. As a result, they increasingly turn to ClickHouse to benefit from superior query performance and data compression, either as a replacement or a "speed layer" deployed alongside existing Redshift workloads. -## ClickHouse vs Redshift {#clickhouse-vs-redshift} +## ClickHouse vs redshift {#clickhouse-vs-redshift} For users heavily invested in the AWS ecosystem, Redshift represents a natural choice when faced with data warehousing needs. Redshift differs from diff --git a/docs/cloud/onboard/02_migrate/01_migration_guides/06_redshift/02_migration_guide.md b/docs/cloud/onboard/02_migrate/01_migration_guides/06_redshift/02_migration_guide.md index cfb41fe1d4f..57b978eeb86 100644 --- a/docs/cloud/onboard/02_migrate/01_migration_guides/06_redshift/02_migration_guide.md +++ b/docs/cloud/onboard/02_migrate/01_migration_guides/06_redshift/02_migration_guide.md @@ -9,6 +9,6 @@ doc_type: 'guide' import MigrationGuide from '@site/docs/integrations/data-ingestion/redshift/_snippets/_migration_guide.md' -# Amazon Redshift to ClickHouse migration guide +# Amazon redshift to ClickHouse migration guide \ No newline at end of file diff --git a/docs/cloud/onboard/02_migrate/01_migration_guides/06_redshift/03_sql_translation_reference.md b/docs/cloud/onboard/02_migrate/01_migration_guides/06_redshift/03_sql_translation_reference.md index e87e5a7e301..7927d8af903 100644 --- a/docs/cloud/onboard/02_migrate/01_migration_guides/06_redshift/03_sql_translation_reference.md +++ b/docs/cloud/onboard/02_migrate/01_migration_guides/06_redshift/03_sql_translation_reference.md @@ -7,7 +7,7 @@ title: 'Amazon Redshift SQL translation guide' doc_type: 'reference' --- -# Amazon Redshift SQL translation guide +# Amazon redshift SQL translation guide ## Data types {#data-types} diff --git a/docs/cloud/onboard/02_migrate/01_migration_guides/07_OSS_to_Cloud/01_clickhouse-to-cloud.md b/docs/cloud/onboard/02_migrate/01_migration_guides/07_OSS_to_Cloud/01_clickhouse-to-cloud.md index 9bd863522b0..ed95fd94aa0 100644 --- a/docs/cloud/onboard/02_migrate/01_migration_guides/07_OSS_to_Cloud/01_clickhouse-to-cloud.md +++ b/docs/cloud/onboard/02_migrate/01_migration_guides/07_OSS_to_Cloud/01_clickhouse-to-cloud.md @@ -16,7 +16,7 @@ import self_managed_04 from '@site/static/images/integrations/migration/self-man import self_managed_05 from '@site/static/images/integrations/migration/self-managed-05.png'; import self_managed_06 from '@site/static/images/integrations/migration/self-managed-06.png'; -# Migrating between self-managed ClickHouse and ClickHouse Cloud +# Migrating between Self-managed ClickHouse and ClickHouse Cloud @@ -192,7 +192,7 @@ Modify the allow list and allow access from **Anywhere** temporarily. See the [I - Verify the data in the destination service -#### Re-establish the IP access list on the source {#re-establish-the-ip-access-list-on-the-source} +#### Re-establish the ip access list on the source {#re-establish-the-ip-access-list-on-the-source} If you exported the access list earlier, then you can re-import it using **Share**, otherwise re-add your entries to the access list. diff --git a/docs/cloud/onboard/02_migrate/01_migration_guides/08_other_methods/01_clickhouse-local-etl.md b/docs/cloud/onboard/02_migrate/01_migration_guides/08_other_methods/01_clickhouse-local-etl.md index 0fd66a80504..8cd36ca5d63 100644 --- a/docs/cloud/onboard/02_migrate/01_migration_guides/08_other_methods/01_clickhouse-local-etl.md +++ b/docs/cloud/onboard/02_migrate/01_migration_guides/08_other_methods/01_clickhouse-local-etl.md @@ -87,10 +87,10 @@ In order for the `remoteSecure` function to connect to your ClickHouse Cloud ser -## Example 1: Migrating from MySQL to ClickHouse Cloud with an Integration engine {#example-1-migrating-from-mysql-to-clickhouse-cloud-with-an-integration-engine} +## Example 1: migrating from MySQL to ClickHouse Cloud with an integration engine {#example-1-migrating-from-mysql-to-clickhouse-cloud-with-an-integration-engine} We will use the [integration table engine](/engines/table-engines/integrations/mysql/) (created on-the-fly by the [mysql table function](/sql-reference/table-functions/mysql/)) for reading data from the source MySQL database and we will use the [remoteSecure table function](/sql-reference/table-functions/remote/) -for writing the data into a destination table on your ClickHouse cloud service. +for writing the data into a destination table on your ClickHouse Cloud service. @@ -128,10 +128,10 @@ No data is stored locally on the `clickhouse-local` host machine. Instead, the d and then immediately written to the destination table on the ClickHouse Cloud service. ::: -## Example 2: Migrating from MySQL to ClickHouse Cloud with the JDBC bridge {#example-2-migrating-from-mysql-to-clickhouse-cloud-with-the-jdbc-bridge} +## Example 2: migrating from MySQL to ClickHouse Cloud with the jdbc bridge {#example-2-migrating-from-mysql-to-clickhouse-cloud-with-the-jdbc-bridge} We will use the [JDBC integration table engine](/engines/table-engines/integrations/jdbc.md) (created on-the-fly by the [jdbc table function](/sql-reference/table-functions/jdbc.md)) together with the [ClickHouse JDBC Bridge](https://github.com/ClickHouse/clickhouse-jdbc-bridge) and the MySQL JDBC driver for reading data from the source MySQL database and we will use the [remoteSecure table function](/sql-reference/table-functions/remote.md) -for writing the data into a destination table on your ClickHouse cloud service. +for writing the data into a destination table on your ClickHouse Cloud service. diff --git a/docs/cloud/reference/01_changelog/01_changelog.md b/docs/cloud/reference/01_changelog/01_changelog.md index 0a773b08b58..9b99fc2f74d 100644 --- a/docs/cloud/reference/01_changelog/01_changelog.md +++ b/docs/cloud/reference/01_changelog/01_changelog.md @@ -245,7 +245,7 @@ to get up and running. ## February 21, 2025 {#february-21-2025} -### ClickHouse Bring Your Own Cloud (BYOC) for AWS is now generally available {#clickhouse-byoc-for-aws-ga} +### ClickHouse bring your own cloud (BYOC) for AWS is now generally available {#clickhouse-byoc-for-aws-ga} In this deployment model, data plane components (compute, storage, backups, logs, metrics) run in the Customer VPC, while the control plane (web access, APIs, and billing) @@ -271,14 +271,14 @@ customers in **us-east-1** and **us-west-2** regions. Users who wish to launch a service in a PCI-compliant environment can contact [support](https://clickhouse.com/support/program) for assistance. -### Transparent Data Encryption and Customer Managed Encryption Keys on Google Cloud Platform {#tde-and-cmek-on-gcp} +### Transparent data encryption and customer managed encryption keys on Google cloud platform {#tde-and-cmek-on-gcp} Support for **Transparent Data Encryption (TDE)** and **Customer Managed Encryption Keys (CMEK)** is now available for ClickHouse Cloud on **Google Cloud Platform (GCP)**. - Please refer to the [documentation](https://clickhouse.com/docs/cloud/security/cmek#transparent-data-encryption-tde) of these features for more information. -### AWS Middle East (UAE) availability {#aws-middle-east-uae-availability} +### AWS middle east (UAE) availability {#aws-middle-east-uae-availability} New region support is added for ClickHouse Cloud, which is now available in the **AWS Middle East (UAE) me-central-1** region. @@ -308,7 +308,7 @@ To support these changes, we are restructuring our current **Development** and * You can read about these and other functional changes in this [blog](https://clickhouse.com/blog/evolution-of-clickhouse-cloud-new-features-superior-performance-tailored-offerings). Existing customers will need to take action to select a [new plan](https://clickhouse.com/pricing). Customer-facing communication was sent via email to organization administrators. -### Warehouses: Compute-compute separation (GA) {#warehouses-compute-compute-separation-ga} +### Warehouses: compute-compute separation (GA) {#warehouses-compute-compute-separation-ga} Compute-compute separation (also known as "Warehouses") is Generally Available; please refer to [blog](https://clickhouse.com/blog/introducing-warehouses-compute-compute-separation-in-clickhouse-cloud) for more details and the [documentation](/cloud/reference/warehouses). @@ -344,7 +344,7 @@ Users can schedule upgrades for their services. This feature is supported for En [Golang](https://github.com/ClickHouse/clickhouse-go/releases/tag/v2.30.1), [Python](https://github.com/ClickHouse/clickhouse-connect/releases/tag/v0.8.11), and [NodeJS](https://github.com/ClickHouse/clickhouse-js/releases/tag/1.10.1) clients added support for Dynamic, Variant, and JSON types. -### DBT support for refreshable materialized views {#dbt-support-for-refreshable-materialized-views} +### Dbt support for refreshable materialized views {#dbt-support-for-refreshable-materialized-views} DBT now [supports Refreshable Materialized Views](https://github.com/ClickHouse/dbt-clickhouse/releases/tag/v1.8.7) in the `1.8.7` release. @@ -386,15 +386,15 @@ Org Admins can now add more email addresses to a specific notification as additi ## December 6, 2024 {#december-6-2024} -### BYOC (beta) {#byoc-beta} +### BYOC (Beta) {#byoc-beta} Bring Your Own Cloud for AWS is now available in Beta. This deployment model allows you to deploy and run ClickHouse Cloud in your own AWS account. We support deployments in 11+ AWS regions, with more coming soon. Please [contact support](https://clickhouse.com/support/program) for access. Note that this deployment is reserved for large-scale deployments. -### Postgres Change Data Capture (CDC) connector in ClickPipes {#postgres-change-data-capture-cdc-connector-in-clickpipes} +### Postgres change data capture (CDC) connector in ClickPipes {#postgres-change-data-capture-cdc-connector-in-clickpipes} This turnkey integration enables customers to replicate their Postgres databases to ClickHouse Cloud in just a few clicks and leverage ClickHouse for blazing-fast analytics. You can use this connector for both continuous replication and one-time migrations from Postgres. -### Dashboards (beta) {#dashboards-beta} +### Dashboards (Beta) {#dashboards-beta} This week, we're excited to announce the Beta launch of Dashboards in ClickHouse Cloud. With Dashboards, users can turn saved queries into visualizations, organize visualizations onto dashboards, and interact with dashboards using query parameters. To get started, follow the [dashboards documentation](/cloud/manage/dashboards). @@ -428,7 +428,7 @@ ClickHouse already offers robust support for vector-based use cases, with a wide To get started, [please sign up for the early access waitlist](https://clickhouse.com/cloud/vector-search-index-waitlist). -### ClickHouse-connect (Python) and ClickHouse Kafka Connect users {#clickhouse-connect-python-and-clickhouse-kafka-connect-users} +### Clickhouse-connect (Python) and ClickHouse Kafka connect users {#clickhouse-connect-python-and-clickhouse-kafka-connect-users} Notification emails went out to customers who had experienced issues where the clients could encounter a `MEMORY_LIMIT_EXCEEDED` exception. @@ -436,13 +436,13 @@ Please upgrade to: - Kafka-Connect: > 1.2.5 - ClickHouse-Connect (Java): > 0.8.6 -### ClickPipes now supports cross-VPC resource access on AWS {#clickpipes-now-supports-cross-vpc-resource-access-on-aws} +### ClickPipes now supports cross-vpc resource access on AWS {#clickpipes-now-supports-cross-vpc-resource-access-on-aws} You can now grant uni-directional access to a specific data source like AWS MSK. With Cross-VPC resource access with AWS PrivateLink and VPC Lattice, you can share individual resources across VPC and account boundaries, or even from on-premise networks without compromising on privacy and security when going over a public network. To get started and set up a resource share, you can read the [announcement post](https://clickhouse.com/blog/clickpipes-crossvpc-resource-endpoints?utm_medium=web&utm_source=changelog). -### ClickPipes now supports IAM for AWS MSK {#clickpipes-now-supports-iam-for-aws-msk} +### ClickPipes now supports iam for AWS msk {#clickpipes-now-supports-iam-for-aws-msk} You can now use IAM authentication to connect to an MSK broker with AWS MSK ClickPipes. To get started, review our [documentation](/integrations/clickpipes/kafka/best-practices/#iam). @@ -460,7 +460,7 @@ This dashboard allows you to view queries based on the [system.dashboards](/oper -### AI-powered SQL autocomplete {#ai-powered-sql-autocomplete} +### Ai-powered SQL autocomplete {#ai-powered-sql-autocomplete} We've improved autocomplete significantly, allowing you to get in-line SQL completions as you write your queries with the new AI Copilot. This feature can be enabled by toggling the **"Enable Inline Code Completion"** setting for any ClickHouse Cloud service. @@ -472,7 +472,7 @@ You can now assign users in your organization to a new **Billing** role that all ## November 8, 2024 {#november-8-2024} -### Customer Notifications in ClickHouse Cloud {#customer-notifications-in-clickhouse-cloud} +### Customer notifications in ClickHouse Cloud {#customer-notifications-in-clickhouse-cloud} ClickHouse Cloud now provides in-console and email notifications for several billing and scaling events. Customers can configure these notifications via the cloud console notification center to only appear on the UI, receive emails, or both. You can configure the category and severity of the notifications you receive at the service level. @@ -486,19 +486,19 @@ Please see the [ClickHouse docs](/cloud/notifications) to learn more about how t ## October 4, 2024 {#october-4-2024} -### ClickHouse Cloud now offers HIPAA-ready services in Beta for GCP {#clickhouse-cloud-now-offers-hipaa-ready-services-in-beta-for-gcp} +### ClickHouse Cloud now offers hipaa-ready services in Beta for GCP {#clickhouse-cloud-now-offers-hipaa-ready-services-in-beta-for-gcp} Customers looking for increased security for protected health information (PHI) can now onboard to ClickHouse Cloud in [Google Cloud Platform (GCP)](https://cloud.google.com/). ClickHouse has implemented administrative, physical and technical safeguards prescribed by the [HIPAA Security Rule](https://www.hhs.gov/hipaa/for-professionals/security/index.html) and now has configurable security settings that can be implemented, depending on your specific use case and workload. For more information on available security settings, please review our [Security features page](/cloud/security). Services are available in GCP `us-central-1` to customers with the **Dedicated** service type and require a Business Associate Agreement (BAA). Contact [sales](mailto:sales@clickhouse.com) or [support](https://clickhouse.com/support/program) to request access to this feature or join the wait list for additional GCP, AWS, and Azure regions. -### Compute-compute separation is now in private preview for GCP and Azure {#compute-compute-separation-is-now-in-private-preview-for-gcp-and-azure} +### Compute-compute separation is now in private Preview for GCP and Azure {#compute-compute-separation-is-now-in-private-preview-for-gcp-and-azure} We recently announced the Private Preview for Compute-Compute Separation for AWS. We're happy to announce that it is now available for GCP and Azure. Compute-compute separation allows you to designate specific services as read-write or read-only services, allowing you to design the optimal compute configuration for your application to optimize cost and performance. Please [read the docs](/cloud/reference/warehouses) for more details. -### Self-service MFA recovery codes {#self-service-mfa-recovery-codes} +### Self-service mfa recovery codes {#self-service-mfa-recovery-codes} Customers using multi-factor authentication can now obtain recovery codes that can be used in the event of a lost phone or accidentally deleted token. Customers enrolling in MFA for the first time will be provided the code on set up. Customers with existing MFA can obtain a recovery code by removing their existing MFA token and adding a new one. @@ -532,7 +532,7 @@ It is now possible to ingest an entire Kafka or Kinesis message without parsing Terraform allows you to control your ClickHouse Cloud services programmatically, then store your configuration as code. Our Terraform provider has almost 200,000 downloads and is now officially v1.0.0. This new version includes improvements such as better retry logic and a new resource to attach private endpoints to your ClickHouse Cloud service. You can download the [Terraform provider here](https://registry.terraform.io/providers/ClickHouse/clickhouse/latest) and view the [full changelog here](https://github.com/ClickHouse/terraform-provider-clickhouse/releases/tag/v1.0.0). -### 2024 SOC 2 Type II report and updated ISO 27001 certificate {#2024-soc-2-type-ii-report-and-updated-iso-27001-certificate} +### 2024 SOC 2 type ii report and updated iso 27001 certificate {#2024-soc-2-type-ii-report-and-updated-iso-27001-certificate} We are proud to announce the availability of our 2024 SOC 2 Type II report and updated ISO 27001 certificate, both of which include our recently launched services on Azure as well as continued coverage of services in AWS and GCP. @@ -542,7 +542,7 @@ Please also check out our [Trust Center](https://trust.clickhouse.com/) for secu ## August 15, 2024 {#august-15-2024} -### Compute-compute separation is now in Private Preview for AWS {#compute-compute-separation-is-now-in-private-preview-for-aws} +### Compute-compute separation is now in private Preview for AWS {#compute-compute-separation-is-now-in-private-preview-for-aws} For existing ClickHouse Cloud services, replicas handle both reads and writes, and there is no way to configure a certain replica to handle only one kind of operation. We have an upcoming new feature called Compute-compute separation that allows you to designate specific services as read-write or read-only services, allowing you to design the optimal compute configuration for your application to optimize cost and performance. @@ -550,7 +550,7 @@ Our new compute-compute separation feature enables you to create multiple comput -### ClickPipes for S3 and GCS now in GA, Continuous mode support {#clickpipes-for-s3-and-gcs-now-in-ga-continuous-mode-support} +### ClickPipes for S3 and gcs now in GA, continuous mode support {#clickpipes-for-s3-and-gcs-now-in-ga-continuous-mode-support} ClickPipes is the easiest way to ingest data into ClickHouse Cloud. We're happy to announce that [ClickPipes](https://clickhouse.com/cloud/clickpipes) for S3 and GCS is now **Generally Available**. ClickPipes supports both one-time batch ingest and "continuous mode". An ingest task will load all the files matched by a pattern from a specific remote bucket into the ClickHouse destination table. In "continuous mode", the ClickPipes job will run constantly, ingesting matching files that get added into the remote object storage bucket as they arrive. This will allow users to turn any object storage bucket into a fully fledged staging area for ingesting data into ClickHouse Cloud. Read more about ClickPipes in [our documentation](/integrations/clickpipes). @@ -560,7 +560,7 @@ ClickPipes is the easiest way to ingest data into ClickHouse Cloud. We're happy In our last cloud changelog, we announced the Private Preview for exporting [Prometheus](https://prometheus.io/) metrics from ClickHouse Cloud. This feature allows you to use the [ClickHouse Cloud API](/cloud/manage/api/api-overview) to get your metrics into tools like [Grafana](https://grafana.com/) and [Datadog](https://www.datadoghq.com/) for visualization. We're happy to announce that this feature is now **Generally Available**. Please see [our docs](/integrations/prometheus) to learn more about this feature. -### Table inspector in Cloud console {#table-inspector-in-cloud-console} +### Table inspector in cloud console {#table-inspector-in-cloud-console} ClickHouse has commands like [`DESCRIBE`](/sql-reference/statements/describe-table) that allow you to introspect your table to examine schema. These commands output to the console, but they are often not convenient to use as you need to combine several queries to retrieve all pertinent data about your tables and columns. @@ -568,11 +568,11 @@ We recently launched a **Table Inspector** in the cloud console which allows you -### New Java Client API {#new-java-client-api} +### New Java client API {#new-java-client-api} Our [Java Client](https://github.com/ClickHouse/clickhouse-java) is one of the most popular clients that users use to connect to ClickHouse. We wanted to make it even easier and more intuitive to use, including a re-designed API and various performance optimizations. These changes will make it much easier to connect to ClickHouse from your Java applications. You can read more about how to use the updated Java Client in this [blog post](https://clickhouse.com/blog/java-client-sequel). -### New Analyzer is enabled by default {#new-analyzer-is-enabled-by-default} +### New analyzer is enabled by default {#new-analyzer-is-enabled-by-default} For the last couple of years, we've been working on a new analyzer for query analysis and optimization. This analyzer improves query performance and will allow us to make further optimizations, including faster and more efficient `JOIN`s. Previously, it was required that new users enable this feature using the setting `allow_experimental_analyzer`. This improved analyzer is now available on new ClickHouse Cloud services by default. @@ -597,7 +597,7 @@ Our new Query Insights UI in the Cloud console makes ClickHouse's built-in query -### Prometheus endpoint for metrics (private preview) {#prometheus-endpoint-for-metrics-private-preview} +### Prometheus endpoint for metrics (private Preview) {#prometheus-endpoint-for-metrics-private-preview} Perhaps one of our most requested features: you can now export [Prometheus](https://prometheus.io/) metrics from ClickHouse Cloud to [Grafana](https://grafana.com/) and [Datadog](https://www.datadoghq.com/) for visualization. Prometheus provides an open-source solution to monitor ClickHouse and set up custom alerts. Access to Prometheus metrics for your ClickHouse Cloud service is available via the [ClickHouse Cloud API](/integrations/prometheus). This feature is currently in Private Preview. Please reach out to the [support team](https://clickhouse.com/support/program) to enable this feature for your organization. @@ -608,7 +608,7 @@ Perhaps one of our most requested features: you can now export [Prometheus](http ## June 13, 2024 {#june-13-2024} -### Configurable offsets for Kafka ClickPipes Connector (Beta) {#configurable-offsets-for-kafka-clickpipes-connector-beta} +### Configurable offsets for Kafka ClickPipes connector (Beta) {#configurable-offsets-for-kafka-clickpipes-connector-beta} Until recently, whenever you set up a new [Kafka Connector for ClickPipes](/integrations/clickpipes/kafka), it always consumed data from the beginning of the Kafka topic. In this situation, it may not be flexible enough to fit specific use cases when you need to reprocess historical data, monitor new incoming data, or resume from a precise point. @@ -621,7 +621,7 @@ The following options are available: -### Enroll services to the Fast release channel {#enroll-services-to-the-fast-release-channel} +### Enroll services to the fast release channel {#enroll-services-to-the-fast-release-channel} The Fast release channel allows your services to receive updates ahead of the release schedule. Previously, this feature required assistance from the support team to enable. Now, you can use the ClickHouse Cloud console to enable this feature for your services directly. Simply navigate to **Settings**, and click **Enroll in fast releases**. Your service will now receive updates as soon as they are available. @@ -643,7 +643,7 @@ We're happy to announce that you can now easily share queries via the ClickHouse -### ClickHouse Cloud for Microsoft Azure is now in beta {#clickhouse-cloud-for-microsoft-azure-is-now-in-beta} +### ClickHouse Cloud for Microsoft Azure is now in Beta {#clickhouse-cloud-for-microsoft-azure-is-now-in-beta} We've finally launched the ability to create ClickHouse Cloud services on Microsoft Azure. We already have many customers using ClickHouse Cloud on Azure in production as part of our Private Preview program. Now, anyone can create their own service on Azure. All of your favorite ClickHouse features that are supported on AWS and GCP will also work on Azure. @@ -651,7 +651,7 @@ We expect to have ClickHouse Cloud for Azure ready for General Availability in t Note: **Development** services for Azure are not supported at this time. -### Set up Private Link via the Cloud console {#set-up-private-link-via-the-cloud-console} +### Set up private link via the cloud console {#set-up-private-link-via-the-cloud-console} Our Private Link feature allows you to connect your ClickHouse Cloud services with internal services in your cloud provider account without having to direct traffic to the public internet, saving costs and enhancing security. Previously, this was difficult to set up and required using the ClickHouse Cloud API. @@ -661,7 +661,7 @@ You can now configure private endpoints in just a few clicks directly from the C ## May 17, 2024 {#may-17-2024} -### Ingest data from Amazon Kinesis using ClickPipes (beta) {#ingest-data-from-amazon-kinesis-using-clickpipes-beta} +### Ingest data from Amazon kinesis using ClickPipes (Beta) {#ingest-data-from-amazon-kinesis-using-clickpipes-beta} ClickPipes is an exclusive service provided by ClickHouse Cloud to ingest data without code. Amazon Kinesis is AWS's fully managed streaming service to ingest and store data streams for processing. We are thrilled to launch the ClickPipes beta for Amazon Kinesis, one of our most requested integrations. We're looking to add more integrations to ClickPipes, so please let us know which data source you'd like us to support. Read more about this feature [here](https://clickhouse.com/blog/clickpipes-amazon-kinesis). @@ -669,7 +669,7 @@ You can try the new Amazon Kinesis integration for ClickPipes in the cloud conso -### Configurable backups (private preview) {#configurable-backups-private-preview} +### Configurable backups (private Preview) {#configurable-backups-private-preview} Backups are important for every database (no matter how reliable), and we've taken backups very seriously since day 1 of ClickHouse Cloud. This week, we launched Configurable Backups, which allows for much more flexibility for your service's backups. You can now control start time, retention, and frequency. This feature is available for **Production** and **Dedicated** services and is not available for **Development** services. As this feature is in private preview, please contact support@clickhouse.com to enable this for your service. Read more about configurable backups [here](https://clickhouse.com/blog/configurable-backups-in-clickhouse-cloud). @@ -679,13 +679,13 @@ When you write a SQL query for ClickHouse, you still need to connect to ClickHou -### Official ClickHouse Certification is now available {#official-clickhouse-certification-is-now-available} +### Official ClickHouse certification is now available {#official-clickhouse-certification-is-now-available} There are 12 free training modules in ClickHouse Develop training course. Prior to this week, there was no official way to prove your mastery in ClickHouse. We recently launched an official exam to become a **ClickHouse Certified Developer**. Completing this exam allows you to share with current and prospective employers your mastery in ClickHouse on topics including data ingestion, modeling, analysis, performance optimization, and more. You can take the exam [here](https://clickhouse.com/learn/certification) or read more about ClickHouse certification in this [blog post](https://clickhouse.com/blog/first-official-clickhouse-certification). ## April 25, 2024 {#april-25-2024} -### Load data from S3 and GCS using ClickPipes {#load-data-from-s3-and-gcs-using-clickpipes} +### Load data from S3 and gcs using ClickPipes {#load-data-from-s3-and-gcs-using-clickpipes} You may have noticed in our newly released cloud console that there's a new section called "Data sources". The "Data sources" page is powered by ClickPipes, a native ClickHouse Cloud feature which lets you easily insert data from a variety of sources into ClickHouse Cloud. @@ -786,7 +786,7 @@ This release makes available in early access the new Cloud console experience, C ### ClickHouse version upgrade {#clickhouse-version-upgrade} - Optimizations for FINAL, vectorization improvements, faster aggregations - see [23.12 release blog](https://clickhouse.com/blog/clickhouse-release-23-12#optimizations-for-final) for details. - New functions for processing punycode, string similarity, detecting outliers, as well as memory optimizations for merges and Keeper - see [24.1 release blog](https://clickhouse.com/blog/clickhouse-release-24-01) and [presentation](https://presentations.clickhouse.com/release_24.1/) for details. -- This ClickHouse cloud version is based on 24.1, you can see dozens of new features, performance improvements, and bug fixes. See core database [changelogs](/whats-new/changelog/2023#2312) for details. +- This ClickHouse Cloud version is based on 24.1, you can see dozens of new features, performance improvements, and bug fixes. See core database [changelogs](/whats-new/changelog/2023#2312) for details. ### Integrations changes {#integrations-changes-2} - Grafana: Fixed dashboard migration for v4, ad-hoc filtering logic @@ -819,7 +819,7 @@ This release upgrades the core database version, adds ability to set up private ### ClickHouse version upgrade {#clickhouse-version-upgrade-1} - S3Queue table engine for continuous, scheduled data loading from S3 is production-ready - [see 23.11 release blog](https://clickhouse.com/blog/clickhouse-release-23-11) for details. - Significant performance improvements for FINAL and vectorization improvements for SIMD instructions resulting in faster queries - [see 23.12 release blog](https://clickhouse.com/blog/clickhouse-release-23-12#optimizations-for-final) for details. -- This ClickHouse cloud version is based on 23.12, you can see dozens of new features, performance improvements, and bug fixes. See [core database changelogs](/whats-new/changelog/2023#2312) for details. +- This ClickHouse Cloud version is based on 23.12, you can see dozens of new features, performance improvements, and bug fixes. See [core database changelogs](/whats-new/changelog/2023#2312) for details. ### Console changes {#console-changes-4} - Added ability to set up AWS Private Link and GCP Private Service Connect through Terraform provider @@ -1141,7 +1141,7 @@ and adds more visibility into the status of MFA users in the Cloud console. ClickHouse Cloud on GCP is now GA, see the entry for June twenty above. ::: -### ClickHouse Cloud on GCP is now available in public beta {#clickhouse-cloud-on-gcp-is-now-available-in-public-beta-now-ga-see-june-20th-entry-above} +### ClickHouse Cloud on GCP is now available in public Beta {#clickhouse-cloud-on-gcp-is-now-available-in-public-beta-now-ga-see-june-20th-entry-above} :::note ClickHouse Cloud on GCP is now GA, see the [June 20th](#june-20-2023) entry above. @@ -1400,7 +1400,7 @@ This release introduces seamless logins for administrators to SQL console, impro ### Integrations changes {#integrations-changes-26} - The [Metabase plugin](/integrations/data-visualization/metabase-and-clickhouse.md) got a long-awaited v0.9.1 major update. Now it is compatible with the latest Metabase version and has been thoroughly tested against ClickHouse Cloud. -## December 6, 2022 - General availability {#december-6-2022---general-availability} +## December 6, 2022 - general availability {#december-6-2022---general-availability} ClickHouse Cloud is now production-ready with SOC2 Type II compliance, uptime SLAs for production workloads, and public status page. This release includes major new capabilities like AWS Marketplace integration, SQL console - a data exploration workbench for ClickHouse users, and ClickHouse Academy - self-paced learning in ClickHouse Cloud. Learn more in this [blog](https://clickhouse.com/blog/clickhouse-cloud-generally-available). diff --git a/docs/cloud/reference/01_changelog/02_release_notes/24_02.md b/docs/cloud/reference/01_changelog/02_release_notes/24_02.md index 2ff557d0e8e..782a6a46853 100644 --- a/docs/cloud/reference/01_changelog/02_release_notes/24_02.md +++ b/docs/cloud/reference/01_changelog/02_release_notes/24_02.md @@ -45,7 +45,7 @@ doc_type: 'changelog' * Separate limits on number of waiting and executing queries. Added new server setting `max_waiting_queries` that limits the number of queries waiting due to `async_load_databases`. Existing limits on number of executing queries no longer count waiting queries. [#61053](https://github.com/ClickHouse/ClickHouse/pull/61053) ([Sergei Trifonov](https://github.com/serxa)). * Add support for `ATTACH PARTITION ALL`. [#61107](https://github.com/ClickHouse/ClickHouse/pull/61107) ([Kirill Nikiforov](https://github.com/allmazz)). -#### Performance Improvement {#performance-improvement} +#### Performance improvement {#performance-improvement} * Eliminates min/max/any/anyLast aggregators of GROUP BY keys in SELECT section. [#52230](https://github.com/ClickHouse/ClickHouse/pull/52230) ([JackyWoo](https://github.com/JackyWoo)). * Improve the performance of serialized aggregation method when involving multiple [nullable] columns. This is a general version of [#51399](https://github.com/ClickHouse/ClickHouse/issues/51399) that doesn't compromise on abstraction integrity. [#55809](https://github.com/ClickHouse/ClickHouse/pull/55809) ([Amos Bird](https://github.com/amosbird)). * Lazy build join output to improve performance of ALL join. [#58278](https://github.com/ClickHouse/ClickHouse/pull/58278) ([LiuNeng](https://github.com/liuneng1994)). @@ -157,7 +157,7 @@ doc_type: 'changelog' * If stdout is a terminal and the output format is not specified, `clickhouse-client` and similar tools will use `PrettyCompact` by default, similarly to the interactive mode. `clickhouse-client` and `clickhouse-local` will handle command line arguments for input and output formats in a unified fashion. This closes [#61272](https://github.com/ClickHouse/ClickHouse/issues/61272). [#61800](https://github.com/ClickHouse/ClickHouse/pull/61800) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Underscore digit groups in Pretty formats for better readability. This is controlled by a new setting, `output_format_pretty_highlight_digit_groups`. [#61802](https://github.com/ClickHouse/ClickHouse/pull/61802) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release} +#### Bug fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release} * Fix bug with `intDiv` for decimal arguments [#59243](https://github.com/ClickHouse/ClickHouse/pull/59243) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). * Fix_kql_issue_found_by_wingfuzz [#59626](https://github.com/ClickHouse/ClickHouse/pull/59626) ([Yong Wang](https://github.com/kashwy)). diff --git a/docs/cloud/reference/01_changelog/02_release_notes/24_05.md b/docs/cloud/reference/01_changelog/02_release_notes/24_05.md index db9d36a3852..11fc6b7e009 100644 --- a/docs/cloud/reference/01_changelog/02_release_notes/24_05.md +++ b/docs/cloud/reference/01_changelog/02_release_notes/24_05.md @@ -8,7 +8,7 @@ sidebar_position: 9 doc_type: 'changelog' --- -# V24.5 changelog for Cloud +# V24.5 changelog for cloud Relevant changes for ClickHouse Cloud services based on the v24.5 release. @@ -74,7 +74,7 @@ Relevant changes for ClickHouse Cloud services based on the v24.5 release. * Added possibility to do cross join in temporary file if size exceeds limits. [#63432](https://github.com/ClickHouse/ClickHouse/pull/63432) (p1rattttt). -## Performance Improvements {#performance-improvements} +## Performance improvements {#performance-improvements} * Skip merging of newly created projection blocks during INSERT-s. [#59405](https://github.com/ClickHouse/ClickHouse/pull/59405) (Nikita Taranov). diff --git a/docs/cloud/reference/01_changelog/02_release_notes/24_06.md b/docs/cloud/reference/01_changelog/02_release_notes/24_06.md index 00a7d5d8228..1faf13d8578 100644 --- a/docs/cloud/reference/01_changelog/02_release_notes/24_06.md +++ b/docs/cloud/reference/01_changelog/02_release_notes/24_06.md @@ -8,7 +8,7 @@ sidebar_position: 8 doc_type: 'changelog' --- -# V24.6 changelog for Cloud +# V24.6 changelog for cloud Relevant changes for ClickHouse Cloud services based on the v24.6 release. @@ -34,7 +34,7 @@ Relevant changes for ClickHouse Cloud services based on the v24.6 release. * Introduced two additional columns in the `system.query_log`: `used_privileges` and `missing_privileges`. `used_privileges` is populated with the privileges that were checked during query execution, and `missing_privileges` contains required privileges that are missing. [#64597](https://github.com/ClickHouse/ClickHouse/pull/64597) ([Alexey Katsman](https://github.com/alexkats)). * Added a setting `output_format_pretty_display_footer_column_names` which when enabled displays column names at the end of the table for long tables (50 rows by default), with the threshold value for minimum number of rows controlled by `output_format_pretty_display_footer_column_names_min_rows`. [#65144](https://github.com/ClickHouse/ClickHouse/pull/65144) ([Shaun Struwig](https://github.com/Blargian)). -## Performance Improvement {#performance-improvement} +## Performance improvement {#performance-improvement} * Fix performance regression in cross join introduced in #60459 (24.5). #65243 (Nikita Taranov). * Improve io_uring resubmits visibility. Rename profile event IOUringSQEsResubmits -> IOUringSQEsResubmitsAsync and add a new one IOUringSQEsResubmitsSync. #63699 (Tomer Shafir). @@ -90,7 +90,7 @@ Relevant changes for ClickHouse Cloud services based on the v24.6 release. * Disable transactions for unsupported storages even for materialized views. [#64918](https://github.com/ClickHouse/ClickHouse/pull/64918) ([alesapin](https://github.com/alesapin)). * Forbid `QUALIFY` clause in the old analyzer. The old analyzer ignored `QUALIFY`, so it could lead to unexpected data removal in mutations. [#65356](https://github.com/ClickHouse/ClickHouse/pull/65356) ([Dmitry Novik](https://github.com/novikd)). -## Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release} +## Bug fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release} * Fixed 'set' skip index not working with IN and indexHint(). #62083 (Michael Kolupaev). * Fix queries with FINAL give wrong result when table does not use adaptive granularity. #62432 (Duc Canh Le). * Support executing function during assignment of parameterized view value. #63502 (SmitaRKulkarni). diff --git a/docs/cloud/reference/01_changelog/02_release_notes/24_12.md b/docs/cloud/reference/01_changelog/02_release_notes/24_12.md index f7d92ec22e0..a3acb2de38d 100644 --- a/docs/cloud/reference/01_changelog/02_release_notes/24_12.md +++ b/docs/cloud/reference/01_changelog/02_release_notes/24_12.md @@ -56,7 +56,7 @@ Relevant changes for ClickHouse Cloud services based on the v24.12 release. - A new setting, `http_response_headers` which allows you to customize the HTTP response headers. For example, you can tell the browser to render a picture that is stored in the database. This closes [#59620](https://github.com/ClickHouse/ClickHouse/issues/59620). [#72656](https://github.com/ClickHouse/ClickHouse/pull/72656) ([Alexey Milovidov](https://github.com/alexey-milovidov)). - Add function `fromUnixTimestamp64Second` which converts a Int64 Unix timestamp value to a DateTime64. [#73146](https://github.com/ClickHouse/ClickHouse/pull/73146) ([Robert Schulze](https://github.com/rschu1ze)). -## Performance Improvements {#performance-improvements} +## Performance improvements {#performance-improvements} - Add 2 new settings `short_circuit_function_evaluation_for_nulls` and `short_circuit_function_evaluation_for_nulls_threshold` that allow to execute functions over `Nullable` columns in short-circuit manner when the ratio of NULL values in the block of data exceeds the specified threshold. It means that the function will be executed only on rows with non-null values. It applies only to functions that return NULL value for rows where at least one argument is NULL. [#60129](https://github.com/ClickHouse/ClickHouse/pull/60129) ([李扬](https://github.com/taiyang-li)). - Memory usage of `clickhouse disks remove --recursive` is reduced for object storage disks. [#67323](https://github.com/ClickHouse/ClickHouse/pull/67323) ([Kirill](https://github.com/kirillgarbar)). @@ -143,7 +143,7 @@ Relevant changes for ClickHouse Cloud services based on the v24.12 release. - Added `JoinBuildTableRowCount/JoinProbeTableRowCount/JoinResultRowCount` profile events. [#72842](https://github.com/ClickHouse/ClickHouse/pull/72842) ([Vladimir Cherkasov](https://github.com/vdimir)). - Support Dynamic in functions toFloat64/touInt32/etc. [#72989](https://github.com/ClickHouse/ClickHouse/pull/72989) ([Pavel Kruglov](https://github.com/Avogar)). -## Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix} +## Bug fix (user-visible misbehavior in an official stable release) {#bug-fix} - The parts deduplicated during `ATTACH PART` query don't get stuck with the `attaching_` prefix anymore. [#65636](https://github.com/ClickHouse/ClickHouse/pull/65636) ([Kirill](https://github.com/kirillgarbar)). - Fix for the bug when dateTime64 losing precision for the `IN` function. [#67230](https://github.com/ClickHouse/ClickHouse/pull/67230) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). diff --git a/docs/cloud/reference/01_changelog/02_release_notes/25_04.md b/docs/cloud/reference/01_changelog/02_release_notes/25_04.md index e638051cd81..72943aefa40 100644 --- a/docs/cloud/reference/01_changelog/02_release_notes/25_04.md +++ b/docs/cloud/reference/01_changelog/02_release_notes/25_04.md @@ -365,7 +365,7 @@ doc_type: 'changelog' * Do not check parts on other disks for MergeTree if disk= is set. [#78855](https://github.com/ClickHouse/ClickHouse/pull/78855) ([Azat Khuzhin](https://github.com/azat)). * Make data types in `used_data_type_families` in `system.query_log` canonical. [#78972](https://github.com/ClickHouse/ClickHouse/pull/78972) ([Kseniia Sumarokova](https://github.com/kssenii)). -## Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix} +## Bug fix (user-visible misbehavior in an official stable release) {#bug-fix} * Fix cannot create SEQUENTIAL node with keeper-client. [#64177](https://github.com/ClickHouse/ClickHouse/pull/64177) ([Duc Canh Le](https://github.com/canhld94)). * Fix identifier resolution from parent scopes. Allow the use of aliases to expressions in the WITH clause. Fixes [#58994](https://github.com/ClickHouse/ClickHouse/issues/58994). Fixes [#62946](https://github.com/ClickHouse/ClickHouse/issues/62946). Fixes [#63239](https://github.com/ClickHouse/ClickHouse/issues/63239). Fixes [#65233](https://github.com/ClickHouse/ClickHouse/issues/65233). Fixes [#71659](https://github.com/ClickHouse/ClickHouse/issues/71659). Fixes [#71828](https://github.com/ClickHouse/ClickHouse/issues/71828). Fixes [#68749](https://github.com/ClickHouse/ClickHouse/issues/68749). [#66143](https://github.com/ClickHouse/ClickHouse/pull/66143) ([Dmitry Novik](https://github.com/novikd)). diff --git a/docs/cloud/reference/01_changelog/02_release_notes/25_08.md b/docs/cloud/reference/01_changelog/02_release_notes/25_08.md index bd169e54099..526f089f78d 100644 --- a/docs/cloud/reference/01_changelog/02_release_notes/25_08.md +++ b/docs/cloud/reference/01_changelog/02_release_notes/25_08.md @@ -75,7 +75,7 @@ doc_type: 'changelog' * Added a new system table [`system.codecs`](/operations/system-tables/codecs) to introspect the available codecs. (issue [#81525](https://github.com/ClickHouse/ClickHouse/issues/81525)). [#81600](https://github.com/ClickHouse/ClickHouse/pull/81600) ([Jimmy Aguilar Mena](https://github.com/Ergus)). * Introduction of `system.completions` table. Closes [#81889](https://github.com/ClickHouse/ClickHouse/issues/81889). [#83833](https://github.com/ClickHouse/ClickHouse/pull/83833) ([|2ustam](https://github.com/RuS2m)). -### Iceberg and DeltaLake {#iceberg-and-deltalake} +### Iceberg and deltalake {#iceberg-and-deltalake} * Support complex types in iceberg schema evolution. [#73714](https://github.com/ClickHouse/ClickHouse/pull/73714) ([scanhex12](https://github.com/scanhex12)). * Introduce Iceberg writes for `insert` queries. [#82692](https://github.com/ClickHouse/ClickHouse/pull/82692) ([scanhex12](https://github.com/scanhex12)). @@ -98,7 +98,7 @@ doc_type: 'changelog' * Added support for lightweight updates for `MergeTree`-family tables. Lightweight updates can be used by a new syntax: `UPDATE
SET col1 = val1, col2 = val2, ... WHERE `. Added implementation of lightweight deletes via lightweight updates which can be enabled by setting `lightweight_delete_mode = 'lightweight_update'`. [#82004](https://github.com/ClickHouse/ClickHouse/pull/82004) ([Anton Popov](https://github.com/CurtizJ)). * Support `_part_granule_offset` virtual column in MergeTree-family tables. This column indicates the zero-based index of the granule/mark each row belongs to within its data part. This addresses [#79572](https://github.com/ClickHouse/ClickHouse/issues/79572). [#82341](https://github.com/ClickHouse/ClickHouse/pull/82341) ([Amos Bird](https://github.com/amosbird)). -### Protocol and client Support {#protocol-and-client-support} +### Protocol and client support {#protocol-and-client-support} * Implement support for [ArrowFlight RPC](https://arrow.apache.org/docs/format/Flight.html) protocol by adding the [`arrowflight`](/engines/table-engines/integrations/arrowflight) table engine. [#74184](https://github.com/ClickHouse/ClickHouse/pull/74184) ([zakr600](https://github.com/zakr600)). * Add PostgreSQL protocol `COPY` command support. [#74344](https://github.com/ClickHouse/ClickHouse/pull/74344) ([scanhex12](https://github.com/scanhex12)). @@ -161,7 +161,7 @@ doc_type: 'changelog' * Calculate serialized key columnarly when group by multiple string or number columns. [#83884](https://github.com/ClickHouse/ClickHouse/pull/83884) ([李扬](https://github.com/taiyang-li)). * Implement `addManyDefaults` for `-If` combinators. [#83870](https://github.com/ClickHouse/ClickHouse/pull/83870) ([Raúl Marín](https://github.com/Algunenano)). -### JOIN optimizations {#join-optimizations} +### Join optimizations {#join-optimizations} * Add new setting [`min_joined_block_size_rows`](/operations/settings/settings#min_joined_block_size_rows) (analogous to `min_joined_block_size_bytes`; defaults to 65409) to control the minimum block size (in rows) for JOIN input and output blocks (if the join algorithm supports it). Small blocks will be squashed. [#81886](https://github.com/ClickHouse/ClickHouse/pull/81886) ([Nikita Taranov](https://github.com/nickitat)). * Performance of `HashJoin` optimised by removing the additional loop over hash maps in the typical case of only one key column, also `null_map` and `join_mask` checks are eliminated when they're always `true`/`false`. [#82308](https://github.com/ClickHouse/ClickHouse/pull/82308) ([Nikita Taranov](https://github.com/nickitat)). @@ -194,7 +194,7 @@ doc_type: 'changelog' * Optimize the generated plan for correlated subqueries by removing redundant `JOIN` operations using equivalence classes. If there are equivalent expressions for all correlated columns, `CROSS JOIN` is not produced if `query_plan_correlated_subqueries_use_substitution` setting is enabled. [#82435](https://github.com/ClickHouse/ClickHouse/pull/82435) ([Dmitry Novik](https://github.com/novikd)). * Read only required columns in correlated subquery when it appears to be an argument of function `EXISTS`. [#82443](https://github.com/ClickHouse/ClickHouse/pull/82443) ([Dmitry Novik](https://github.com/novikd)). -### Azure Blob Storage improvements {#azure-blob-storage-improvements} +### Azure blob storage improvements {#azure-blob-storage-improvements} * [`azureBlobStorage`](/engines/table-engines/integrations/azureBlobStorage) table engine now cachees and reuses managed identity authentication tokens when possible to avoid throttling. [#79860](https://github.com/ClickHouse/ClickHouse/pull/79860) ([Nick Blakely](https://github.com/niblak)). * Replace curl HTTP client with poco HTTP client for azure blob storage. Introduced multiple settings for these clients which mirror settings from S3. Introduced aggressive connect timeouts for both Azure and S3. Improved introspection into Azure profile events and metrics. New client is enabled by default, provide much better latencies for cold queries on top of Azure Blob Storage. Old `Curl` client can be returned back by setting `azure_sdk_use_native_client=false`. [#83294](https://github.com/ClickHouse/ClickHouse/pull/83294) ([alesapin](https://github.com/alesapin)). @@ -288,7 +288,7 @@ doc_type: 'changelog' * Allow asynchronously iterating objects from Iceberg table without storing objects for each data file explicitly. [#85369](https://github.com/ClickHouse/ClickHouse/pull/85369) ([Daniil Ivanik](https://github.com/divanik)). * Support Iceberg equality deletes. [#85843](https://github.com/ClickHouse/ClickHouse/pull/85843) ([Han Fei](https://github.com/hanfei1991)). -### DeltaLake table engine {#deltalake-table-engine} +### Deltalake table engine {#deltalake-table-engine} * Improve `DeltaLake` table engine: delta-kernel-rs has `ExpressionVisitor` API which is implemented in this PR and is applied to partition column expressions transform (it will replace an old deprecated within the delta-kernel-rs way, which was used before in our code). In the future this `ExpressionVisitor` will also allow to implement statistics based pruning and some delta-lake proprietary features. Additionally the purpose of this change is to support partition pruning in `DeltaLakeCluster` table engine (the result of a parsed expression - ActionsDAG - will be serialized and sent from the initiator along with the data path, because this kind of information, which is needed for pruning, is only available as meta information on data files listing, which is done by initiator only, but it has to be applied to data on each reading server). [#81136](https://github.com/ClickHouse/ClickHouse/pull/81136) ([Kseniia Sumarokova](https://github.com/kssenii)). * Fix partition pruning with data lake cluster functions. [#82131](https://github.com/ClickHouse/ClickHouse/pull/82131) ([Kseniia Sumarokova](https://github.com/kssenii)). @@ -315,7 +315,7 @@ doc_type: 'changelog' * Allow to use any storage policy (i.e. object storage, such as S3) for external aggregation/sorting. [#84734](https://github.com/ClickHouse/ClickHouse/pull/84734) ([Azat Khuzhin](https://github.com/azat)). * Collect all removed objects to execute single object storage remove operation. [#85316](https://github.com/ClickHouse/ClickHouse/pull/85316) ([Mikhail Artemenko](https://github.com/Michicosun)). -### S3Queue table engine {#s3queue-table-engine} +### S3queue table engine {#s3queue-table-engine} * Macros like `{uuid}` can now be used in the `keeper_path` setting of the S3Queue table engine. [#82463](https://github.com/ClickHouse/ClickHouse/pull/82463) ([Nikolay Degterinsky](https://github.com/evillique)). * Add a new server setting `s3queue_disable_streaming` which disables streaming in tables with S3Queue table engine. This setting is changeable without server restart. [#82515](https://github.com/ClickHouse/ClickHouse/pull/82515) ([Kseniia Sumarokova](https://github.com/kssenii)). @@ -330,7 +330,7 @@ doc_type: 'changelog' * Count consumed messages manually to avoid depending on previous committed offset in StorageKafka2. [#81662](https://github.com/ClickHouse/ClickHouse/pull/81662) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). * Integrate `StorageKafka2` to [`system.kafka_consumers`](/operations/system-tables/kafka_consumers). [#82652](https://github.com/ClickHouse/ClickHouse/pull/82652) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). -### ClickHouse Keeper improvements {#clickhouse-keeper-improvements} +### ClickHouse keeper improvements {#clickhouse-keeper-improvements} * Keeper improvement: move changelog files between disk in a background thread. Previously, moving changelog to a different disk would block Keeper globally until the move is finished. This lead to performance degradation if moving is a long operation (e.g. to S3 disk). [#82485](https://github.com/ClickHouse/ClickHouse/pull/82485) ([Antonio Andelic](https://github.com/antonio2368)). * Keeper improvement: add new config `keeper_server.cleanup_old_and_ignore_new_acl`. If enabled, all nodes will have their ACLs cleared while ACL for new requests will be ignored. If the goal is to completely remove ACL from nodes, it's important to leave the config enabled until a new snapshot is created. [#82496](https://github.com/ClickHouse/ClickHouse/pull/82496) ([Antonio Andelic](https://github.com/antonio2368)). @@ -345,7 +345,7 @@ doc_type: 'changelog' * The `encrypt_decrypt` tool now supports encrypted ZooKeeper connections. [#84764](https://github.com/ClickHouse/ClickHouse/pull/84764) ([Roman Vasin](https://github.com/rvasin)). * Limit Keeper log entry cache size by number of entries using `keeper_server.coordination_settings.latest_logs_cache_entry_count_threshold` and `keeper_server.coordination_settings.commit_logs_cache_entry_count_threshold`. [#84877](https://github.com/ClickHouse/ClickHouse/pull/84877) ([Antonio Andelic](https://github.com/antonio2368)). -### JSON and Dynamic types {#json-and-dynamic-types} +### JSON and dynamic types {#json-and-dynamic-types} * Add `columns_substreams.txt` file to Wide part to track all substreams stored in the part. It helps to track dynamic streams in JSON and Dynamic types and so avoid reading sample of these columns to get the list of dynamic streams (for example for columns sizes calculation). Also now all dynamic streams are reflected in `system.parts_columns`. [#81091](https://github.com/ClickHouse/ClickHouse/pull/81091) ([Pavel Kruglov](https://github.com/Avogar)). * Allow `ALTER UPDATE` in JSON and Dynamic columns. [#82419](https://github.com/ClickHouse/ClickHouse/pull/82419) ([Pavel Kruglov](https://github.com/Avogar)). @@ -410,7 +410,7 @@ doc_type: 'changelog' ### Replication and synchronization {#replication-and-synchronization} -### SystemAndInternalImprovements {#systemandinternalimprovements} +### Systemandinternalimprovements {#systemandinternalimprovements} * Improve `SYSTEM RESTART REPLICA` to retry table creation when ZooKeeper connection issues occur, preventing tables from being forgotten. [#82616](https://github.com/ClickHouse/ClickHouse/pull/82616) ([Nikolay Degterinsky](https://github.com/evillique)). * Add UUID validation in `ReplicatedMergeTree::executeMetadataAlter` to prevent incorrect table definitions when tables are exchanged between getting the StorageID and calling `IDatabase::alterTable`. [#82666](https://github.com/ClickHouse/ClickHouse/pull/82666) ([Nikolay Degterinsky](https://github.com/evillique)). @@ -529,7 +529,7 @@ doc_type: 'changelog' * Fix `Not-ready Set` for `IN (subquery)` inside `additional_table_filters expression` setting. [#85210](https://github.com/ClickHouse/ClickHouse/pull/85210) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix logical error with duplicate subqueries when `optimize_syntax_fuse_functions` is enabled, close [#75511](https://github.com/ClickHouse/ClickHouse/issues/75511). [#83300](https://github.com/ClickHouse/ClickHouse/pull/83300) ([Vladimir Cherkasov](https://github.com/vdimir)). -### Iceberg and DataLake fixes {#iceberg-and-datalake-fixes} +### Iceberg and datalake fixes {#iceberg-and-datalake-fixes} * Fix the metadata resolution when querying iceberg tables through rest catalog. ... [#80562](https://github.com/ClickHouse/ClickHouse/pull/80562) ([Saurabh Kumar Ojha](https://github.com/saurabhojha)). * Fix data races in Iceberg. [#82088](https://github.com/ClickHouse/ClickHouse/pull/82088) ([Azat Khuzhin](https://github.com/azat)). @@ -548,7 +548,7 @@ doc_type: 'changelog' * Fix the metadata resolution when querying iceberg tables through rest catalog. ... [#85531](https://github.com/ClickHouse/ClickHouse/pull/85531) ([Saurabh Kumar Ojha](https://github.com/saurabhojha)). * Fix secrets masking in icebergS3Cluster and icebergAzureCluster table functions. [#85658](https://github.com/ClickHouse/ClickHouse/pull/85658) ([MikhailBurdukov](https://github.com/MikhailBurdukov)). -### DeltaLake fixes {#deltalake-fixes} +### Deltalake fixes {#deltalake-fixes} * Fix column pruning with delta-kernel in storage `DeltaLake`. Closes [#84543](https://github.com/ClickHouse/ClickHouse/issues/84543). [#84745](https://github.com/ClickHouse/ClickHouse/pull/84745) ([Kseniia Sumarokova](https://github.com/kssenii)). * Refresh credentials in delta-kernel in storage DeltaLake. [#84751](https://github.com/ClickHouse/ClickHouse/pull/84751) ([Kseniia Sumarokova](https://github.com/kssenii)). @@ -654,7 +654,7 @@ doc_type: 'changelog' * Fix logical error from S3Queue "Table is already registered". Closes [#84433](https://github.com/ClickHouse/ClickHouse/issues/84433). Broken after https://github.com/ClickHouse/ClickHouse/pull/83530. [#84677](https://github.com/ClickHouse/ClickHouse/pull/84677) ([Kseniia Sumarokova](https://github.com/kssenii)). * Fix large setting values breaking S3Queue tables and replica restart. [#86074](https://github.com/ClickHouse/ClickHouse/pull/86074) ([Nikolay Degterinsky](https://github.com/evillique)). -### DynamicAndVariantTypeFixes {#dynamicandvarianttypefixes} +### Dynamicandvarianttypefixes {#dynamicandvarianttypefixes} * Fix rollback of Dynamic column on parsing failure. [#82169](https://github.com/ClickHouse/ClickHouse/pull/82169) ([Pavel Kruglov](https://github.com/Avogar)). * Fix possible crash in Variant type in UNION. [#83295](https://github.com/ClickHouse/ClickHouse/pull/83295) ([Pavel Kruglov](https://github.com/Avogar)). * Fix reading Variant column with lazy materialization. [#84400](https://github.com/ClickHouse/ClickHouse/pull/84400) ([Pavel Kruglov](https://github.com/Avogar)). diff --git a/docs/cloud/reference/01_changelog/02_release_notes/25_10.md b/docs/cloud/reference/01_changelog/02_release_notes/25_10.md index 8cdc6659dd1..6a3cfef9230 100644 --- a/docs/cloud/reference/01_changelog/02_release_notes/25_10.md +++ b/docs/cloud/reference/01_changelog/02_release_notes/25_10.md @@ -122,7 +122,7 @@ doc_type: 'changelog' * Improve pre where optimization for conditions like `func(primary_column) = 'xx'` and `column in (xxx)`. [#85529](https://github.com/ClickHouse/ClickHouse/pull/85529) ([李扬](https://github.com/taiyang-li)). * Avoid full scan for `system.tables` with filter by `uuid` (Can be useful if you have only UUID from logs or zookeeper path). [#88379](https://github.com/ClickHouse/ClickHouse/pull/88379) ([Azat Khuzhin](https://github.com/azat)). -### JOIN optimizations {#join-optimizations} +### Join optimizations {#join-optimizations} * Provides a logic regarding pushing down the disjunction JOIN predicates. Example: in TPC-H Q7 for a condition on 2 tables n1 and n2 like `(n1.n_name = 'FRANCE' AND n2.n_name = 'GERMANY') OR (n1.n_name = 'GERMANY' AND n2.n_name = 'FRANCE')` we extract separate partial filters for each table `n1.n_name = 'FRANCE' OR n1.n_name = 'GERMANY'` for n1 and `n2.n_name = 'GERMANY' OR n2.n_name = 'FRANCE'` for n2. [#84735](https://github.com/ClickHouse/ClickHouse/pull/84735) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). * Implemented rewriting of JOIN: 1. Convert `LEFT ANY JOIN` and `RIGHT ANY JOIN` to `SEMI`/`ANTI` JOIN if the filter condition is always false for matched or non-matched rows. This optimization is controlled by a new setting `query_plan_convert_any_join_to_semi_or_anti_join`. 2. Convert `FULL ALL JOIN` to `LEFT ALL` or `RIGHT ALL` JOIN if the filter condition is always false for non-matched rows from one side. [#86028](https://github.com/ClickHouse/ClickHouse/pull/86028) ([Dmitry Novik](https://github.com/novikd)). @@ -147,7 +147,7 @@ doc_type: 'changelog' * Enable saving marks in cache and avoid direct IO for the MergeTreeLazy reader. [#87989](https://github.com/ClickHouse/ClickHouse/pull/87989) ([Nikita Taranov](https://github.com/nickitat)). * SELECT query with `FINAL` clause on.a `ReplacingMergeTree` table with the `is_deleted` column now executes faster because of improved parallelization from 2 existing optimizations : 1) `do_not_merge_across_partitions_select_final` optimization for partitions of the table that have only a single `part` 2) Split other selected ranges of the table into `intersecting / non-intersecting` and only intersecting ranges have to pass through FINAL merging transform. [#88090](https://github.com/ClickHouse/ClickHouse/pull/88090) ([Shankar Iyer](https://github.com/shankar-iyer)). -### Aggregation and GROUP BY optimizations {#aggregation-and-group-by-optimizations} +### Aggregation and group by optimizations {#aggregation-and-group-by-optimizations} * Fix performance degradation caused by a large serialized key while grouping by multiple string/number columns. Close https://github.com/ClickHouse/ClickHouse/pull/83884#issuecomment-3187972297 cc @mkmkme . It is a follow-up of https://github.com/ClickHouse/ClickHouse/pull/83884. [#85924](https://github.com/ClickHouse/ClickHouse/pull/85924) ([李扬](https://github.com/taiyang-li)). * RadixSort: Help the compiler use SIMD and the CPU do better prefetching. Uses dynamic dispatch to use software prefetching with Intel CPUs only. Continues the work by @taiyang-li in https://github.com/ClickHouse/ClickHouse/pull/77029. [#86378](https://github.com/ClickHouse/ClickHouse/pull/86378) ([Raúl Marín](https://github.com/Algunenano)). diff --git a/docs/cloud/reference/03_billing/01_billing_overview.md b/docs/cloud/reference/03_billing/01_billing_overview.md index 65bf2c55fba..1e978700a3f 100644 --- a/docs/cloud/reference/03_billing/01_billing_overview.md +++ b/docs/cloud/reference/03_billing/01_billing_overview.md @@ -11,7 +11,7 @@ For pricing information, see the [ClickHouse Cloud Pricing](https://clickhouse.c ClickHouse Cloud bills based on the usage of compute, storage, [data transfer](/cloud/manage/network-data-transfer) (egress over the internet and cross-region), and [ClickPipes](/integrations/clickpipes). To understand what can affect your bill, and ways that you can manage your spend, keep reading. -## Amazon Web Services (AWS) example {#amazon-web-services-aws-example} +## Amazon web services (AWS) example {#amazon-web-services-aws-example} :::note - Prices reflect AWS us-east-1 pricing. @@ -125,7 +125,7 @@ Pricing breakdown for this example:
-### Enterprise: Starting prices vary {#enterprise-starting-prices-vary} +### Enterprise: starting prices vary {#enterprise-starting-prices-vary} Best for: large scale, mission critical deployments that have stringent security and compliance needs @@ -177,7 +177,7 @@ Best for: large scale, mission critical deployments that have stringent security ## Frequently asked questions {#faqs} -### What is a ClickHouse Credit (CHC)? {#what-is-chc} +### What is a ClickHouse credit (chc)? {#what-is-chc} A ClickHouse Credit is a unit of credit toward Customer's usage of ClickHouse Cloud equal to one (1) US dollar, to be applied based on ClickHouse's then-current published price list. @@ -185,7 +185,7 @@ A ClickHouse Credit is a unit of credit toward Customer's usage of ClickHouse Cl If you are being billed through Stripe then you will see that 1 CHC is equal to \$0.01 USD on your Stripe invoice. This is to allow accurate billing on Stripe due to their limitation on not being able to bill fractional quantities of our standard SKU of 1 CHC = \$1 USD. ::: -### Where can I find legacy pricing? {#find-legacy-pricing} +### Where can i find legacy pricing? {#find-legacy-pricing} Legacy pricing information can be found [here](https://clickhouse.com/pricing?legacy=true). @@ -205,7 +205,7 @@ Storage and backups are counted towards storage costs and billed separately. All services will default to one backup, retained for a day. Users who need additional backups can do so by configuring additional [backups](/cloud/manage/backups/overview) under the settings tab of the Cloud console. -### How do I estimate compression? {#how-do-i-estimate-compression} +### How do i estimate compression? {#how-do-i-estimate-compression} Compression can vary from dataset to dataset. How much it varies is dependent on how compressible the data is in the first place (number of high vs. low cardinality fields), @@ -222,7 +222,7 @@ FROM system.tables WHERE name = ``` -### What tools does ClickHouse offer to estimate the cost of running a service in the cloud if I have a self-managed deployment? {#what-tools-does-clickhouse-offer-to-estimate-the-cost-of-running-a-service-in-the-cloud-if-i-have-a-self-managed-deployment} +### What tools does ClickHouse offer to estimate the cost of running a service in the cloud if i have a Self-managed deployment? {#what-tools-does-clickhouse-offer-to-estimate-the-cost-of-running-a-service-in-the-cloud-if-i-have-a-self-managed-deployment} The ClickHouse query log captures [key metrics](/operations/system-tables/query_log) that can be used to estimate the cost of running a workload in ClickHouse Cloud. For details on migrating from self-managed to ClickHouse Cloud please refer to the [migration documentation](/cloud/migration/clickhouse-to-cloud), and contact [ClickHouse Cloud support](https://console.clickhouse.cloud/support) if you have further questions. @@ -239,45 +239,45 @@ ClickHouse Cloud supports the following billing options: ClickHouse Cloud credits for PAYG are invoiced in \$0.01 units, allowing us to charge customers for partial ClickHouse credits based on their usage. This differs from committed spend ClickHouse credits, which are purchased in advance in whole \$1 units. ::: -### Can I delete my credit card? {#can-i-delete-my-credit-card} +### Can i delete my credit card? {#can-i-delete-my-credit-card} You can’t remove a credit card in the Billing UI, but you can update it anytime. This helps ensure your organization always has a valid payment method. If you need to remove your credit card, please contact [ClickHouse Cloud support](https://console.clickhouse.cloud/support) for help. ### How long is the billing cycle? {#how-long-is-the-billing-cycle} Billing follows a monthly billing cycle and the start date is tracked as the date when the ClickHouse Cloud organization was created. -### If I have an active PAYG marketplace subscription and then sign a committed contract, will my committed credits be consumed first? {#committed-credits-consumed-first-with-active-payg-subscription} +### If i have an active payg marketplace subscription and then sign a committed contract, will my committed credits be consumed first? {#committed-credits-consumed-first-with-active-payg-subscription} Yes. Usage is consumed with the following payment methods in this order: - Committed (prepaid) credits - Marketplace subscription (PAYG) - Credit card -### What controls does ClickHouse Cloud offer to manage costs for Scale and Enterprise services? {#what-controls-does-clickhouse-cloud-offer-to-manage-costs-for-scale-and-enterprise-services} +### What controls does ClickHouse Cloud offer to manage costs for scale and enterprise services? {#what-controls-does-clickhouse-cloud-offer-to-manage-costs-for-scale-and-enterprise-services} - Trial and Annual Commit customers are notified automatically by email when their consumption hits certain thresholds: `50%`, `75%`, and `90%`. This allows users to proactively manage their usage. - ClickHouse Cloud allows users to set a maximum auto-scaling limit on their compute via [Advanced scaling control](/manage/scaling), a significant cost factor for analytical workloads. - The [Advanced scaling control](/manage/scaling) lets you set memory limits with an option to control the behavior of pausing/idling during inactivity. -### What controls does ClickHouse Cloud offer to manage costs for Basic services? {#what-controls-does-clickhouse-cloud-offer-to-manage-costs-for-basic-services} +### What controls does ClickHouse Cloud offer to manage costs for basic services? {#what-controls-does-clickhouse-cloud-offer-to-manage-costs-for-basic-services} - The [Advanced scaling control](/manage/scaling) lets you control the behavior of pausing/idling during inactivity. Adjusting memory allocation is not supported for Basic services. - Note that the default setting pauses the service after a period of inactivity. -### If I have multiple services, do I get an invoice per service or a consolidated invoice? {#if-i-have-multiple-services-do-i-get-an-invoice-per-service-or-a-consolidated-invoice} +### If i have multiple services, do i get an invoice per service or a consolidated invoice? {#if-i-have-multiple-services-do-i-get-an-invoice-per-service-or-a-consolidated-invoice} A consolidated invoice is generated for all services in a given organization for a billing period. -### If I add my credit card and upgrade before my trial period and credits expire, will I be charged? {#if-i-add-my-credit-card-and-upgrade-before-my-trial-period-and-credits-expire-will-i-be-charged} +### If i add my credit card and upgrade before my trial period and credits expire, will i be charged? {#if-i-add-my-credit-card-and-upgrade-before-my-trial-period-and-credits-expire-will-i-be-charged} When a user converts from trial to paid before the 30-day trial period ends, but with credits remaining from the trial credit allowance, we continue to draw down from the trial credits during the initial 30-day trial period, and then charge the credit card. -### How can I keep track of my spending? {#how-can-i-keep-track-of-my-spending} +### How can i keep track of my spending? {#how-can-i-keep-track-of-my-spending} The ClickHouse Cloud console provides a Usage display that details usage per service. This breakdown, organized by usage dimensions, helps you understand the cost associated with each metered unit. -### How do I access my invoices for my subscription to the ClickHouse Cloud service? {#how-do-i-access-my-invoice-for-my-subscription-to-the-clickhouse-cloud-service} +### How do i access my invoices for my subscription to the ClickHouse Cloud service? {#how-do-i-access-my-invoice-for-my-subscription-to-the-clickhouse-cloud-service} For direct subscriptions using a credit card: @@ -287,7 +287,7 @@ For subscriptions through a cloud marketplace: All marketplace subscriptions are billed and invoiced by the marketplace. You can view your invoice through the respective cloud provider marketplace directly. -### Why do the dates on the Usage statements not match my Marketplace Invoice? {#why-do-the-dates-on-the-usage-statements-not-match-my-marketplace-invoice} +### Why do the dates on the usage statements not match my marketplace invoice? {#why-do-the-dates-on-the-usage-statements-not-match-my-marketplace-invoice} AWS Marketplace billing follows the calendar month cycle. For example, for usage between dates 01-Dec-2024 and 01-Jan-2025, diff --git a/docs/cloud/reference/03_billing/02_marketplace/aws-marketplace-committed.md b/docs/cloud/reference/03_billing/02_marketplace/aws-marketplace-committed.md index de3546cb79f..5fe2b259062 100644 --- a/docs/cloud/reference/03_billing/02_marketplace/aws-marketplace-committed.md +++ b/docs/cloud/reference/03_billing/02_marketplace/aws-marketplace-committed.md @@ -58,7 +58,7 @@ Without completing this step, we will not be able to link your AWS Marketplace c AWS Marketplace private offer email -### Login to Cloud {#login-cloud} +### Login to cloud {#login-cloud} Once you redirect to ClickHouse Cloud, you can either login with an existing account, or register with a new account. This step is necessary so that we can bind your ClickHouse Cloud organization to AWS Marketplace billing. diff --git a/docs/cloud/reference/03_billing/02_marketplace/aws-marketplace-payg.md b/docs/cloud/reference/03_billing/02_marketplace/aws-marketplace-payg.md index f66b920bcac..6f12406fc2c 100644 --- a/docs/cloud/reference/03_billing/02_marketplace/aws-marketplace-payg.md +++ b/docs/cloud/reference/03_billing/02_marketplace/aws-marketplace-payg.md @@ -33,7 +33,7 @@ One AWS account can only subscribe to one “ClickHouse Cloud - Pay As You Go” -### Search for Clickhouse Cloud - Pay As You Go {#search-payg} +### Search for ClickHouse Cloud - pay as you Go {#search-payg} Go to the [AWS Marketplace](https://aws.amazon.com/marketplace) and search for “ClickHouse Cloud - Pay As You Go”. @@ -87,7 +87,7 @@ You will also need to provide some basic information about your business. See th If you are an existing ClickHouse Cloud user, simply log in using your credentials. -### Add the Marketplace Subscription to an Organization {#add-marketplace-subscription} +### Add the marketplace subscription to an organization {#add-marketplace-subscription} After successfully logging in, you can decide whether to create a new organization to bill to this marketplace subscription or choose an existing organization to bill to this subscription. diff --git a/docs/cloud/reference/03_billing/02_marketplace/migrate-marketplace-payg-committed.md b/docs/cloud/reference/03_billing/02_marketplace/migrate-marketplace-payg-committed.md index 8c181037619..f42db2667b8 100644 --- a/docs/cloud/reference/03_billing/02_marketplace/migrate-marketplace-payg-committed.md +++ b/docs/cloud/reference/03_billing/02_marketplace/migrate-marketplace-payg-committed.md @@ -6,11 +6,11 @@ keywords: ['marketplace', 'billing', 'PAYG', 'pay-as-you-go', 'committed spend c doc_type: 'guide' --- -# Migrate billing from pay-as-you-go (PAYG) to a committed spend contract in a cloud marketplace {#migrate-payg-to-committed} +# Migrate billing from pay-as-you-go (payg) to a committed spend contract in a cloud marketplace {#migrate-payg-to-committed} If your ClickHouse organization is currently billed through an active cloud marketplace pay-as-you-go (PAYG) subscription (or order) and you wish to migrate to billing via a committed spend contract through the same cloud marketplace, please accept your new offer and then follow the steps below based on your cloud service provider. -## Important Notes {#important-notes} +## Important notes {#important-notes} Please note that canceling your marketplace PAYG subscription does not delete your ClickHouse Cloud account - only the billing relationship via the marketplace. Once canceled, our system will stop billing for ClickHouse Cloud services through the marketplace. (Note: this process is not immediate and may take a few minutes to complete). @@ -22,13 +22,13 @@ Any usage accrued after the subscription is canceled will be billed to the next For any questions or support with issues configuring your organization to a new marketplace subscription please reach out to ClickHouse [support](https://clickhouse.com/support/program) for help. -## AWS Marketplace {#aws-marketplace} +## AWS marketplace {#aws-marketplace} If you want to use the same AWS Account ID for migrating your PAYG subscription to a committed spend contract then our recommended method is to [contact sales](https://clickhouse.com/company/contact) to make this amendment. Doing so means no additional steps are needed and no disruption to your ClickHouse organization or services will occur. If you want to use a different AWS Account ID for migrating your ClickHouse organization from a PAYG subscription to a committed spend contract then follow these steps: -### Steps to Cancel AWS PAYG Subscription {#cancel-aws-payg} +### Steps to cancel AWS payg subscription {#cancel-aws-payg} 1. **Go to the [AWS Marketplace](https://us-east-1.console.aws.amazon.com/marketplace)** 2. **Click on the "Manage Subscriptions" button** @@ -44,9 +44,9 @@ If you want to use a different AWS Account ID for migrating your ClickHouse orga Next follow these [steps](/cloud/billing/marketplace/aws-marketplace-committed-contract) to configure your ClickHouse organization to the new AWS committed spend contract you accepted. -## GCP Marketplace {#gcp-marketplace} +## GCP marketplace {#gcp-marketplace} -### Steps to Cancel GCP PAYG Order {#cancel-gcp-payg} +### Steps to cancel GCP payg order {#cancel-gcp-payg} 1. **Go to your [Google Cloud Marketplace Console](https://console.cloud.google.com/marketplace):** - Make sure you are logged in to the correct GCP account and have selected the appropriate project @@ -60,9 +60,9 @@ Next follow these [steps](/cloud/billing/marketplace/aws-marketplace-committed-c Next follow these [steps](/cloud/billing/marketplace/gcp-marketplace-committed-contract) to configure your ClickHouse organization to your new GCP committed spend contract. -## Azure Marketplace {#azure-marketplace} +## Azure marketplace {#azure-marketplace} -### Steps to Cancel Azure PAYG Subscription {#cancel-azure-payg} +### Steps to cancel Azure payg subscription {#cancel-azure-payg} 1. **Go to the [Microsoft Azure Portal](http://portal.azure.com)** 2. **Navigate to "Subscriptions"** @@ -75,7 +75,7 @@ Next follow these [steps](/cloud/billing/marketplace/gcp-marketplace-committed-c Next, follow these [steps](/cloud/billing/marketplace/azure-marketplace-committed-contract) to configure your ClickHouse organization to your new Azure committed spend contract. -## Requirements for Linking to Committed Spend Contract {#linking-requirements} +## Requirements for linking to committed spend contract {#linking-requirements} > **Note:** In order to link your organization to a marketplace committed spend contract: > - The user following the steps must be an admin user of the ClickHouse organization you are attaching the subscription to diff --git a/docs/cloud/reference/03_billing/02_marketplace/overview.md b/docs/cloud/reference/03_billing/02_marketplace/overview.md index 578c6f7e428..ff3bd6f39a5 100644 --- a/docs/cloud/reference/03_billing/02_marketplace/overview.md +++ b/docs/cloud/reference/03_billing/02_marketplace/overview.md @@ -20,13 +20,13 @@ You can either use pay-as-you-go (PAYG) or commit to a contract with ClickHouse - [Azure Marketplace PAYG](/cloud/billing/marketplace/azure-marketplace-payg) - [Azure Marketplace Committed Contract](/cloud/billing/marketplace/azure-marketplace-committed-contract) -## FAQs {#faqs} +## Faqs {#faqs} -### How can I verify that my organization is connected to marketplace billing?​ {#how-can-i-verify-that-my-organization-is-connected-to-marketplace-billing} +### How can i verify that my organization is connected to marketplace billing?​ {#how-can-i-verify-that-my-organization-is-connected-to-marketplace-billing} In the ClickHouse Cloud console, navigate to **Billing**. You should see the name of the marketplace and the link in the **Payment details** section. -### I am an existing ClickHouse Cloud user. What happens when I subscribe to ClickHouse Cloud via AWS / GCP / Azure marketplace?​ {#i-am-an-existing-clickhouse-cloud-user-what-happens-when-i-subscribe-to-clickhouse-cloud-via-aws--gcp--azure-marketplace} +### I am an existing ClickHouse Cloud user. what happens when i subscribe to ClickHouse Cloud via AWS / GCP / Azure marketplace?​ {#i-am-an-existing-clickhouse-cloud-user-what-happens-when-i-subscribe-to-clickhouse-cloud-via-aws--gcp--azure-marketplace} Signing up for ClickHouse Cloud from the cloud provider marketplace is a two step process: 1. You first "subscribe" to ClickHouse Cloud on the cloud providers' marketplace portal. After you have finished subscribing, you click on "Pay Now" or "Manage on Provider" (depending on the marketplace). This redirects you to ClickHouse Cloud. @@ -36,7 +36,7 @@ NOTE: Your existing services and organizations from any prior ClickHouse Cloud s You can switch between organizations from the bottom left menu of the ClickHouse Cloud console. -### I am an existing ClickHouse Cloud user. What should I do if I want my existing services to be billed via marketplace?​ {#i-am-an-existing-clickhouse-cloud-user-what-should-i-do-if-i-want-my-existing-services-to-be-billed-via-marketplace} +### I am an existing ClickHouse Cloud user. what should i do if i want my existing services to be billed via marketplace?​ {#i-am-an-existing-clickhouse-cloud-user-what-should-i-do-if-i-want-my-existing-services-to-be-billed-via-marketplace} You will need to subscribe to ClickHouse Cloud via the cloud provider marketplace. Once you finish subscribing on the marketplace, and redirect to ClickHouse Cloud you will have the option of linking an existing ClickHouse Cloud organization to marketplace billing. From that point on, your existing resources will now get billed via the marketplace. @@ -48,13 +48,13 @@ You can confirm from the organization's billing page that billing is indeed now Your existing services and organizations from any prior ClickHouse Cloud signups will remain and not be connected to the marketplace billing. ::: -### I subscribed to ClickHouse Cloud as a marketplace user. How can I unsubscribe?​ {#i-subscribed-to-clickhouse-cloud-as-a-marketplace-user-how-can-i-unsubscribe} +### I subscribed to ClickHouse Cloud as a marketplace user. how can i unsubscribe?​ {#i-subscribed-to-clickhouse-cloud-as-a-marketplace-user-how-can-i-unsubscribe} Note that you can simply stop using ClickHouse Cloud and delete all existing ClickHouse Cloud services. Even though the subscription will still be active, you will not be paying anything as ClickHouse Cloud doesn't have any recurring fees. If you want to unsubscribe, please navigate to the Cloud Provider console and cancel the subscription renewal there. Once the subscription ends, all existing services will be stopped and you will be prompted to add a credit card. If no card was added, after two weeks all existing services will be deleted. -### I subscribed to ClickHouse Cloud as a marketplace user, and then unsubscribed. Now I want to subscribe back, what is the process?​ {#i-subscribed-to-clickhouse-cloud-as-a-marketplace-user-and-then-unsubscribed-now-i-want-to-subscribe-back-what-is-the-process} +### I subscribed to ClickHouse Cloud as a marketplace user, and then unsubscribed. now i want to subscribe back, what is the process?​ {#i-subscribed-to-clickhouse-cloud-as-a-marketplace-user-and-then-unsubscribed-now-i-want-to-subscribe-back-what-is-the-process} In that case please subscribe to the ClickHouse Cloud as usual (see sections on subscribing to ClickHouse Cloud via the marketplace). @@ -63,12 +63,12 @@ In that case please subscribe to the ClickHouse Cloud as usual (see sections on If you have any trouble with reactivating your marketplace org, please contact [ClickHouse Cloud Support](https://clickhouse.com/support/program). -### How do I access my invoice for my marketplace subscription to the ClickHouse Cloud service?​ {#how-do-i-access-my-invoice-for-my-marketplace-subscription-to-the-clickhouse-cloud-service} +### How do i access my invoice for my marketplace subscription to the ClickHouse Cloud service?​ {#how-do-i-access-my-invoice-for-my-marketplace-subscription-to-the-clickhouse-cloud-service} - [AWS billing Console](https://us-east-1.console.aws.amazon.com/billing/home) - [GCP Marketplace orders](https://console.cloud.google.com/marketplace/orders) (select the billing account that you used for subscription) -### Why do the dates on the Usage statements not match my Marketplace Invoice?​ {#why-do-the-dates-on-the-usage-statements-not-match-my-marketplace-invoice} +### Why do the dates on the usage statements not match my marketplace invoice?​ {#why-do-the-dates-on-the-usage-statements-not-match-my-marketplace-invoice} Marketplace billing follows the calendar month cycle. For example, for usage between December 1st and January 1st, an invoice will be generated between January 3rd and January 5th. @@ -76,7 +76,7 @@ ClickHouse Cloud usage statements follow a different billing cycle where usage i The usage and invoice dates will differ if these dates are not the same. Since usage statements track usage by day for a given service, users can rely on statements to see the breakdown of costs. -### Where can I find general billing information​? {#where-can-i-find-general-billing-information} +### Where can i find general billing information​? {#where-can-i-find-general-billing-information} Please see the [Billing overview page](/cloud/manage/billing). @@ -84,10 +84,10 @@ Please see the [Billing overview page](/cloud/manage/billing). There is no difference in pricing between marketplace billing and signing up directly with ClickHouse. In either case, your usage of ClickHouse Cloud is tracked in terms of ClickHouse Cloud Credits (CHCs), which are metered in the same way and billed accordingly. -### Can I set up multiple ClickHouse Organizations to bill to a single cloud marketplace billing account or sub account (AWS, GCP, or Azure)? {#multiple-organizations-to-bill-to-single-cloud-marketplace-account} +### Can i set up multiple ClickHouse organizations to bill to a single cloud marketplace billing account or sub account (AWS, GCP, or Azure)? {#multiple-organizations-to-bill-to-single-cloud-marketplace-account} A single ClickHouse organization can only be configured to bill to a single Cloud marketplace billing account or sub account. -### If my ClickHouse Organization is billed through a cloud marketplace committed spend agreement will I automatically move to PAYG billing when I run out of credits? {#automatically-move-to-PAYG-when-running-out-of-credit} +### If my ClickHouse organization is billed through a cloud marketplace committed spend agreement will i automatically move to payg billing when i run out of credits? {#automatically-move-to-PAYG-when-running-out-of-credit} If your marketplace committed spend contract is active and you run out of credits we will automatically move your organization to PAYG billing. However, when your existing contract expires, you will need to link a new marketplace contract to your organization or move your organization to direct billing via credit card. diff --git a/docs/cloud/reference/05_supported-regions.md b/docs/cloud/reference/05_supported-regions.md index 623e5d7e48a..116ee62a2e0 100644 --- a/docs/cloud/reference/05_supported-regions.md +++ b/docs/cloud/reference/05_supported-regions.md @@ -33,7 +33,7 @@ import EnterprisePlanFeatureBadge from '@theme/badges/EnterprisePlanFeatureBadge - eu-north-1 (Stockholm) - sa-east-1 (South America) -## Google Cloud regions {#google-cloud-regions} +## Google cloud regions {#google-cloud-regions} - asia-southeast1 (Singapore) - asia-northeast1 (Tokyo) diff --git a/docs/cloud/reference/09_security/03_compliance-overview.md b/docs/cloud/reference/09_security/03_compliance-overview.md index f6dc02382b0..a03252fa5a0 100644 --- a/docs/cloud/reference/09_security/03_compliance-overview.md +++ b/docs/cloud/reference/09_security/03_compliance-overview.md @@ -12,15 +12,15 @@ import EnterprisePlanFeatureBadge from '@theme/badges/EnterprisePlanFeatureBadge # Security and compliance reports ClickHouse evaluates the security and compliance needs of our customers and is continuously expanding the program as additional reports are requested. For additional information or to download the reports visit our [Trust Center](https://trust.clickhouse.com). -## SOC 2 Type II (since 2022) {#soc-2-type-ii-since-2022} +## SOC 2 type ii (since 2022) {#soc-2-type-ii-since-2022} System and Organization Controls (SOC) 2 is a report focusing on security, availability, confidentiality, processing integrity and privacy criteria contained in the Trust Services Criteria (TSC) as applied to an organization's systems and is designed to provide assurance about these controls to relying parties (our customers). ClickHouse works with independent external auditors to undergo an audit at least once per year addressing security, availability and processing integrity of our systems and confidentiality and privacy of the data processed by our systems. The report addresses both our ClickHouse Cloud and Bring Your Own Cloud (BYOC) offerings. -## ISO 27001 (Since 2023) {#iso-27001-since-2023} +## Iso 27001 (since 2023) {#iso-27001-since-2023} International Standards Organization (ISO) 27001 is an international standard for information security. It requires companies to implement an Information Security Management System (ISMS) that includes processes for managing risks, creating and communicating policies, implementing security controls, and monitoring to ensure components remain relevant and effective. ClickHouse conducts internal audits and works with independent external auditors to undergo audits and interim inspections for the 2 years between certificate issuance. -## U.S. DPF (since 2024) {#us-dpf-since-2024} +## U.S. dpf (since 2024) {#us-dpf-since-2024} The U.S. Data Privacy Framework was developed to provide U.S. organizations with reliable mechanisms for personal data transfers to the United States from the European Union/ European Economic Area, the United Kingdom, and Switzerland that are consistent with EU, UK and Swiss law (https://dataprivacyframework.gov/Program-Overview). ClickHouse self-certified to the framework and is listed on the [Data Privacy Framework List](https://dataprivacyframework.gov/list). diff --git a/docs/cloud/reference/data-resiliency.md b/docs/cloud/reference/data-resiliency.md index b2acc5bc302..ad79d54db91 100644 --- a/docs/cloud/reference/data-resiliency.md +++ b/docs/cloud/reference/data-resiliency.md @@ -58,7 +58,7 @@ Customers can also configure longer retention. The backups currently available for the service are listed on the “backups” page on the ClickHouse Cloud console. This section also provides the success / failure status for each backup. -## Restoring from a Backup {#restoring-from-a-backup} +## Restoring from a backup {#restoring-from-a-backup} 1. Default backups, in the ClickHouse Cloud bucket, can be restored to a new service in the same region. 2. External backups (in customer object storage) can be restored to a new service in the same or different region. diff --git a/docs/data-modeling/backfilling.md b/docs/data-modeling/backfilling.md index 6f690b10849..04f08ed3d1e 100644 --- a/docs/data-modeling/backfilling.md +++ b/docs/data-modeling/backfilling.md @@ -9,7 +9,7 @@ doc_type: 'guide' import nullTableMV from '@site/static/images/data-modeling/null_table_mv.png'; import Image from '@theme/IdealImage'; -# Backfilling Data +# Backfilling data Whether new to ClickHouse or responsible for an existing deployment, users will invariably need to backfill tables with historical data. In some cases, this is relatively simple but can become more complex when materialized views need to be populated. This guide documents some processes for this task that users can apply to their use case. @@ -257,7 +257,7 @@ FROM s3('https://datasets-documentation.s3.eu-west-3.amazonaws.com/pypi/2024-12- ClickPipes uses this approach when loading data from object storage, automatically creating duplicates of the target table and its materialized views and avoiding the need for the user to perform the above steps. By also using multiple worker threads, each handling different subsets (via glob patterns) and with its own duplicate tables, data can be loaded quickly with exactly-once semantics. For those interested, further details can be found [in this blog](https://clickhouse.com/blog/supercharge-your-clickhouse-data-loads-part3). ::: -## Scenario 1: Backfilling data with existing data ingestion {#scenario-1-backfilling-data-with-existing-data-ingestion} +## Scenario 1: backfilling data with existing data ingestion {#scenario-1-backfilling-data-with-existing-data-ingestion} In this scenario, we assume that the data to backfill is not in an isolated bucket and thus filtering is required. Data is already inserting and a timestamp or monotonically increasing column can be identified from which historical data needs to be backfilled. @@ -319,7 +319,7 @@ If the historical data is an isolated bucket, the above time filter is not requi ClickHouse Cloud users should use ClickPipes for restoring historical backups if the data can be isolated in its own bucket (and a filter is not required). As well as parallelizing the load with multiple workers, thus reducing the load time, ClickPipes automates the above process - creating duplicate tables for both the main table and materialized views. ::: -## Scenario 2: Adding materialized views to existing tables {#scenario-2-adding-materialized-views-to-existing-tables} +## Scenario 2: adding materialized views to existing tables {#scenario-2-adding-materialized-views-to-existing-tables} It is not uncommon for new materialized views to need to be added to a setup for which significant data has been populated and data is being inserted. A timestamp or monotonically increasing column, which can be used to identify a point in the stream, is useful here and avoids pauses in data ingestion. In the examples below, we assume both cases, preferring approaches that avoid pauses in ingestion. @@ -327,7 +327,7 @@ It is not uncommon for new materialized views to need to be added to a setup for We do not recommend using the [`POPULATE`](/sql-reference/statements/create/view#materialized-view) command for backfilling materialized views for anything other than small datasets where ingest is paused. This operator can miss rows inserted into its source table, with the materialized view created after the populate hash is finished. Furthermore, this populate runs against all data and is vulnerable to interruptions or memory limits on large datasets. ::: -### Timestamp or Monotonically increasing column available {#timestamp-or-monotonically-increasing-column-available} +### Timestamp or monotonically increasing column available {#timestamp-or-monotonically-increasing-column-available} In this case, we recommend the new materialized view include a filter that restricts rows to those greater than arbitrary data in the future. The materialized view can subsequently be backfilled from this date using historical data from the main table. The backfilling approach depends on the data size and the complexity of the associated query. @@ -410,7 +410,7 @@ In this case, users have several options: We explore (2) further below. -#### Using a Null table engine for filling materialized views {#using-a-null-table-engine-for-filling-materialized-views} +#### Using a null table engine for filling materialized views {#using-a-null-table-engine-for-filling-materialized-views} The [Null table engine](/engines/table-engines/special/null) provides a storage engine which doesn't persist data (think of it as the `/dev/null` of the table engine world). While this seems contradictory, materialized views will still execute on data inserted into this table engine. This allows materialized views to be constructed without persisting the original data - avoiding I/O and the associated storage. diff --git a/docs/data-modeling/denormalization.md b/docs/data-modeling/denormalization.md index 866e1f2c0b3..ed2a8edd583 100644 --- a/docs/data-modeling/denormalization.md +++ b/docs/data-modeling/denormalization.md @@ -10,11 +10,11 @@ import denormalizationDiagram from '@site/static/images/data-modeling/denormaliz import denormalizationSchema from '@site/static/images/data-modeling/denormalization-schema.png'; import Image from '@theme/IdealImage'; -# Denormalizing Data +# Denormalizing data Data denormalization is a technique in ClickHouse to use flattened tables to help minimize query latency by avoiding joins. -## Comparing Normalized vs. Denormalized Schemas {#comparing-normalized-vs-denormalized-schemas} +## Comparing normalized vs. denormalized schemas {#comparing-normalized-vs-denormalized-schemas} Denormalizing data involves intentionally reversing the normalization process to optimize database performance for specific query patterns. In normalized databases, data is split into multiple related tables to minimize redundancy and ensure data integrity. Denormalization reintroduces redundancy by combining tables, duplicating data, and incorporating calculated fields into either a single table or fewer tables - effectively moving any joins from query to insert time. @@ -64,7 +64,7 @@ We also only consider denormalizing other tables onto `Posts`, as we consider th *For each of the following examples, assume a query exists which requires both tables to be used in a join.* -### Posts and Votes {#posts-and-votes} +### Posts and votes {#posts-and-votes} Votes for posts are represented as separate tables. The optimized schema for this is shown below as well as the insert command to load the data: @@ -129,7 +129,7 @@ LIMIT 5 The main observation here is that aggregated vote statistics for each post would be sufficient for most analysis - we do not need to denormalize all of the vote information. For example, the current `Score` column represents such a statistic i.e. total up votes minus down votes. Ideally, we would just be able to retrieve these statistics at query time with a simple lookup (see [dictionaries](/dictionary)). -### Users and Badges {#users-and-badges} +### Users and badges {#users-and-badges} Now let's consider our `Users` and `Badges`: @@ -199,7 +199,7 @@ It's probably not realistic to denormalize 19k objects onto a single row. This r > We may wish to denormalize statistics from badges on to users e.g. the number of badges. We consider such an example when using dictionaries for this dataset at insert time. -### Posts and PostLinks {#posts-and-postlinks} +### Posts and postlinks {#posts-and-postlinks} `PostLinks` connect `Posts` which users consider to be related or duplicated. The following query shows the schema and load command: diff --git a/docs/data-modeling/index.md b/docs/data-modeling/index.md index d40f0d09c9f..755942cc08e 100644 --- a/docs/data-modeling/index.md +++ b/docs/data-modeling/index.md @@ -6,7 +6,7 @@ keywords: ['data modelling', 'schema design', 'dictionary', 'materialized view', doc_type: 'landing-page' --- -# Data Modeling +# Data modeling This section is about data modeling in ClickHouse and contains the following topics: diff --git a/docs/data-modeling/projections/1_projections.md b/docs/data-modeling/projections/1_projections.md index 0a13906f7d5..c2922fc97cb 100644 --- a/docs/data-modeling/projections/1_projections.md +++ b/docs/data-modeling/projections/1_projections.md @@ -29,7 +29,7 @@ queries by creating a reordering of data by attributes of interest. This can be:
-## How do Projections work? {#how-do-projections-work} +## How do projections work? {#how-do-projections-work} Practically, a Projection can be thought of as an additional, hidden table to the original table. The projection can have a different row order, and therefore a @@ -71,7 +71,7 @@ There are now two ways to define a projection: The approaches above can also be mixed, storing some columns in the projection and others indirectly via `_part_offset`. -## When to use Projections? {#when-to-use-projections} +## When to use projections? {#when-to-use-projections} Projections are an appealing feature for new users as they are automatically maintained as data is inserted. Furthermore, queries can just be sent to a @@ -194,7 +194,7 @@ WHERE query_id='' └───────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────┘ ``` -### Using projections to speed up UK price paid queries {#using-projections-to-speed-up-UK-price-paid} +### Using projections to speed up uk price paid queries {#using-projections-to-speed-up-UK-price-paid} To demonstrate how projections can be used to speed up query performance, let's take a look at an example using a real life dataset. For this example we'll be @@ -409,7 +409,7 @@ CREATE TABLE uk.uk_price_paid_with_projections_v2 AS uk.uk_price_paid; INSERT INTO uk.uk_price_paid_with_projections_v2 SELECT * FROM uk.uk_price_paid; ``` -#### Build a Projection {#build-projection} +#### Build a projection {#build-projection} Let's create an aggregate projection by the dimensions `toYear(date)`, `district`, and `town`: @@ -441,7 +441,7 @@ SETTINGS mutations_sync = 1 The following queries contrast performance with and without projections. To disable projection use we use the setting [`optimize_use_projections`](/operations/settings/settings#optimize_use_projections), which is enabled by default. -#### Query 1. Average price per year {#average-price-projections} +#### Query 1. average price per year {#average-price-projections} ```sql runnable SELECT @@ -466,7 +466,7 @@ ORDER BY year ASC ``` The results should be the same, but the performance better on the latter example! -#### Query 2. Average price per year in London {#average-price-london-projections} +#### Query 2. average price per year in London {#average-price-london-projections} ```sql runnable SELECT @@ -491,7 +491,7 @@ GROUP BY year ORDER BY year ASC ``` -#### Query 3. The most expensive neighborhoods {#most-expensive-neighborhoods-projections} +#### Query 3. the most expensive neighborhoods {#most-expensive-neighborhoods-projections} The condition (date >= '2020-01-01') needs to be modified so that it matches the projection dimension (`toYear(date) >= 2020)`: diff --git a/docs/data-modeling/schema-design.md b/docs/data-modeling/schema-design.md index 7b3aba6c19f..9dbe164db98 100644 --- a/docs/data-modeling/schema-design.md +++ b/docs/data-modeling/schema-design.md @@ -12,7 +12,7 @@ import Image from '@theme/IdealImage'; Understanding effective schema design is key to optimizing ClickHouse performance and includes choices that often involve trade-offs, with the optimal approach depending on the queries being served as well as factors such as data update frequency, latency requirements, and data volume. This guide provides an overview of schema design best practices and data modeling techniques for optimizing ClickHouse performance. -## Stack Overflow dataset {#stack-overflow-dataset} +## Stack overflow dataset {#stack-overflow-dataset} For the examples in this guide, we use a subset of the Stack Overflow dataset. This contains every post, vote, user, comment and badge that has occurred on Stack Overflow from 2008 to Apr 2024. This data is available in Parquet using the schemas below under the S3 bucket `s3://datasets-documentation/stackoverflow/parquet/`: @@ -126,7 +126,7 @@ INSERT INTO posts SELECT * FROM s3('https://datasets-documentation.s3.eu-west-3. > The above query loads 60m rows. While small for ClickHouse, users with slower internet connections may wish to load a subset of data. This can be achieved by simply specifying the years they wish to load via a glob pattern e.g. `https://datasets-documentation.s3.eu-west-3.amazonaws.com/stackoverflow/parquet/posts/2008.parquet` or `https://datasets-documentation.s3.eu-west-3.amazonaws.com/stackoverflow/parquet/posts/{2008, 2009}.parquet`. See [here](/sql-reference/table-functions/file#globs-in-path) for how glob patterns can be used to target subsets of files. -## Optimizing Types {#optimizing-types} +## Optimizing types {#optimizing-types} One of the secrets to ClickHouse query performance is compression. @@ -327,7 +327,7 @@ LIMIT 3 For users interested in the compression improvements achieved by using specific types and appropriate ordering keys, see [Compression in ClickHouse](/data-compression/compression-in-clickhouse). If users need to further improve compression we also recommend the section [Choosing the right column compression codec](/data-compression/compression-in-clickhouse#choosing-the-right-column-compression-codec). -## Next: Data Modeling Techniques {#next-data-modeling-techniques} +## Next: data modeling techniques {#next-data-modeling-techniques} Until now, we've migrated only a single table. While this has allowed us to introduce some core ClickHouse concepts, most schemas are unfortunately not this simple. diff --git a/docs/deployment-guides/replication-sharding-examples/01_1_shard_2_replicas.md b/docs/deployment-guides/replication-sharding-examples/01_1_shard_2_replicas.md index 9007a1a2878..5d57e4c3e51 100644 --- a/docs/deployment-guides/replication-sharding-examples/01_1_shard_2_replicas.md +++ b/docs/deployment-guides/replication-sharding-examples/01_1_shard_2_replicas.md @@ -384,7 +384,7 @@ In practice, this is discouraged. In this example, each `users.xml` file is identical for all nodes in the cluster. ::: -## Configure ClickHouse Keeper {#configure-clickhouse-keeper-nodes} +## Configure ClickHouse keeper {#configure-clickhouse-keeper-nodes} ### Keeper setup {#configuration-explanation} diff --git a/docs/deployment-guides/replication-sharding-examples/02_2_shards_1_replica.md b/docs/deployment-guides/replication-sharding-examples/02_2_shards_1_replica.md index 03fae707423..69c754e05c7 100644 --- a/docs/deployment-guides/replication-sharding-examples/02_2_shards_1_replica.md +++ b/docs/deployment-guides/replication-sharding-examples/02_2_shards_1_replica.md @@ -402,7 +402,7 @@ In practice, this is discouraged. In this example, each `users.xml` file is identical for all nodes in the cluster. ::: -## Configure ClickHouse Keeper {#configure-clickhouse-keeper-nodes} +## Configure ClickHouse keeper {#configure-clickhouse-keeper-nodes} ### Keeper setup {#configuration-explanation} diff --git a/docs/deployment-guides/replication-sharding-examples/03_2_shards_2_replicas.md b/docs/deployment-guides/replication-sharding-examples/03_2_shards_2_replicas.md index 2571128f99b..69c41ed3254 100644 --- a/docs/deployment-guides/replication-sharding-examples/03_2_shards_2_replicas.md +++ b/docs/deployment-guides/replication-sharding-examples/03_2_shards_2_replicas.md @@ -425,7 +425,7 @@ In practice, this is discouraged. In this example, each `users.xml` file is identical for all nodes in the cluster. ::: -## Configure ClickHouse Keeper {#configure-clickhouse-keeper-nodes} +## Configure ClickHouse keeper {#configure-clickhouse-keeper-nodes} Next you will configure ClickHouse Keeper, which is used for coordination. diff --git a/docs/deployment-modes.md b/docs/deployment-modes.md index edc0afed0d0..11918f583d2 100644 --- a/docs/deployment-modes.md +++ b/docs/deployment-modes.md @@ -28,7 +28,7 @@ Each deployment mode has its own strengths and ideal use cases, which we'll expl -## ClickHouse Server {#clickhouse-server} +## ClickHouse server {#clickhouse-server} ClickHouse Server represents the traditional client/server architecture and is ideal for production deployments. This deployment mode provides the full OLAP database capabilities with high throughput and low latency queries that ClickHouse is known for. diff --git a/docs/dictionary/index.md b/docs/dictionary/index.md index cd568574dd5..4ae3d996092 100644 --- a/docs/dictionary/index.md +++ b/docs/dictionary/index.md @@ -20,7 +20,7 @@ Dictionaries are useful for: Use cases for Dictionary in ClickHouse -## Speeding up joins using a Dictionary {#speeding-up-joins-using-a-dictionary} +## Speeding up joins using a dictionary {#speeding-up-joins-using-a-dictionary} Dictionaries can be used to speed up a specific type of `JOIN`: the [`LEFT ANY` type](/sql-reference/statements/select/join#supported-types-of-join) where the join key needs to match the key attribute of the underlying key-value storage. @@ -314,7 +314,7 @@ Peak memory usage: 666.82 MiB. ## Advanced dictionary topics {#advanced-dictionary-topics} -### Choosing the Dictionary `LAYOUT` {#choosing-the-dictionary-layout} +### Choosing the dictionary `LAYOUT` {#choosing-the-dictionary-layout} The `LAYOUT` clause controls the internal data structure for the dictionary. A number of options exist and are documented [here](/sql-reference/dictionaries#ways-to-store-dictionaries-in-memory). Some tips on choosing the correct layout can be found [here](https://clickhouse.com/blog/faster-queries-dictionaries-clickhouse#choosing-a-layout). diff --git a/docs/faq/general/datalake.md b/docs/faq/general/datalake.md index e434b47e46d..b809e0a36af 100644 --- a/docs/faq/general/datalake.md +++ b/docs/faq/general/datalake.md @@ -8,7 +8,7 @@ doc_type: 'reference' keywords: ['data lake', 'lakehouse'] --- -# Does ClickHouse support Data Lakes? +# Does ClickHouse support data lakes? ClickHouse supports data lakes, including Iceberg, Delta Lake, Apache Hudi, Apache Paimon, Hive. diff --git a/docs/faq/general/distributed-join.md b/docs/faq/general/distributed-join.md index ac3065546b0..f897b65be44 100644 --- a/docs/faq/general/distributed-join.md +++ b/docs/faq/general/distributed-join.md @@ -8,7 +8,7 @@ doc_type: 'reference' keywords: ['distributed', 'join'] --- -# Does ClickHouse support distributed JOIN? +# Does ClickHouse support distributed join? ClickHouse supports distributed JOIN on a cluster. diff --git a/docs/faq/general/mapreduce.md b/docs/faq/general/mapreduce.md index 34c1e23601f..6ce684729bc 100644 --- a/docs/faq/general/mapreduce.md +++ b/docs/faq/general/mapreduce.md @@ -8,7 +8,7 @@ keywords: ['MapReduce'] doc_type: 'reference' --- -# Why not use something like MapReduce? {#why-not-use-something-like-mapreduce} +# Why not use something like mapreduce? {#why-not-use-something-like-mapreduce} We can refer to systems like MapReduce as distributed computing systems in which the reduce operation is based on distributed sorting. The most common open-source solution in this class is [Apache Hadoop](http://hadoop.apache.org). diff --git a/docs/faq/general/ne-tormozit.md b/docs/faq/general/ne-tormozit.md index dc1336c1b45..faee5c740b5 100644 --- a/docs/faq/general/ne-tormozit.md +++ b/docs/faq/general/ne-tormozit.md @@ -8,7 +8,7 @@ keywords: ['Yandex'] doc_type: 'reference' --- -# What does "Не тормозит" mean? {#what-does-ne-tormozit-mean} +# What does "не тормозит" mean? {#what-does-ne-tormozit-mean} We often get this question when people see vintage (limited production) ClickHouse t-shirts. They have the words **"ClickHouse не тормозит"** written in big bold text on the front. diff --git a/docs/faq/general/olap.md b/docs/faq/general/olap.md index bbb939b7182..da0f61af168 100644 --- a/docs/faq/general/olap.md +++ b/docs/faq/general/olap.md @@ -8,7 +8,7 @@ keywords: ['OLAP'] doc_type: 'reference' --- -# What Is OLAP? {#what-is-olap} +# What is OLAP? {#what-is-olap} [OLAP](https://en.wikipedia.org/wiki/Online_analytical_processing) stands for Online Analytical Processing. It is a broad term that can be looked at from two perspectives: technical and business. But at the very high level, you can just read these words backward: diff --git a/docs/faq/integration/json-import.md b/docs/faq/integration/json-import.md index 7c856032ac0..782fc628006 100644 --- a/docs/faq/integration/json-import.md +++ b/docs/faq/integration/json-import.md @@ -8,7 +8,7 @@ keywords: ['JSON import', 'JSONEachRow format', 'data import', 'JSON ingestion', doc_type: 'guide' --- -# How to Import JSON Into ClickHouse? {#how-to-import-json-into-clickhouse} +# How to import JSON into ClickHouse? {#how-to-import-json-into-clickhouse} ClickHouse supports a wide range of [data formats for input and output](/interfaces/formats). There are multiple JSON variations among them, but the most commonly used for data ingestion is [JSONEachRow](/interfaces/formats/JSONEachRow). It expects one JSON object per row, each object separated by a newline. diff --git a/docs/faq/integration/oracle-odbc.md b/docs/faq/integration/oracle-odbc.md index 71f92b09e82..b16926df3ad 100644 --- a/docs/faq/integration/oracle-odbc.md +++ b/docs/faq/integration/oracle-odbc.md @@ -8,7 +8,7 @@ doc_type: 'guide' keywords: ['oracle', 'odbc', 'encoding', 'integration', 'external dictionary'] --- -# What if I have a problem with encodings when using Oracle via ODBC? {#oracle-odbc-encodings} +# What if i have a problem with encodings when using oracle via odbc? {#oracle-odbc-encodings} If you use Oracle as a source of ClickHouse external dictionaries via Oracle ODBC driver, you need to set the correct value for the `NLS_LANG` environment variable in `/etc/default/clickhouse`. For more information, see the [Oracle NLS_LANG FAQ](https://www.oracle.com/technetwork/products/globalization/nls-lang-099431.html). diff --git a/docs/faq/operations/delete-old-data.md b/docs/faq/operations/delete-old-data.md index b597ee9f52a..22dee462a54 100644 --- a/docs/faq/operations/delete-old-data.md +++ b/docs/faq/operations/delete-old-data.md @@ -24,7 +24,7 @@ TTL can also be used to move data not only to [/dev/null](https://en.wikipedia.o More details on [configuring TTL](../../engines/table-engines/mergetree-family/mergetree.md#table_engine-mergetree-ttl). -## DELETE FROM {#delete-from} +## Delete from {#delete-from} [DELETE FROM](/sql-reference/statements/delete.md) allows standard DELETE queries to be run in ClickHouse. The rows targeted in the filter clause are marked as deleted, and removed from future result sets. Cleanup of the rows happens asynchronously. :::note @@ -34,7 +34,7 @@ SET allow_experimental_lightweight_delete = true; ``` ::: -## ALTER DELETE {#alter-delete} +## Alter delete {#alter-delete} ALTER DELETE removes rows using asynchronous batch operations. Unlike DELETE FROM, queries run after the ALTER DELETE and before the batch operations complete will include the rows targeted for deletion. For more details see the [ALTER DELETE](/sql-reference/statements/alter/delete.md) docs. @@ -44,13 +44,13 @@ This is the most common approach to make your system based on ClickHouse [GDPR]( More details on [mutations](/sql-reference/statements/alter#mutations). -## DROP PARTITION {#drop-partition} +## Drop partition {#drop-partition} `ALTER TABLE ... DROP PARTITION` provides a cost-efficient way to drop a whole partition. It's not that flexible and needs proper partitioning scheme configured on table creation, but still covers most common cases. Like mutations need to be executed from an external system for regular use. More details on [manipulating partitions](/sql-reference/statements/alter/partition). -## TRUNCATE {#truncate} +## Truncate {#truncate} It's rather radical to drop all data from a table, but in some cases it might be exactly what you need. diff --git a/docs/faq/use-cases/key-value.md b/docs/faq/use-cases/key-value.md index e20d155a8ab..82aed83335c 100644 --- a/docs/faq/use-cases/key-value.md +++ b/docs/faq/use-cases/key-value.md @@ -8,7 +8,7 @@ doc_type: 'reference' keywords: ['key-value', 'data model', 'use case', 'schema design', 'storage pattern'] --- -# Can I use ClickHouse as a key-value storage? {#can-i-use-clickhouse-as-a-key-value-storage} +# Can i use ClickHouse as a key-value storage? {#can-i-use-clickhouse-as-a-key-value-storage} The short answer is **"no"**. The key-value workload is among top positions in the list of cases when **NOT** to use ClickHouse. It's an [OLAP](../../faq/general/olap.md) system after all, while there are many excellent key-value storage systems out there. diff --git a/docs/faq/use-cases/time-series.md b/docs/faq/use-cases/time-series.md index abf1f6e1ce6..5d460279adc 100644 --- a/docs/faq/use-cases/time-series.md +++ b/docs/faq/use-cases/time-series.md @@ -8,7 +8,7 @@ doc_type: 'guide' keywords: ['time series', 'temporal data', 'use case', 'time-based analytics', 'timeseries'] --- -# Can I use ClickHouse as a time-series database? {#can-i-use-clickhouse-as-a-time-series-database} +# Can i use ClickHouse as a time-series database? {#can-i-use-clickhouse-as-a-time-series-database} _Note: Please see the blog [Working with Time series data in ClickHouse](https://clickhouse.com/blog/working-with-time-series-data-and-functions-ClickHouse) for additional examples of using ClickHouse for time series analysis._ diff --git a/docs/getting-started/example-datasets/anon_web_analytics_metrica.md b/docs/getting-started/example-datasets/anon_web_analytics_metrica.md index 040b6521f7b..f3b1e61c22f 100644 --- a/docs/getting-started/example-datasets/anon_web_analytics_metrica.md +++ b/docs/getting-started/example-datasets/anon_web_analytics_metrica.md @@ -88,7 +88,7 @@ clickhouse-client --query "SELECT COUNT(*) FROM datasets.visits_v1" 1680609 ``` -## An example JOIN {#an-example-join} +## An example join {#an-example-join} The hits and visits dataset is used in the ClickHouse test routines, this is one of the queries from the test suite. The rest diff --git a/docs/getting-started/example-datasets/dbpedia.md b/docs/getting-started/example-datasets/dbpedia.md index d1f784bf84e..7a0d075ab72 100644 --- a/docs/getting-started/example-datasets/dbpedia.md +++ b/docs/getting-started/example-datasets/dbpedia.md @@ -137,7 +137,7 @@ The parameters and performance considerations for index creation and search are Building and saving the index could take a few minutes depending on number of CPU cores available and the storage bandwidth. -## Perform ANN search {#perform-ann-search} +## Perform ann search {#perform-ann-search} _Approximate Nearest Neighbours_ or ANN refers to group of techniques (e.g., special data structures like graphs and random forests) which compute results much faster than exact vector search. The result accuracy is typically "good enough" for practical use. Many approximate techniques provide parameters to tune the trade-off between the result accuracy and the search time. diff --git a/docs/getting-started/example-datasets/github.md b/docs/getting-started/example-datasets/github.md index 0c7d9d9a9dc..491a3871215 100644 --- a/docs/getting-started/example-datasets/github.md +++ b/docs/getting-started/example-datasets/github.md @@ -1830,7 +1830,7 @@ LIMIT 10 10 rows in set. Elapsed: 3.134 sec. Processed 16.13 million rows, 1.83 GB (5.15 million rows/s., 582.99 MB/s.) ``` -### Who tends to write more tests / CPP code / comments? {#who-tends-to-write-more-tests--cpp-code--comments} +### Who tends to write more tests / cpp code / comments? {#who-tends-to-write-more-tests--cpp-code--comments} There are a few ways we can address this question. Focusing on the code to test ratio, this query is relatively simple - count the number of contributions to folders containing `tests` and compute the ratio to total contributions. diff --git a/docs/getting-started/example-datasets/hacker-news-vector-search.md b/docs/getting-started/example-datasets/hacker-news-vector-search.md index 29b37b681e5..d34dc4b9d79 100644 --- a/docs/getting-started/example-datasets/hacker-news-vector-search.md +++ b/docs/getting-started/example-datasets/hacker-news-vector-search.md @@ -83,7 +83,7 @@ corresponding to selected values. Building and saving the index could even take a few minutes/hour for the full 28.74 million dataset, depending on the number of CPU cores available and the storage bandwidth. -### Perform ANN search {#perform-ann-search} +### Perform ann search {#perform-ann-search} Once the vector similarity index has been built, vector search queries will automatically use the index: diff --git a/docs/getting-started/example-datasets/hacker-news.md b/docs/getting-started/example-datasets/hacker-news.md index c20204c997d..d224d8811ca 100644 --- a/docs/getting-started/example-datasets/hacker-news.md +++ b/docs/getting-started/example-datasets/hacker-news.md @@ -7,7 +7,7 @@ doc_type: 'guide' keywords: ['example dataset', 'hacker news', 'sample data', 'text analysis', 'vector search'] --- -# Hacker News dataset +# Hacker news dataset > In this tutorial, you'll insert 28 million rows of Hacker News data into a ClickHouse > table from both CSV and Parquet formats and run some simple queries to explore the data. @@ -244,7 +244,7 @@ INSERT INTO hackernews FROM INFILE '/data/hacknernews.csv.gz' FORMAT CSVWithName Some sample queries are presented below to give you inspiration for writing your own queries. -#### How pervasive a topic is "ClickHouse" in Hacker News? {#how-pervasive} +#### How pervasive a topic is "ClickHouse" in hacker news? {#how-pervasive} The score field provides a metric of popularity for stories, while the `id` field and `||` concatenation operator can be used to produce a link to the original post. diff --git a/docs/getting-started/example-datasets/laion.md b/docs/getting-started/example-datasets/laion.md index f8dc1bc2333..b2cca83d58d 100644 --- a/docs/getting-started/example-datasets/laion.md +++ b/docs/getting-started/example-datasets/laion.md @@ -184,7 +184,7 @@ The query latency decreased significantly because the nearest neighbours were re Vector similarity search using a vector similarity index may return results that differ slightly from the brute-force search results. An HNSW index can potentially achieve a recall close to 1 (same accuracy as brute force search) with a careful selection of the HNSW parameters and evaluating the index quality. -## Creating embeddings with UDFs {#creating-embeddings-with-udfs} +## Creating embeddings with udfs {#creating-embeddings-with-udfs} One usually wants to create embeddings for new images or new image captions and search for similar image / image caption pairs in the data. We can use [UDF](/sql-reference/functions/udf) to create the `target` vector without leaving the client. It is important to use the same model to create the data and new embeddings for searches. The following scripts utilize the `ViT-B/32` model which also underlies the dataset. diff --git a/docs/getting-started/example-datasets/laion5b.md b/docs/getting-started/example-datasets/laion5b.md index 5397fd656fe..61308297507 100644 --- a/docs/getting-started/example-datasets/laion5b.md +++ b/docs/getting-started/example-datasets/laion5b.md @@ -143,7 +143,7 @@ corresponding to selected values. Building and saving the index could even take a few hours for the full l00 million dataset, depending on the number of CPU cores available and the storage bandwidth. -### Perform ANN search {#perform-ann-search} +### Perform ann search {#perform-ann-search} Once the vector similarity index has been built, vector search queries will automatically use the index: diff --git a/docs/getting-started/example-datasets/nypd_complaint_data.md b/docs/getting-started/example-datasets/nypd_complaint_data.md index fab286b52c0..e30740d15af 100644 --- a/docs/getting-started/example-datasets/nypd_complaint_data.md +++ b/docs/getting-started/example-datasets/nypd_complaint_data.md @@ -306,7 +306,7 @@ Result: └─────────────────────┘ ``` -## Convert the date and time String to a DateTime64 type {#convert-the-date-and-time-string-to-a-datetime64-type} +## Convert the date and time string to a datetime64 type {#convert-the-date-and-time-string-to-a-datetime64-type} Earlier in the guide we discovered that there are dates in the TSV file before January 1st 1970, which means that we need a 64 bit DateTime type for the dates. The dates also need to be converted from `MM/DD/YYYY` to `YYYY/MM/DD` format. Both of these can be done with [`parseDateTime64BestEffort()`](../../sql-reference/functions/type-conversion-functions.md#parsedatetime64besteffort). @@ -582,7 +582,7 @@ Result: ## Run some queries {#run-queries} -### Query 1. Compare the number of complaints by month {#query-1-compare-the-number-of-complaints-by-month} +### Query 1. compare the number of complaints by month {#query-1-compare-the-number-of-complaints-by-month} Query: @@ -618,7 +618,7 @@ Query id: 7fbd4244-b32a-4acf-b1f3-c3aa198e74d9 12 rows in set. Elapsed: 0.006 sec. Processed 208.99 thousand rows, 417.99 KB (37.48 million rows/s., 74.96 MB/s.) ``` -### Query 2. Compare total number of complaints by borough {#query-2-compare-total-number-of-complaints-by-borough} +### Query 2. compare total number of complaints by borough {#query-2-compare-total-number-of-complaints-by-borough} Query: diff --git a/docs/getting-started/example-datasets/stackoverflow.md b/docs/getting-started/example-datasets/stackoverflow.md index d0814c738b3..850d93406ac 100644 --- a/docs/getting-started/example-datasets/stackoverflow.md +++ b/docs/getting-started/example-datasets/stackoverflow.md @@ -161,7 +161,7 @@ INSERT INTO stackoverflow.badges SELECT * FROM s3('https://datasets-documentatio 0 rows in set. Elapsed: 6.635 sec. Processed 51.29 million rows, 797.05 MB (7.73 million rows/s., 120.13 MB/s.) ``` -### PostLinks {#postlinks} +### Postlinks {#postlinks} ```sql CREATE TABLE stackoverflow.postlinks @@ -180,7 +180,7 @@ INSERT INTO stackoverflow.postlinks SELECT * FROM s3('https://datasets-documenta 0 rows in set. Elapsed: 1.534 sec. Processed 6.55 million rows, 129.70 MB (4.27 million rows/s., 84.57 MB/s.) ``` -### PostHistory {#posthistory} +### Posthistory {#posthistory} ```sql CREATE TABLE stackoverflow.posthistory @@ -272,7 +272,7 @@ clickhouse local --query "SELECT * FROM file('posts.json', JSONEachRow, 'Id Int3 A few simple questions to you get started. -### Most popular tags on Stack Overflow {#most-popular-tags-on-stack-overflow} +### Most popular tags on stack overflow {#most-popular-tags-on-stack-overflow} ```sql diff --git a/docs/getting-started/example-datasets/tpch.md b/docs/getting-started/example-datasets/tpch.md index 595fb4ade4b..f26baee597b 100644 --- a/docs/getting-started/example-datasets/tpch.md +++ b/docs/getting-started/example-datasets/tpch.md @@ -18,7 +18,7 @@ Despite its age and its unrealistic assumption that the data is uniformly and in - [TPC-H Analyzed: Hidden Messages and Lessons Learned from an Influential Benchmark](https://doi.org/10.1007/978-3-319-04936-6_5) (Boncz et. al.), 2013 - [Quantifying TPC-H Choke Points and Their Optimizations](https://doi.org/10.14778/3389133.3389138) (Dresseler et. al.), 2020 -## Data Generation and Import {#data-generation-and-import} +## Data generation and import {#data-generation-and-import} First, checkout the TPC-H repository and compile the data generator: diff --git a/docs/getting-started/example-datasets/tw-weather.md b/docs/getting-started/example-datasets/tw-weather.md index 5ea2b823cf8..73cd41d7644 100644 --- a/docs/getting-started/example-datasets/tw-weather.md +++ b/docs/getting-started/example-datasets/tw-weather.md @@ -88,7 +88,7 @@ cat *.csv | md5sum # Checksum should be equal to: b26db404bf84d4063fac42e576464ce1 ``` -#### Retrieve the Taiwan weather stations {#retrieve-the-taiwan-weather-stations} +#### Retrieve the taiwan weather stations {#retrieve-the-taiwan-weather-stations} ```bash wget -O weather_sta_list.csv https://github.com/Raingel/weather_station_list/raw/main/data/weather_sta_list.csv @@ -202,7 +202,7 @@ WHERE (`table` = 'tw_weather_data') AND active ## Sample queries {#sample-queries} -### Q1: Retrieve the highest dew point temperature for each weather station in the specific year {#q1-retrieve-the-highest-dew-point-temperature-for-each-weather-station-in-the-specific-year} +### Q1: retrieve the highest dew point temperature for each weather station in the specific year {#q1-retrieve-the-highest-dew-point-temperature-for-each-weather-station-in-the-specific-year} ```sql SELECT @@ -248,7 +248,7 @@ GROUP BY StationId 30 rows in set. Elapsed: 0.045 sec. Processed 6.41 million rows, 187.33 MB (143.92 million rows/s., 4.21 GB/s.) ``` -### Q2: Raw data fetching with the specific duration time range, fields and weather station {#q2-raw-data-fetching-with-the-specific-duration-time-range-fields-and-weather-station} +### Q2: raw data fetching with the specific duration time range, fields and weather station {#q2-raw-data-fetching-with-the-specific-duration-time-range-fields-and-weather-station} ```sql SELECT diff --git a/docs/getting-started/example-datasets/uk-price-paid.md b/docs/getting-started/example-datasets/uk-price-paid.md index 08dee72feae..230f83a524c 100644 --- a/docs/getting-started/example-datasets/uk-price-paid.md +++ b/docs/getting-started/example-datasets/uk-price-paid.md @@ -117,7 +117,7 @@ Notice the size of the table is just 221.43 MiB! Let's run some queries to analyze the data: -### Query 1. Average price per year {#average-price} +### Query 1. average price per year {#average-price} ```sql runnable SELECT @@ -146,7 +146,7 @@ ORDER BY year Something happened to home prices in 2020! But that is probably not a surprise... -### Query 3. The most expensive neighborhoods {#most-expensive-neighborhoods} +### Query 3. the most expensive neighborhoods {#most-expensive-neighborhoods} ```sql runnable SELECT diff --git a/docs/getting-started/example-datasets/youtube-dislikes.md b/docs/getting-started/example-datasets/youtube-dislikes.md index 93fcf8571a0..00c7eee863a 100644 --- a/docs/getting-started/example-datasets/youtube-dislikes.md +++ b/docs/getting-started/example-datasets/youtube-dislikes.md @@ -416,7 +416,7 @@ ORDER BY │ 2008-09-01 │ WWE │ 3717092 │ 0.07872802579349912 │ ``` -### How do like ratio changes as views go up? {#how-do-like-ratio-changes-as-views-go-up} +### How do like ratio changes as views Go up? {#how-do-like-ratio-changes-as-views-go-up} ```sql SELECT diff --git a/docs/getting-started/install/_snippets/_deb_install.md b/docs/getting-started/install/_snippets/_deb_install.md index 31e1fa6532d..c349a611ea9 100644 --- a/docs/getting-started/install/_snippets/_deb_install.md +++ b/docs/getting-started/install/_snippets/_deb_install.md @@ -86,7 +86,7 @@ If you set up a password for your server, then you will need to run: clickhouse-client --password ``` -## Install standalone ClickHouse Keeper {#install-standalone-clickhouse-keeper} +## Install standalone ClickHouse keeper {#install-standalone-clickhouse-keeper} :::tip In production environments we strongly recommend running ClickHouse Keeper on dedicated nodes. @@ -100,7 +100,7 @@ To install `clickhouse-keeper` on standalone ClickHouse Keeper servers, run: sudo apt-get install -y clickhouse-keeper ``` -## Enable and start ClickHouse Keeper {#enable-and-start-clickhouse-keeper} +## Enable and start ClickHouse keeper {#enable-and-start-clickhouse-keeper} ```bash sudo systemctl enable clickhouse-keeper diff --git a/docs/getting-started/install/_snippets/_rpm_install.md b/docs/getting-started/install/_snippets/_rpm_install.md index 94ae1de6d88..1284281a4dd 100644 --- a/docs/getting-started/install/_snippets/_rpm_install.md +++ b/docs/getting-started/install/_snippets/_rpm_install.md @@ -5,7 +5,7 @@ -## Setup the RPM repository {#setup-the-rpm-repository} +## Setup the rpm repository {#setup-the-rpm-repository} Add the official repository by running the following command: @@ -63,7 +63,7 @@ If you set up a password for your server, then you will need to run: clickhouse-client --password ``` -## Install standalone ClickHouse Keeper {#install-standalone-clickhouse-keeper-1} +## Install standalone ClickHouse keeper {#install-standalone-clickhouse-keeper-1} :::tip In production environments we strongly recommend running ClickHouse Keeper on dedicated nodes. @@ -77,7 +77,7 @@ To install `clickhouse-keeper` on standalone ClickHouse Keeper servers, run: sudo yum install -y clickhouse-keeper ``` -## Enable and start ClickHouse Keeper {#enable-and-start-clickhouse-keeper-1} +## Enable and start ClickHouse keeper {#enable-and-start-clickhouse-keeper-1} ```bash sudo systemctl enable clickhouse-keeper diff --git a/docs/getting-started/install/advanced.md b/docs/getting-started/install/advanced.md index e42613d21d7..c7e93f6b2a8 100644 --- a/docs/getting-started/install/advanced.md +++ b/docs/getting-started/install/advanced.md @@ -28,7 +28,7 @@ You'll need to create data and metadata folders manually and `chown` them for th On Gentoo, you can just use `emerge clickhouse` to install ClickHouse from sources. -## Install a CI-generated Binary {#install-a-ci-generated-binary} +## Install a ci-generated binary {#install-a-ci-generated-binary} ClickHouse's continuous integration (CI) infrastructure produces specialized builds for each commit in the [ClickHouse repository](https://github.com/clickhouse/clickhouse/), e.g. [sanitized](https://github.com/google/sanitizers) builds, unoptimized (Debug) diff --git a/docs/guides/best-practices/index.md b/docs/guides/best-practices/index.md index 2fa3db48c91..0602214f9e1 100644 --- a/docs/guides/best-practices/index.md +++ b/docs/guides/best-practices/index.md @@ -9,7 +9,7 @@ doc_type: 'reference' import TableOfContents from '@site/docs/guides/best-practices/_snippets/_performance_optimizations_table_of_contents.md'; -# Performance and Optimizations +# Performance and optimizations This section contains tips and best practices for improving performance with ClickHouse. We recommend users read [Core Concepts](/parts) as a precursor to this section, diff --git a/docs/guides/best-practices/prewhere.md b/docs/guides/best-practices/prewhere.md index 0a0372d2210..076f3cb8885 100644 --- a/docs/guides/best-practices/prewhere.md +++ b/docs/guides/best-practices/prewhere.md @@ -16,13 +16,13 @@ import visual05 from '@site/static/images/guides/best-practices/prewhere_05.gif' import Image from '@theme/IdealImage'; -# How does the PREWHERE optimization work? +# How does the prewhere optimization work? The [PREWHERE clause](/sql-reference/statements/select/prewhere) is a query execution optimization in ClickHouse. It reduces I/O and improves query speed by avoiding unnecessary data reads, and filtering out irrelevant data before reading non-filter columns from disk. This guide explains how PREWHERE works, how to measure its impact, and how to tune it for best performance. -## Query processing without PREWHERE optimization {#query-processing-without-prewhere-optimization} +## Query processing without prewhere optimization {#query-processing-without-prewhere-optimization} We'll start by illustrating how a query on the [uk_price_paid_simple](/parts) table is processed without using PREWHERE: @@ -41,7 +41,7 @@ We'll start by illustrating how a query on the [uk_price_paid_simple](/parts) ta As you can see, without PREWHERE, all potentially relevant columns are loaded before filtering, even if only a few rows actually match. -## How PREWHERE improves query efficiency {#how-prewhere-improves-query-efficiency} +## How prewhere improves query efficiency {#how-prewhere-improves-query-efficiency} The following animations show how the query from above is processed with a PREWHERE clause applied to all query predicates. @@ -90,7 +90,7 @@ By the final step, only the minimal set of column granules, those containing mat Note that ClickHouse processes the same number of rows in both the PREWHERE and non-PREWHERE versions of the query. However, with PREWHERE optimizations applied, not all column values need to be loaded for every processed row. ::: -## PREWHERE optimization is automatically applied {#prewhere-optimization-is-automatically-applied} +## Prewhere optimization is automatically applied {#prewhere-optimization-is-automatically-applied} The PREWHERE clause can be added manually, as shown in the example above. However, you don't need to write PREWHERE manually. When the setting [`optimize_move_to_prewhere`](/operations/settings/settings#optimize_move_to_prewhere) is enabled (true by default), ClickHouse automatically moves filter conditions from WHERE to PREWHERE, prioritizing those that will reduce read volume the most. @@ -100,7 +100,7 @@ ClickHouse follows this strategy by default as of version [23.2](https://clickho Starting with version [23.11](https://clickhouse.com/blog/clickhouse-release-23-11#column-statistics-for-prewhere), optional column statistics can further improve this by choosing the filter processing order based on actual data selectivity, not just column size. -## How to measure PREWHERE impact {#how-to-measure-prewhere-impact} +## How to measure prewhere impact {#how-to-measure-prewhere-impact} To validate that PREWHERE is helping your queries, you can compare query performance with and without the `optimize_move_to_prewhere setting` enabled. diff --git a/docs/guides/best-practices/query-parallelism.md b/docs/guides/best-practices/query-parallelism.md index d05e7ec521e..a44dce8f665 100644 --- a/docs/guides/best-practices/query-parallelism.md +++ b/docs/guides/best-practices/query-parallelism.md @@ -24,7 +24,7 @@ This guide walks through how query parallelism works in ClickHouse and how you c We use an aggregation query on the [uk_price_paid_simple](/parts) dataset to illustrate key concepts. -## Step-by-step: How ClickHouse parallelizes an aggregation query {#step-by-step-how-clickHouse-parallelizes-an-aggregation-query} +## Step-by-step: how ClickHouse parallelizes an aggregation query {#step-by-step-how-clickHouse-parallelizes-an-aggregation-query} When ClickHouse ① runs an aggregation query with a filter on the table's primary key, it ② loads the primary index into memory to ③ identify which granules need to be processed, and which can be safely skipped: diff --git a/docs/guides/best-practices/skipping-indexes-examples.md b/docs/guides/best-practices/skipping-indexes-examples.md index 31ecafa7c63..08bf6d6df18 100644 --- a/docs/guides/best-practices/skipping-indexes-examples.md +++ b/docs/guides/best-practices/skipping-indexes-examples.md @@ -30,7 +30,7 @@ ClickHouse supports five skip index types: Each section provides examples with sample data and demonstrates how to verify index usage in query execution. -## MinMax index {#minmax-index} +## Minmax index {#minmax-index} The`minmax` index is best for range predicates on loosely sorted data or columns correlated with `ORDER BY`. @@ -76,7 +76,7 @@ SELECT * FROM events WHERE user_id IN (101, 202); A creation/materialization workflow and the before/after effect are shown in the [basic operation guide](/optimize/skipping-indexes#basic-operation). -## Generic Bloom filter (scalar) {#generic-bloom-filter-scalar} +## Generic bloom filter (scalar) {#generic-bloom-filter-scalar} The `bloom_filter` index is good for "needle in a haystack" equality/IN membership. It accepts an optional parameter which is the false-positive rate (default 0.025). @@ -90,7 +90,7 @@ EXPLAIN indexes = 1 SELECT * FROM events WHERE value IN (7, 42, 99); ``` -## N-gram Bloom filter (ngrambf\_v1) for substring search {#n-gram-bloom-filter-ngrambf-v1-for-substring-search} +## N-gram bloom filter (ngrambf\_v1) for substring search {#n-gram-bloom-filter-ngrambf-v1-for-substring-search} The `ngrambf_v1` index splits strings into n-grams. It works well for `LIKE '%...%'` queries. It supports String/FixedString/Map (via mapKeys/mapValues), as well as tunable size, hash count, and seed. See the documentation for [N-gram bloom filter](/engines/table-engines/mergetree-family/mergetree#n-gram-bloom-filter) for further details. @@ -126,7 +126,7 @@ SELECT bfEstimateFunctions(4300, bfEstimateBmSize(4300, 0.0001)) AS k; -- ~13 See [parameter docs](/engines/table-engines/mergetree-family/mergetree#n-gram-bloom-filter) for complete tuning guidance. -## Token Bloom filter (tokenbf\_v1) for word-based search {#token-bloom-filter-tokenbf-v1-for-word-based-search} +## Token bloom filter (tokenbf\_v1) for word-based search {#token-bloom-filter-tokenbf-v1-for-word-based-search} `tokenbf_v1` indexes tokens separated by non-alphanumeric characters. You should use it with [`hasToken`](/sql-reference/functions/string-search-functions#hasToken), `LIKE` word patterns or equals/IN. It supports `String`/`FixedString`/`Map` types. @@ -145,7 +145,7 @@ SELECT count() FROM logs WHERE hasToken(lower(msg), 'exception'); See observability examples and guidance on token vs ngram [here](/use-cases/observability/schema-design#bloom-filters-for-text-search). -## Add indexes during CREATE TABLE (multiple examples) {#add-indexes-during-create-table-multiple-examples} +## Add indexes during create table (multiple examples) {#add-indexes-during-create-table-multiple-examples} Skipping indexes also support composite expressions and `Map`/`Tuple`/`Nested` types. This is demonstrated in the example below: diff --git a/docs/guides/best-practices/skipping-indexes.md b/docs/guides/best-practices/skipping-indexes.md index 7b8644f4423..28ff93e8d9c 100644 --- a/docs/guides/best-practices/skipping-indexes.md +++ b/docs/guides/best-practices/skipping-indexes.md @@ -118,7 +118,7 @@ example, the debug log shows that the skip index dropped all but two granules: ## Skip index types {#skip-index-types} -### minmax {#minmax} +### Minmax {#minmax} This lightweight index type requires no parameters. It stores the minimum and maximum values of the index expression @@ -128,7 +128,7 @@ of the tuple). This type is ideal for columns that tend to be loosely sorted by This type of index only works correctly with a scalar or tuple expression -- the index will never be applied to expressions that return an array or map data type. -### set {#set} +### Set {#set} This lightweight index type accepts a single parameter of the max_size of the value set per block (0 permits diff --git a/docs/guides/best-practices/sparse-primary-indexes.md b/docs/guides/best-practices/sparse-primary-indexes.md index 6be000ddc0d..bad5f262846 100644 --- a/docs/guides/best-practices/sparse-primary-indexes.md +++ b/docs/guides/best-practices/sparse-primary-indexes.md @@ -864,7 +864,7 @@ In the following we discuss this three options for creating and using multiple p -### Option 1: Secondary Tables {#option-1-secondary-tables} +### Option 1: secondary tables {#option-1-secondary-tables} We are creating a new additional table where we switch the order of the key columns (compared to our original table) in the primary key: @@ -1024,7 +1024,7 @@ Server Log: We now have two tables. Optimized for speeding up queries filtering on `UserIDs`, and speeding up queries filtering on URLs, respectively: -### Option 2: Materialized Views {#option-2-materialized-views} +### Option 2: materialized views {#option-2-materialized-views} Create a [materialized view](/sql-reference/statements/create/view.md) on our existing table. ```sql @@ -1110,7 +1110,7 @@ The corresponding trace log in the ClickHouse server log file confirms that Clic ...Executor): Reading approx. 335872 rows with 4 streams ``` -### Option 3: Projections {#option-3-projections} +### Option 3: projections {#option-3-projections} Create a projection on our existing table: ```sql diff --git a/docs/guides/developer/alternative-query-languages.md b/docs/guides/developer/alternative-query-languages.md index 654ab456388..564dadf0da9 100644 --- a/docs/guides/developer/alternative-query-languages.md +++ b/docs/guides/developer/alternative-query-languages.md @@ -26,7 +26,7 @@ Standard SQL is the default query language of ClickHouse. SET dialect = 'clickhouse' ``` -## Pipelined relational query language (PRQL) {#pipelined-relational-query-language-prql} +## Pipelined relational query language (prql) {#pipelined-relational-query-language-prql} @@ -49,7 +49,7 @@ aggregate { Under the hood, ClickHouse uses transpilation from PRQL to SQL to run PRQL queries. -## Kusto query language (KQL) {#kusto-query-language-kql} +## Kusto query language (kql) {#kusto-query-language-kql} diff --git a/docs/guides/developer/deduplication.md b/docs/guides/developer/deduplication.md index 6424f38aa8d..8749682de91 100644 --- a/docs/guides/developer/deduplication.md +++ b/docs/guides/developer/deduplication.md @@ -39,7 +39,7 @@ Deduplication is implemented in ClickHouse using the following table engines: We walk through both of these techniques below. For more details, check out our free on-demand [Deleting and Updating Data training module](https://learn.clickhouse.com/visitor_catalog_class/show/1328954/?utm_source=clickhouse&utm_medium=docs). -## Using ReplacingMergeTree for Upserts {#using-replacingmergetree-for-upserts} +## Using ReplacingMergeTree for upserts {#using-replacingmergetree-for-upserts} Let's look at a simple example where a table contains Hacker News comments with a views column representing the number of times a comment was viewed. Suppose we insert a new row when an article is published and upsert a new row once a day with the total number of views if the value increases: @@ -111,7 +111,7 @@ using `FINAL` is probably not the best option. Let's discuss a better option for finding the latest value of a column. ::: -### Avoiding FINAL {#avoiding-final} +### Avoiding final {#avoiding-final} Let's update the `views` column again for both unique rows: diff --git a/docs/guides/developer/replacing-merge-tree.md b/docs/guides/developer/replacing-merge-tree.md index 3def2b9a483..fc5c2f501f4 100644 --- a/docs/guides/developer/replacing-merge-tree.md +++ b/docs/guides/developer/replacing-merge-tree.md @@ -219,7 +219,7 @@ FINAL Peak memory usage: 8.14 MiB. ``` -## FINAL performance {#final-performance} +## Final performance {#final-performance} The `FINAL` operator does have a small performance overhead on queries. This will be most noticeable when queries are not filtering on primary key columns, diff --git a/docs/guides/developer/stored-procedures-and-prepared-statements.md b/docs/guides/developer/stored-procedures-and-prepared-statements.md index 8a4453dd7bc..a17bf22af39 100644 --- a/docs/guides/developer/stored-procedures-and-prepared-statements.md +++ b/docs/guides/developer/stored-procedures-and-prepared-statements.md @@ -29,11 +29,11 @@ ClickHouse is optimized for: Stored procedures with procedural logic work against these optimizations. Instead, ClickHouse provides alternatives that align with its strengths. -### User-Defined Functions (UDFs) {#user-defined-functions} +### User-defined functions (udfs) {#user-defined-functions} User-Defined Functions let you encapsulate reusable logic without control flow. ClickHouse supports two types: -#### Lambda-based UDFs {#lambda-based-udfs} +#### Lambda-based udfs {#lambda-based-udfs} Create functions using SQL expressions and lambda syntax: @@ -107,7 +107,7 @@ SELECT format_phone('5551234567'); See [`CREATE FUNCTION`](/sql-reference/statements/create/function) for complete syntax. -#### Executable UDFs {#executable-udfs} +#### Executable udfs {#executable-udfs} For more complex logic, use executable UDFs that call external programs: diff --git a/docs/guides/developer/time-series-filling-gaps.md b/docs/guides/developer/time-series-filling-gaps.md index c3b8ea2cdce..00946eb040c 100644 --- a/docs/guides/developer/time-series-filling-gaps.md +++ b/docs/guides/developer/time-series-filling-gaps.md @@ -77,7 +77,7 @@ ORDER BY bucket ASC The result set only includes the buckets where an image was created, but for time-series analysis, we might want to return each 100ms bucket, even if it doesn't have any entries. -## WITH FILL {#with-fill} +## With fill {#with-fill} We can use the `WITH FILL` clause to fill in these gaps. We'll also specify the `STEP`, which is the size of the gaps to fill. @@ -115,7 +115,7 @@ STEP toIntervalMillisecond(100); We can see that the gaps have been filled with 0 values in the `count` column. -## WITH FILL...FROM {#with-fillfrom} +## With FILL...FROM {#with-fillfrom} There is, however, still a gap at the beginning of the time range, which we can fix by specifying `FROM`: @@ -158,7 +158,7 @@ STEP toIntervalMillisecond(100); We can see from the results that the buckets from `00:24:03.000` to `00:24:03.500` all now appear. -## WITH FILL...TO {#with-fillto} +## With FILL...TO {#with-fillto} We're still missing some buckets from the end of the time range though, which we can fill by providing a `TO` value. `TO` is not inclusive, so we'll add a small amount to the end time to make sure that it's included: @@ -253,7 +253,7 @@ STEP toIntervalMillisecond(100); The values in the cumulative column aren't working how we'd like them to. -## WITH FILL...INTERPOLATE {#with-fillinterpolate} +## With FILL...INTERPOLATE {#with-fillinterpolate} Any rows that have `0` in the `count` column also have `0` in the cumulative column, whereas we'd rather it use the previous value in the `cumulative` column. We can do this by using the `INTERPOLATE` clause, as shown below: diff --git a/docs/guides/examples/aggregate_function_combinators/anyIf.md b/docs/guides/examples/aggregate_function_combinators/anyIf.md index 896d9704f45..7cb84a696fb 100644 --- a/docs/guides/examples/aggregate_function_combinators/anyIf.md +++ b/docs/guides/examples/aggregate_function_combinators/anyIf.md @@ -7,7 +7,7 @@ sidebar_label: 'anyIf' doc_type: 'reference' --- -# anyIf {#avgif} +# Anyif {#avgif} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/argMaxIf.md b/docs/guides/examples/aggregate_function_combinators/argMaxIf.md index bfa860a8680..17668c814fb 100644 --- a/docs/guides/examples/aggregate_function_combinators/argMaxIf.md +++ b/docs/guides/examples/aggregate_function_combinators/argMaxIf.md @@ -7,7 +7,7 @@ sidebar_label: 'argMaxIf' doc_type: 'reference' --- -# argMaxIf {#argmaxif} +# Argmaxif {#argmaxif} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/argMinIf.md b/docs/guides/examples/aggregate_function_combinators/argMinIf.md index 35197ba5297..9aafd603dc4 100644 --- a/docs/guides/examples/aggregate_function_combinators/argMinIf.md +++ b/docs/guides/examples/aggregate_function_combinators/argMinIf.md @@ -7,7 +7,7 @@ sidebar_label: 'argMinIf' doc_type: 'reference' --- -# argMinIf {#argminif} +# Argminif {#argminif} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/avgIf.md b/docs/guides/examples/aggregate_function_combinators/avgIf.md index 535ee31d1da..13fc5e3747d 100644 --- a/docs/guides/examples/aggregate_function_combinators/avgIf.md +++ b/docs/guides/examples/aggregate_function_combinators/avgIf.md @@ -7,7 +7,7 @@ sidebar_label: 'avgIf' doc_type: 'reference' --- -# avgIf {#avgif} +# Avgif {#avgif} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/avgMap.md b/docs/guides/examples/aggregate_function_combinators/avgMap.md index 16a91058834..4bdf60c0941 100644 --- a/docs/guides/examples/aggregate_function_combinators/avgMap.md +++ b/docs/guides/examples/aggregate_function_combinators/avgMap.md @@ -7,7 +7,7 @@ sidebar_label: 'avgMap' doc_type: 'reference' --- -# avgMap {#avgmap} +# Avgmap {#avgmap} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/avgMerge.md b/docs/guides/examples/aggregate_function_combinators/avgMerge.md index e209a3aeb54..4bf8d6d9726 100644 --- a/docs/guides/examples/aggregate_function_combinators/avgMerge.md +++ b/docs/guides/examples/aggregate_function_combinators/avgMerge.md @@ -7,7 +7,7 @@ sidebar_label: 'avgMerge' doc_type: 'reference' --- -# avgMerge {#avgMerge} +# Avgmerge {#avgMerge} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/avgMergeState.md b/docs/guides/examples/aggregate_function_combinators/avgMergeState.md index 2e1de95c91e..78fb255b593 100644 --- a/docs/guides/examples/aggregate_function_combinators/avgMergeState.md +++ b/docs/guides/examples/aggregate_function_combinators/avgMergeState.md @@ -10,7 +10,7 @@ doc_type: 'reference' import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -# avgMergeState {#avgMergeState} +# Avgmergestate {#avgMergeState} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/avgResample.md b/docs/guides/examples/aggregate_function_combinators/avgResample.md index 7e54f873241..b41fc4b447c 100644 --- a/docs/guides/examples/aggregate_function_combinators/avgResample.md +++ b/docs/guides/examples/aggregate_function_combinators/avgResample.md @@ -7,7 +7,7 @@ sidebar_label: 'avgResample' doc_type: 'reference' --- -# countResample {#countResample} +# Countresample {#countResample} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/avgState.md b/docs/guides/examples/aggregate_function_combinators/avgState.md index 1dec7554042..5359b5db3a0 100644 --- a/docs/guides/examples/aggregate_function_combinators/avgState.md +++ b/docs/guides/examples/aggregate_function_combinators/avgState.md @@ -7,7 +7,7 @@ sidebar_label: 'avgState' doc_type: 'reference' --- -# avgState {#avgState} +# Avgstate {#avgState} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/countIf.md b/docs/guides/examples/aggregate_function_combinators/countIf.md index ee5131edf08..22424e25ff7 100644 --- a/docs/guides/examples/aggregate_function_combinators/countIf.md +++ b/docs/guides/examples/aggregate_function_combinators/countIf.md @@ -7,7 +7,7 @@ sidebar_label: 'countIf' doc_type: 'reference' --- -# countIf {#countif} +# Countif {#countif} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/countResample.md b/docs/guides/examples/aggregate_function_combinators/countResample.md index 826e8806fa8..16fb59a1eb0 100644 --- a/docs/guides/examples/aggregate_function_combinators/countResample.md +++ b/docs/guides/examples/aggregate_function_combinators/countResample.md @@ -7,7 +7,7 @@ sidebar_label: 'countResample' doc_type: 'reference' --- -# countResample {#countResample} +# Countresample {#countResample} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/groupArrayDistinct.md b/docs/guides/examples/aggregate_function_combinators/groupArrayDistinct.md index 9f950dde551..7476e42f7fe 100644 --- a/docs/guides/examples/aggregate_function_combinators/groupArrayDistinct.md +++ b/docs/guides/examples/aggregate_function_combinators/groupArrayDistinct.md @@ -7,7 +7,7 @@ sidebar_label: 'groupArrayDistinct' doc_type: 'reference' --- -# groupArrayDistinct {#sumdistinct} +# Grouparraydistinct {#sumdistinct} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/groupArrayResample.md b/docs/guides/examples/aggregate_function_combinators/groupArrayResample.md index c392ffebbfe..dffc89fb314 100644 --- a/docs/guides/examples/aggregate_function_combinators/groupArrayResample.md +++ b/docs/guides/examples/aggregate_function_combinators/groupArrayResample.md @@ -7,7 +7,7 @@ sidebar_label: 'groupArrayResample' doc_type: 'reference' --- -# groupArrayResample {#grouparrayresample} +# Grouparrayresample {#grouparrayresample} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/maxMap.md b/docs/guides/examples/aggregate_function_combinators/maxMap.md index 68fb587d575..adc1ff9f71b 100644 --- a/docs/guides/examples/aggregate_function_combinators/maxMap.md +++ b/docs/guides/examples/aggregate_function_combinators/maxMap.md @@ -7,7 +7,7 @@ sidebar_label: 'maxMap' doc_type: 'reference' --- -# maxMap {#maxmap} +# Maxmap {#maxmap} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/maxSimpleState.md b/docs/guides/examples/aggregate_function_combinators/maxSimpleState.md index 22d5e544001..ec7b95b310b 100644 --- a/docs/guides/examples/aggregate_function_combinators/maxSimpleState.md +++ b/docs/guides/examples/aggregate_function_combinators/maxSimpleState.md @@ -7,7 +7,7 @@ sidebar_label: 'maxSimpleState' doc_type: 'reference' --- -# maxSimpleState {#maxsimplestate} +# Maxsimplestate {#maxsimplestate} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/minMap.md b/docs/guides/examples/aggregate_function_combinators/minMap.md index d73f9a91b25..066316aea3e 100644 --- a/docs/guides/examples/aggregate_function_combinators/minMap.md +++ b/docs/guides/examples/aggregate_function_combinators/minMap.md @@ -7,7 +7,7 @@ sidebar_label: 'minMap' doc_type: 'reference' --- -# minMap {#minmap} +# Minmap {#minmap} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/minSimpleState.md b/docs/guides/examples/aggregate_function_combinators/minSimpleState.md index 0e70bc48160..bd70709fcfa 100644 --- a/docs/guides/examples/aggregate_function_combinators/minSimpleState.md +++ b/docs/guides/examples/aggregate_function_combinators/minSimpleState.md @@ -7,7 +7,7 @@ sidebar_label: 'minSimpleState' doc_type: 'reference' --- -# minSimpleState {#minsimplestate} +# Minsimplestate {#minsimplestate} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/quantilesTimingArrayIf.md b/docs/guides/examples/aggregate_function_combinators/quantilesTimingArrayIf.md index 837c008d03a..33455bbd522 100644 --- a/docs/guides/examples/aggregate_function_combinators/quantilesTimingArrayIf.md +++ b/docs/guides/examples/aggregate_function_combinators/quantilesTimingArrayIf.md @@ -7,7 +7,7 @@ sidebar_label: 'quantilesTimingArrayIf' doc_type: 'reference' --- -# quantilesTimingArrayIf {#quantilestimingarrayif} +# Quantilestimingarrayif {#quantilestimingarrayif} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/quantilesTimingIf.md b/docs/guides/examples/aggregate_function_combinators/quantilesTimingIf.md index 0564ab5eb62..fe29a1431d2 100644 --- a/docs/guides/examples/aggregate_function_combinators/quantilesTimingIf.md +++ b/docs/guides/examples/aggregate_function_combinators/quantilesTimingIf.md @@ -7,7 +7,7 @@ sidebar_label: 'quantilesTimingIf' doc_type: 'reference' --- -# quantilesTimingIf {#quantilestimingif} +# Quantilestimingif {#quantilestimingif} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/sumArray.md b/docs/guides/examples/aggregate_function_combinators/sumArray.md index ff05b916c8e..b282957ba6a 100644 --- a/docs/guides/examples/aggregate_function_combinators/sumArray.md +++ b/docs/guides/examples/aggregate_function_combinators/sumArray.md @@ -7,7 +7,7 @@ sidebar_label: 'sumArray' doc_type: 'reference' --- -# sumArray {#sumarray} +# Sumarray {#sumarray} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/sumForEach.md b/docs/guides/examples/aggregate_function_combinators/sumForEach.md index ffe94333ef4..59ba34e60bd 100644 --- a/docs/guides/examples/aggregate_function_combinators/sumForEach.md +++ b/docs/guides/examples/aggregate_function_combinators/sumForEach.md @@ -7,7 +7,7 @@ sidebar_label: 'sumForEach' doc_type: 'reference' --- -# sumForEach {#sumforeach} +# Sumforeach {#sumforeach} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/sumIf.md b/docs/guides/examples/aggregate_function_combinators/sumIf.md index 624eb0aef1c..4a38c7ef2bf 100644 --- a/docs/guides/examples/aggregate_function_combinators/sumIf.md +++ b/docs/guides/examples/aggregate_function_combinators/sumIf.md @@ -7,7 +7,7 @@ sidebar_label: 'sumIf' doc_type: 'reference' --- -# sumIf {#sumif} +# Sumif {#sumif} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/sumMap.md b/docs/guides/examples/aggregate_function_combinators/sumMap.md index edf49b60d05..c671b425458 100644 --- a/docs/guides/examples/aggregate_function_combinators/sumMap.md +++ b/docs/guides/examples/aggregate_function_combinators/sumMap.md @@ -7,7 +7,7 @@ sidebar_label: 'sumMap' doc_type: 'reference' --- -# sumMap {#summap} +# Summap {#summap} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/sumSimpleState.md b/docs/guides/examples/aggregate_function_combinators/sumSimpleState.md index a6a6d315f9c..7a640808b1d 100644 --- a/docs/guides/examples/aggregate_function_combinators/sumSimpleState.md +++ b/docs/guides/examples/aggregate_function_combinators/sumSimpleState.md @@ -7,7 +7,7 @@ sidebar_label: 'sumSimpleState' doc_type: 'reference' --- -# sumSimpleState {#sumsimplestate} +# Sumsimplestate {#sumsimplestate} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/uniqArray.md b/docs/guides/examples/aggregate_function_combinators/uniqArray.md index 84f9aa59848..a2ddd9b298d 100644 --- a/docs/guides/examples/aggregate_function_combinators/uniqArray.md +++ b/docs/guides/examples/aggregate_function_combinators/uniqArray.md @@ -7,7 +7,7 @@ sidebar_label: 'uniqArray' doc_type: 'reference' --- -# uniqArray {#uniqarray} +# Uniqarray {#uniqarray} ## Description {#description} diff --git a/docs/guides/examples/aggregate_function_combinators/uniqArrayIf.md b/docs/guides/examples/aggregate_function_combinators/uniqArrayIf.md index 38f07ce03f3..fe0d5f7ad78 100644 --- a/docs/guides/examples/aggregate_function_combinators/uniqArrayIf.md +++ b/docs/guides/examples/aggregate_function_combinators/uniqArrayIf.md @@ -7,7 +7,7 @@ sidebar_label: 'uniqArrayIf' doc_type: 'reference' --- -# uniqArrayIf {#uniqarrayif} +# Uniqarrayif {#uniqarrayif} ## Description {#description} diff --git a/docs/guides/joining-tables.md b/docs/guides/joining-tables.md index 2dff656e607..6b9bfd7b783 100644 --- a/docs/guides/joining-tables.md +++ b/docs/guides/joining-tables.md @@ -124,7 +124,7 @@ WHERE (VoteTypeId = 2) AND (PostId IN ( Peak memory usage: 250.66 MiB. ``` -## Choosing a JOIN algorithm {#choosing-a-join-algorithm} +## Choosing a join algorithm {#choosing-a-join-algorithm} ClickHouse supports a number of [join algorithms](https://clickhouse.com/blog/clickhouse-fully-supports-joins-part1). These algorithms typically trade memory usage for performance. The following provides an overview of the ClickHouse join algorithms based on their relative memory consumption and execution time: @@ -148,7 +148,7 @@ A full detailed description of each `JOIN` algorithm can be found [here](https:/ Selecting the appropriate join algorithms depends on whether you are looking to optimize for memory or performance. -## Optimizing JOIN performance {#optimizing-join-performance} +## Optimizing join performance {#optimizing-join-performance} If your key optimization metric is performance and you are looking to execute the join as fast as possible, you can use the following decision tree for choosing the right join algorithm: diff --git a/docs/guides/separation-storage-compute.md b/docs/guides/separation-storage-compute.md index 07c33420513..95823b10e73 100644 --- a/docs/guides/separation-storage-compute.md +++ b/docs/guides/separation-storage-compute.md @@ -30,7 +30,7 @@ Please note that implementing and managing a separation of storage and compute a Do not configure any AWS/GCS life cycle policy. This is not supported and could lead to broken tables. ::: -## 1. Use S3 as a ClickHouse disk {#1-use-s3-as-a-clickhouse-disk} +## 1. use S3 as a ClickHouse disk {#1-use-s3-as-a-clickhouse-disk} ### Creating a disk {#creating-a-disk} @@ -93,7 +93,7 @@ You can now restart the ClickHouse server to have the changes take effect: service clickhouse-server restart ``` -## 2. Create a table backed by S3 {#2-create-a-table-backed-by-s3} +## 2. create a table backed by S3 {#2-create-a-table-backed-by-s3} To test that we've configured the S3 disk properly, we can attempt to create and query a table. @@ -161,7 +161,7 @@ If everything worked successfully, you are now using ClickHouse with separated s S3 bucket example using separation of compute and storage -## 3. Implementing replication for fault tolerance (optional) {#3-implementing-replication-for-fault-tolerance-optional} +## 3. implementing replication for fault tolerance (optional) {#3-implementing-replication-for-fault-tolerance-optional} :::warning Do not configure any AWS/GCS life cycle policy. This is not supported and could lead to broken tables. diff --git a/docs/guides/sre/configuring-ssl.md b/docs/guides/sre/configuring-ssl.md index 6dc687bf3d0..9c94b6d3a5e 100644 --- a/docs/guides/sre/configuring-ssl.md +++ b/docs/guides/sre/configuring-ssl.md @@ -12,7 +12,7 @@ import SelfManaged from '@site/docs/_snippets/_self_managed_only_automated.md'; import configuringSsl01 from '@site/static/images/guides/sre/configuring-ssl_01.png'; import Image from '@theme/IdealImage'; -# Configuring SSL-TLS +# Configuring ssl-tls @@ -24,7 +24,7 @@ TLS implementation is complex and there are many options to consider to ensure a Review this [basic tutorial on certificate usage](https://ubuntu.com/server/docs/security-certificates) for an introductory overview. ::: -## 1. Create a ClickHouse Deployment {#1-create-a-clickhouse-deployment} +## 1. create a ClickHouse deployment {#1-create-a-clickhouse-deployment} This guide was written using Ubuntu 20.04 and ClickHouse installed on the following hosts using the DEB package (using apt). The domain is `marsnet.local`: @@ -38,7 +38,7 @@ This guide was written using Ubuntu 20.04 and ClickHouse installed on the follow View the [Quick Start](/getting-started/install/install.mdx) for more details on how to install ClickHouse. ::: -## 2. Create SSL certificates {#2-create-ssl-certificates} +## 2. create SSL certificates {#2-create-ssl-certificates} :::note Using self-signed certificates are for demonstration purposes only and should not used in production. Certificate requests should be created to be signed by the organization and validated using the CA chain that will be configured in the settings. However, these steps can be used to configure and test settings, then can be replaced by the actual certificates that will be used. ::: @@ -87,7 +87,7 @@ Using self-signed certificates are for demonstration purposes only and should no chnode1.crt: OK ``` -## 3. Create and Configure a directory to store certificates and keys. {#3-create-and-configure-a-directory-to-store-certificates-and-keys} +## 3. create and configure a directory to store certificates and keys. {#3-create-and-configure-a-directory-to-store-certificates-and-keys} :::note This must be done on each node. Use appropriate certificates and keys on each host. @@ -117,7 +117,7 @@ This must be done on each node. Use appropriate certificates and keys on each ho -rw------- 1 clickhouse clickhouse 1131 Apr 12 20:23 marsnet_ca.crt ``` -## 4. Configure the environment with basic clusters using ClickHouse Keeper {#4-configure-the-environment-with-basic-clusters-using-clickhouse-keeper} +## 4. configure the environment with basic clusters using ClickHouse keeper {#4-configure-the-environment-with-basic-clusters-using-clickhouse-keeper} For this deployment environment, the following ClickHouse Keeper settings are used in each node. Each server will have its own ``. (For example, `1` for node `chnode1`, and so on.) @@ -235,7 +235,7 @@ For a full explanation of all options, visit https://clickhouse.com/docs/operati ``` -## 5. Configure SSL-TLS interfaces on ClickHouse nodes {#5-configure-ssl-tls-interfaces-on-clickhouse-nodes} +## 5. configure ssl-tls interfaces on ClickHouse nodes {#5-configure-ssl-tls-interfaces-on-clickhouse-nodes} The settings below are configured in the ClickHouse server `config.xml` 1. Set the display name for the deployment (optional): @@ -340,7 +340,7 @@ The settings below are configured in the ClickHouse server `config.xml` ``` -## 6. Testing {#6-testing} +## 6. testing {#6-testing} 1. Start all nodes, one at a time: ```bash service clickhouse-server start diff --git a/docs/guides/sre/keeper/index.md b/docs/guides/sre/keeper/index.md index 39e18dc7baa..aaa7d1e38fc 100644 --- a/docs/guides/sre/keeper/index.md +++ b/docs/guides/sre/keeper/index.md @@ -9,7 +9,7 @@ title: 'ClickHouse Keeper' doc_type: 'guide' --- -# ClickHouse Keeper (clickhouse-keeper) +# ClickHouse keeper (clickhouse-keeper) import SelfManaged from '@site/docs/_snippets/_self_managed_only_automated.md'; @@ -462,7 +462,7 @@ Some of the feature flags are enabled by default from version 25.7. The recommended way of upgrading Keeper to 25.7+ is to first upgrade to version 24.9+. ::: -### Migration from ZooKeeper {#migration-from-zookeeper} +### Migration from zookeeper {#migration-from-zookeeper} Seamless migration from ZooKeeper to ClickHouse Keeper is not possible. You have to stop your ZooKeeper cluster, convert data, and start ClickHouse Keeper. `clickhouse-keeper-converter` tool allows converting ZooKeeper logs and snapshots to ClickHouse Keeper snapshot. It works only with ZooKeeper > 3.4. Steps for migration: @@ -514,7 +514,7 @@ After making sure that the above things are true, you need to do following: 6. While in the recovery mode, the leader node will return error message for `mntr` command until it achieves quorum with the new nodes and refuse any requests from the client and the followers. 7. After quorum is achieved, the leader node will return to the normal mode of operation, accepting all the requests using Raft-verify with `mntr` which should return `leader` for the `zk_server_state`. -## Using disks with Keeper {#using-disks-with-keeper} +## Using disks with keeper {#using-disks-with-keeper} Keeper supports a subset of [external disks](/operations/storing-data.md) for storing snapshots, log files, and the state file. @@ -676,11 +676,11 @@ curl 127.0.0.1:9363/metrics Please also see the ClickHouse Cloud [Prometheus integration](/integrations/prometheus). -## ClickHouse Keeper user guide {#clickhouse-keeper-user-guide} +## ClickHouse keeper user guide {#clickhouse-keeper-user-guide} This guide provides simple and minimal settings to configure ClickHouse Keeper with an example on how to test distributed operations. This example is performed using 3 nodes on Linux. -### 1. Configure nodes with Keeper settings {#1-configure-nodes-with-keeper-settings} +### 1. configure nodes with keeper settings {#1-configure-nodes-with-keeper-settings} 1. Install 3 ClickHouse instances on 3 hosts (`chnode1`, `chnode2`, `chnode3`). (View the [Quick Start](/getting-started/install/install.mdx) for details on installing ClickHouse.) @@ -784,7 +784,7 @@ This guide provides simple and minimal settings to configure ClickHouse Keeper w └────────────┴───────┴───────┴───────┴─────────────────────┴─────────────────────┴─────────┴──────────┴──────────┴────────────────┴────────────┴─────────────┴───────┴─────────────┘ ``` -### 2. Configure a cluster in ClickHouse {#2--configure-a-cluster-in-clickhouse} +### 2. configure a cluster in ClickHouse {#2--configure-a-cluster-in-clickhouse} 1. Let's configure a simple cluster with 2 shards and only one replica on 2 of the nodes. The third node will be used to achieve a quorum for the requirement in ClickHouse Keeper. Update the configuration on `chnode1` and `chnode2`. The following cluster defines 1 shard on each node for a total of 2 shards with no replication. In this example, some of the data will be on node and some will be on the other node: ```xml @@ -831,7 +831,7 @@ This guide provides simple and minimal settings to configure ClickHouse Keeper w └───────────────┘ ``` -### 3. Create and test distributed table {#3-create-and-test-distributed-table} +### 3. create and test distributed table {#3-create-and-test-distributed-table} 1. Create a new database on the new cluster using ClickHouse client on `chnode1`. The `ON CLUSTER` clause automatically creates the database on both nodes. ```sql @@ -934,7 +934,7 @@ This guide provides simple and minimal settings to configure ClickHouse Keeper w This guide demonstrated how to set up a cluster using ClickHouse Keeper. With ClickHouse Keeper, you can configure clusters and define distributed tables that can be replicated across shards. -## Configuring ClickHouse Keeper with unique paths {#configuring-clickhouse-keeper-with-unique-paths} +## Configuring ClickHouse keeper with unique paths {#configuring-clickhouse-keeper-with-unique-paths} @@ -1235,7 +1235,7 @@ Query id: b047d459-a1d2-4016-bcf9-3e97e30e49c2 1 row in set. Elapsed: 0.004 sec. ``` -## ClickHouse Keeper dynamic reconfiguration {#reconfiguration} +## ClickHouse keeper dynamic reconfiguration {#reconfiguration} diff --git a/docs/guides/sre/user-management/configuring-ldap.md b/docs/guides/sre/user-management/configuring-ldap.md index 40d9b015c53..3aea57d7a5b 100644 --- a/docs/guides/sre/user-management/configuring-ldap.md +++ b/docs/guides/sre/user-management/configuring-ldap.md @@ -10,13 +10,13 @@ doc_type: 'guide' import SelfManaged from '@site/docs/_snippets/_self_managed_only_no_roadmap.md'; -# Configuring ClickHouse to use LDAP for authentication and role mapping +# Configuring ClickHouse to use ldap for authentication and role mapping ClickHouse can be configured to use LDAP to authenticate ClickHouse database users. This guide provides a simple example of integrating ClickHouse with an LDAP system authenticating to a publicly available directory. -## 1. Configure LDAP connection settings in ClickHouse {#1-configure-ldap-connection-settings-in-clickhouse} +## 1. configure ldap connection settings in ClickHouse {#1-configure-ldap-connection-settings-in-clickhouse} 1. Test your connection to this public LDAP server: ```bash @@ -113,7 +113,7 @@ ClickHouse can be configured to use LDAP to authenticate ClickHouse database use 4. Restart your ClickHouse server to apply the settings. -## 2. Configure ClickHouse database roles and permissions {#2-configure-clickhouse-database-roles-and-permissions} +## 2. configure ClickHouse database roles and permissions {#2-configure-clickhouse-database-roles-and-permissions} :::note The procedures in this section assumes that SQL Access Control and Account Management in ClickHouse has been enabled. To enable, view the [SQL Users and Roles guide](index.md). @@ -129,7 +129,7 @@ The procedures in this section assumes that SQL Access Control and Account Manag GRANT ALL ON *.* TO scientists_role; ``` -## 3. Test the LDAP configuration {#3-test-the-ldap-configuration} +## 3. test the ldap configuration {#3-test-the-ldap-configuration} 1. Login using the ClickHouse client ```bash diff --git a/docs/guides/sre/user-management/index.md b/docs/guides/sre/user-management/index.md index 170a3cc0358..e97088f2423 100644 --- a/docs/guides/sre/user-management/index.md +++ b/docs/guides/sre/user-management/index.md @@ -150,7 +150,7 @@ Management queries: - [SHOW QUOTA](/sql-reference/statements/show#show-quota) - [SHOW QUOTAS](/sql-reference/statements/show#show-quotas) -### Enabling SQL-driven access control and account management {#enabling-access-control} +### Enabling sql-driven access control and account management {#enabling-access-control} - Setup a directory for configuration storage. diff --git a/docs/guides/sre/user-management/ssl-user-auth.md b/docs/guides/sre/user-management/ssl-user-auth.md index 72578a1fc79..eeb8944ca66 100644 --- a/docs/guides/sre/user-management/ssl-user-auth.md +++ b/docs/guides/sre/user-management/ssl-user-auth.md @@ -25,7 +25,7 @@ If you use AWS NLB with the MySQL interface, you have to ask AWS support to enab > I would like to be able to configure our NLB proxy protocol v2 as below `proxy_protocol_v2.client_to_server.header_placement,Value=on_first_ack`. ::: -## 1. Create SSL user certificates {#1-create-ssl-user-certificates} +## 1. create SSL user certificates {#1-create-ssl-user-certificates} :::note This example uses self-signed certificates with a self-signed CA. For production environments, create a CSR and submit to your PKI team or certificate provider to obtain a proper certificate. @@ -52,7 +52,7 @@ This example uses self-signed certificates with a self-signed CA. For production openssl x509 -req -in chnode1_cert_user.csr -out chnode1_cert_user.crt -CA marsnet_ca.crt -CAkey marsnet_ca.key -days 365 ``` -## 2. Create a SQL user and grant permissions {#2-create-a-sql-user-and-grant-permissions} +## 2. create a SQL user and grant permissions {#2-create-a-sql-user-and-grant-permissions} :::note For details on how to enable SQL users and set roles, refer to [Defining SQL Users and Roles](index.md) user guide. @@ -90,7 +90,7 @@ For details on how to enable SQL users and set roles, refer to [Defining SQL Use ``` ::: -## 3. Testing {#3-testing} +## 3. testing {#3-testing} 1. Copy the user certificate, user key and CA certificate to a remote node. @@ -114,7 +114,7 @@ For details on how to enable SQL users and set roles, refer to [Defining SQL Use Note that the password passed to clickhouse-client is ignored when a certificate is specified in the config. ::: -## 4. Testing HTTP {#4-testing-http} +## 4. testing HTTP {#4-testing-http} 1. Copy the user certificate, user key and CA certificate to a remote node. diff --git a/docs/guides/starter_guides/generating-test-data.md b/docs/guides/starter_guides/generating-test-data.md index de1764d5c29..b89b7877b61 100644 --- a/docs/guides/starter_guides/generating-test-data.md +++ b/docs/guides/starter_guides/generating-test-data.md @@ -68,7 +68,7 @@ FROM numbers(500000); --- -## Time-distributed events (Poisson) {#poisson-distribution} +## Time-distributed events (poisson) {#poisson-distribution} **Use-case**: Simulate event arrivals that cluster around a specific period (e.g., peak hour). diff --git a/docs/guides/starter_guides/inserting-data.md b/docs/guides/starter_guides/inserting-data.md index 32241da4e02..773506a09e6 100644 --- a/docs/guides/starter_guides/inserting-data.md +++ b/docs/guides/starter_guides/inserting-data.md @@ -24,7 +24,7 @@ These transactions can potentially involve a small number of rows at a time, wit To achieve high insert performance while maintaining strong consistency guarantees, users should adhere to the simple rules described below when inserting data into ClickHouse. Following these rules will help to avoid issues users commonly encounter the first time they use ClickHouse, and try to replicate an insert strategy that works for OLTP databases. -## Best practices for Inserts {#best-practices-for-inserts} +## Best practices for inserts {#best-practices-for-inserts} ### Insert in large batch sizes {#insert-in-large-batch-sizes} @@ -237,7 +237,7 @@ clickhouse-client --query "SELECT formatReadableQuantity(count(*)) FROM hackerne 28.74 million ``` -### inserting data via command line with curl {#insert-using-curl} +### Inserting data via command line with curl {#insert-using-curl} In the previous steps you first downloaded the csv file to your local machine using `wget`. It is also possible to directly insert the data from the remote URL using a single command. diff --git a/docs/guides/starter_guides/working_with_arrays.md b/docs/guides/starter_guides/working_with_arrays.md index 93d005b84e2..42754a7fe30 100644 --- a/docs/guides/starter_guides/working_with_arrays.md +++ b/docs/guides/starter_guides/working_with_arrays.md @@ -166,7 +166,7 @@ SELECT string_array[0] ClickHouse offers a host of useful functions which operate on arrays. In this section, we'll look at some of the most useful ones, starting from the simplest and working up in complexity. -### length, arrayEnumerate, indexOf, has* functions {#length-arrayEnumerate-indexOf-has-functions} +### Length, arrayenumerate, indexof, has* functions {#length-arrayEnumerate-indexOf-has-functions} The `length` function is used to return the number of elements in the array: @@ -242,7 +242,7 @@ otherwise complex queries. Click the "play" button below to run the queries directly in the docs and see the result live. ::: -### groupArray {#grouparray} +### Grouparray {#grouparray} There are many columns in this dataset, but we will focus on a subset of the columns. Run the query below to see what our data looks like: @@ -310,7 +310,7 @@ FROM busy_airports ORDER BY outward_flights DESC ``` -### arrayMap and arrayZip {#arraymap} +### Arraymap and arrayzip {#arraymap} We saw in the previous query that Denver International Airport was the airport with the most outward flights for our particular chosen day. Let's take a look at how many of those flights were on-time, delayed by 15-30 minutes or delayed by more than 30 minutes. @@ -343,7 +343,7 @@ In the above query, the `arrayMap` function takes a single-element array `[DepDe Then the first element of the resulting array is extracted with `[DepDelayMinutes][1]`. The [`arrayZip`](/sql-reference/functions/array-functions#arrayZip) function combines the `Tail_Number` array and the `statuses` array into a single array. -### arrayFilter {#arrayfilter} +### Arrayfilter {#arrayfilter} Next we'll look only at the number of flights that were delayed by 30 minutes or more, for airports `DEN`, `ATL` and `DFW`: @@ -367,7 +367,7 @@ This lambda function itself takes the delay in minutes (d) and returns `1` if th d -> d >= 30 ``` -### arraySort and arrayIntersect {#arraysort-and-arrayintersect} +### Arraysort and arrayintersect {#arraysort-and-arrayintersect} Next, we'll figure out which pairs of major US airports serve the most common destinations with the help of the [`arraySort`](/sql-reference/functions/array-functions#arraySort) and [`arrayIntersect`](/sql-reference/functions/array-functions#arrayIntersect) functions. `arraySort` takes an array and sorts the elements in ascending order by default, although you can also pass a lambda function to it to define the sorting order. @@ -413,7 +413,7 @@ Without this, you'd get both JFK-LAX and LAX-JFK as separate results, which woul Finally, the query sorts the results to show which airport pairs have the highest number of shared destinations and returns just the top 10. This reveals which major hubs have the most overlapping route networks, which could indicate competitive markets where multiple airlines are serving the same city pairs, or hubs that serve similar geographic regions and could potentially be used as alternative connection points for travelers. -### arrayReduce {#arrayReduce} +### Arrayreduce {#arrayReduce} While we're looking at delays, let's use yet another higher-order array function, `arrayReduce`, to find the average and maximum delay for each route from Denver International Airport: @@ -438,7 +438,7 @@ ORDER BY avg_delay DESC In the example above, we used `arrayReduce` to find the average and maximum delays for various outward flights from `DEN`. `arrayReduce` applies an aggregate function, specified in the first parameter to the function, to the elements of the provided array, specified in the second parameter of the function. -### arrayJoin {#arrayJoin} +### Arrayjoin {#arrayJoin} Regular functions in ClickHouse have the property that they return the same number of rows than they receive. There is however, one interesting and unique function that breaks this rule, which is worth learning about - the `arrayJoin` function. diff --git a/docs/guides/starter_guides/working_with_joins.md b/docs/guides/starter_guides/working_with_joins.md index c7ed8f0c31e..69e0d70f11b 100644 --- a/docs/guides/starter_guides/working_with_joins.md +++ b/docs/guides/starter_guides/working_with_joins.md @@ -21,7 +21,7 @@ import asof_example from '@site/static/images/starter_guides/joins/asof_example. ClickHouse fully supports standard SQL joins, enabling efficient data analysis. In this guide, you'll explore some of the available commonly used join types and how to use them with the help of Venn diagrams and example queries on a normalized [IMDB](https://en.wikipedia.org/wiki/IMDb) dataset originating from the [relational dataset repository](https://relational.fit.cvut.cz/dataset/IMDb). -## Test Data and Resources {#test-data-and-resources} +## Test data and resources {#test-data-and-resources} Instructions for creating and loading the tables can be found [here](/integrations/dbt/guides). The dataset is also available in the [playground](https://sql.clickhouse.com?query_id=AACTS8ZBT3G7SSGN8ZJBJY) for users who don't want to create and load @@ -54,7 +54,7 @@ ClickHouse supports the following join types: You'll write example queries for each of the JOIN types above in the following sections. -## INNER JOIN {#inner-join} +## Inner join {#inner-join} The `INNER JOIN` returns, for each pair of rows matching on join keys, the column values of the row from the left table, combined with the column values of the row from the right table. If a row has more than one match, then all matches are returned (meaning that the [cartesian product](https://en.wikipedia.org/wiki/Cartesian_product) is produced for rows with matching join keys). @@ -97,7 +97,7 @@ The `INNER` keyword can be omitted. The behavior of the `INNER JOIN` can be extended or changed, by using one of the following other join types. -## (LEFT / RIGHT / FULL) OUTER JOIN {#left--right--full-outer-join} +## (Left / right / full) outer join {#left--right--full-outer-join} The `LEFT OUTER JOIN` behaves like `INNER JOIN`; plus, for non-matching left table rows, ClickHouse returns [default values](/sql-reference/statements/create/table#default_values) for the right table’s columns. @@ -143,7 +143,7 @@ LIMIT 10; The `OUTER` keyword can be omitted. ::: -## CROSS JOIN {#cross-join} +## Cross join {#cross-join} The `CROSS JOIN` produces the full cartesian product of the two tables without considering join keys. Each row from the left table is combined with each row from the right table. @@ -239,7 +239,7 @@ ALL And because, as mentioned above, the `OUTER` keyword can be omitted for a `RIGHT OUTER JOIN`, and the optional `ALL` keyword can be added, you can write `ALL RIGHT JOIN` and it will work all right. -## (LEFT / RIGHT) SEMI JOIN {#left--right-semi-join} +## (Left / right) semi join {#left--right-semi-join} A `LEFT SEMI JOIN` query returns column values for each row from the left table that has at least one join key match in the right table. Only the first found match is returned (the cartesian product is disabled). @@ -277,7 +277,7 @@ LIMIT 10; └────────────┴────────────────────────┘ ``` -## (LEFT / RIGHT) ANTI JOIN {#left--right-anti-join} +## (Left / right) anti join {#left--right-anti-join} A `LEFT ANTI JOIN` returns column values for all non-matching rows from the left table. @@ -312,7 +312,7 @@ LIMIT 10; └───────────────────────────────────────────┘ ``` -## (LEFT / RIGHT / INNER) ANY JOIN {#left--right--inner-any-join} +## (Left / right / inner) any join {#left--right--inner-any-join} A `LEFT ANY JOIN` is the combination of the `LEFT OUTER JOIN` + the `LEFT SEMI JOIN`, meaning that ClickHouse returns column values for each row from the left table, either combined with the column values of a matching row from the right table or combined with default column values for the right table, in case no match exists. If a row from the left table has more than one match in the right table, ClickHouse only returns the combined column values from the first found match (the cartesian product is disabled). @@ -387,7 +387,7 @@ INNER ANY JOIN right_table AS r ON l.c = r.c; └─────┴─────┘ ``` -## ASOF JOIN {#asof-join} +## Asof join {#asof-join} The `ASOF JOIN`, provides non-exact matching capabilities. If a row from the left table doesn’t have an exact match in the right table, then the closest matching row from the right table is used as a match instead. diff --git a/docs/guides/troubleshooting.md b/docs/guides/troubleshooting.md index c79badc8abc..8e678f75f32 100644 --- a/docs/guides/troubleshooting.md +++ b/docs/guides/troubleshooting.md @@ -8,11 +8,11 @@ keywords: ['troubleshooting', 'debugging', 'problem solving', 'errors', 'diagnos ## Installation {#installation} -### Cannot import GPG keys from keyserver.ubuntu.com with apt-key {#cannot-import-gpg-keys-from-keyserverubuntucom-with-apt-key} +### Cannot import gpg keys from keyserver.ubuntu.com with apt-key {#cannot-import-gpg-keys-from-keyserverubuntucom-with-apt-key} The `apt-key` feature with the [Advanced package tool (APT) has been deprecated](https://manpages.debian.org/bookworm/apt/apt-key.8.en.html). Users should use the `gpg` command instead. Please refer the [install guide](../getting-started/install/install.mdx) article. -### Cannot import GPG keys from keyserver.ubuntu.com with gpg {#cannot-import-gpg-keys-from-keyserverubuntucom-with-gpg} +### Cannot import gpg keys from keyserver.ubuntu.com with gpg {#cannot-import-gpg-keys-from-keyserverubuntucom-with-gpg} 1. See if your `gpg` is installed: @@ -63,7 +63,7 @@ sudo apt-get clean sudo apt-get autoclean ``` -### Can't get packages with Yum because of wrong signature {#cant-get-packages-with-yum-because-of-wrong-signature} +### Can't get packages with yum because of wrong signature {#cant-get-packages-with-yum-because-of-wrong-signature} Possible issue: the cache is wrong, maybe it's broken after updated GPG key in 2022-09. diff --git a/docs/integrations/data-ingestion/apache-spark/index.md b/docs/integrations/data-ingestion/apache-spark/index.md index 6f68e30663e..01192b72068 100644 --- a/docs/integrations/data-ingestion/apache-spark/index.md +++ b/docs/integrations/data-ingestion/apache-spark/index.md @@ -16,7 +16,7 @@ import TabItem from '@theme/TabItem'; import TOCInline from '@theme/TOCInline'; import ClickHouseSupportedBadge from '@theme/badges/ClickHouseSupported'; -# Integrating Apache Spark with ClickHouse +# Integrating Apache spark with ClickHouse diff --git a/docs/integrations/data-ingestion/apache-spark/spark-jdbc.md b/docs/integrations/data-ingestion/apache-spark/spark-jdbc.md index 06551f82d9d..f392fc270a4 100644 --- a/docs/integrations/data-ingestion/apache-spark/spark-jdbc.md +++ b/docs/integrations/data-ingestion/apache-spark/spark-jdbc.md @@ -13,7 +13,7 @@ import TabItem from '@theme/TabItem'; import TOCInline from '@theme/TOCInline'; import ClickHouseSupportedBadge from '@theme/badges/ClickHouseSupported'; -# Spark JDBC +# Spark jdbc @@ -340,7 +340,7 @@ reading in parallel from multiple workers. Please visit Apache Spark's official documentation for more information on [JDBC configurations](https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html#data-source-option). -## JDBC limitations {#jdbc-limitations} +## Jdbc limitations {#jdbc-limitations} * As of today, you can insert data using JDBC only into existing tables (currently there is no way to auto create the table on DF insertion, as Spark does with other connectors). diff --git a/docs/integrations/data-ingestion/apache-spark/spark-native-connector.md b/docs/integrations/data-ingestion/apache-spark/spark-native-connector.md index 9c9f6f883dd..2934bdc71e6 100644 --- a/docs/integrations/data-ingestion/apache-spark/spark-native-connector.md +++ b/docs/integrations/data-ingestion/apache-spark/spark-native-connector.md @@ -60,7 +60,7 @@ Alternatively, you can put the required JAR files in your `$SPARK_HOME/jars/` fo option using the `--jars` flag in the `spark-submit` command. Both approaches ensure the ClickHouse connector is available in your Spark environment. -### Import as a Dependency {#import-as-a-dependency} +### Import as a dependency {#import-as-a-dependency} @@ -539,7 +539,7 @@ The following are the adjustable configurations available in the connector: This section outlines the mapping of data types between Spark and ClickHouse. The tables below provide quick references for converting data types when reading from ClickHouse into Spark and when inserting data from Spark into ClickHouse. -### Reading data from ClickHouse into Spark {#reading-data-from-clickhouse-into-spark} +### Reading data from ClickHouse into spark {#reading-data-from-clickhouse-into-spark} | ClickHouse Data Type | Spark Data Type | Supported | Is Primitive | Notes | |-------------------------------------------------------------------|--------------------------------|-----------|--------------|----------------------------------------------------| @@ -578,7 +578,7 @@ for converting data types when reading from ClickHouse into Spark and when inser | `AggregateFunction` | | ❌ | | | | `SimpleAggregateFunction` | | ❌ | | | -### Inserting data from Spark into ClickHouse {#inserting-data-from-spark-into-clickhouse} +### Inserting data from spark into ClickHouse {#inserting-data-from-spark-into-clickhouse} | Spark Data Type | ClickHouse Data Type | Supported | Is Primitive | Notes | |-------------------------------------|----------------------|-----------|--------------|----------------------------------------| diff --git a/docs/integrations/data-ingestion/aws-glue/index.md b/docs/integrations/data-ingestion/aws-glue/index.md index ab2ab3b9e3e..47e0c8380fc 100644 --- a/docs/integrations/data-ingestion/aws-glue/index.md +++ b/docs/integrations/data-ingestion/aws-glue/index.md @@ -15,7 +15,7 @@ import notebook_connections_config from '@site/static/images/integrations/data-i import dependent_jars_path_option from '@site/static/images/integrations/data-ingestion/aws-glue/dependent_jars_path_option.png'; import ClickHouseSupportedBadge from '@theme/badges/ClickHouseSupported'; -# Integrating Amazon Glue with ClickHouse and Spark +# Integrating Amazon glue with ClickHouse and spark @@ -87,7 +87,7 @@ object ClickHouseGlueExample { .config("spark.sql.catalog.clickhouse.user", "default") .config("spark.sql.catalog.clickhouse.password", "") .config("spark.sql.catalog.clickhouse.database", "default") - // for ClickHouse cloud + // for ClickHouse Cloud .config("spark.sql.catalog.clickhouse.option.ssl", "true") .config("spark.sql.catalog.clickhouse.option.ssl_mode", "NONE") .getOrCreate @@ -163,7 +163,7 @@ spark.conf.set("spark.sql.catalog.clickhouse.password", "") spark.conf.set("spark.sql.catalog.clickhouse.database", "default") spark.conf.set("spark.clickhouse.write.format", "json") spark.conf.set("spark.clickhouse.read.format", "arrow") -# for ClickHouse cloud +# for ClickHouse Cloud spark.conf.set("spark.sql.catalog.clickhouse.option.ssl", "true") spark.conf.set("spark.sql.catalog.clickhouse.option.ssl_mode", "NONE") diff --git a/docs/integrations/data-ingestion/azure-data-factory/overview.md b/docs/integrations/data-ingestion/azure-data-factory/overview.md index 32afbddfccb..b0ef3794859 100644 --- a/docs/integrations/data-ingestion/azure-data-factory/overview.md +++ b/docs/integrations/data-ingestion/azure-data-factory/overview.md @@ -9,7 +9,7 @@ doc_type: 'guide' import ClickHouseSupportedBadge from '@theme/badges/ClickHouseSupported'; -# Bringing Azure Data into ClickHouse +# Bringing Azure data into ClickHouse diff --git a/docs/integrations/data-ingestion/azure-data-factory/using_azureblobstorage.md b/docs/integrations/data-ingestion/azure-data-factory/using_azureblobstorage.md index 1ab31546e19..f17e4008bf8 100644 --- a/docs/integrations/data-ingestion/azure-data-factory/using_azureblobstorage.md +++ b/docs/integrations/data-ingestion/azure-data-factory/using_azureblobstorage.md @@ -11,7 +11,7 @@ import Image from '@theme/IdealImage'; import azureDataStoreSettings from '@site/static/images/integrations/data-ingestion/azure-data-factory/azure-data-store-settings.png'; import azureDataStoreAccessKeys from '@site/static/images/integrations/data-ingestion/azure-data-factory/azure-data-store-access-keys.png'; -# Using ClickHouse's azureBlobStorage table function {#using-azureBlobStorage-function} +# Using clickhouse's azureblobstorage table function {#using-azureBlobStorage-function} This is one of the most efficient and straightforward ways to copy data from Azure Blob Storage or Azure Data Lake Storage into ClickHouse. With this table @@ -29,7 +29,7 @@ for using this function effectively. For more details and advanced options, refer to the official documentation: [`azureBlobStorage` Table Function documentation page](https://clickhouse.com/docs/sql-reference/table-functions/azureBlobStorage) -## Acquiring Azure Blob Storage Access Keys {#acquiring-azure-blob-storage-access-keys} +## Acquiring Azure blob storage access keys {#acquiring-azure-blob-storage-access-keys} To allow ClickHouse to access your Azure Blob Storage, you'll need a connection string with an access key. @@ -45,7 +45,7 @@ To allow ClickHouse to access your Azure Blob Storage, you'll need a connection 4. Copy the connection string — you'll use this as a parameter in the azureBlobStorage table function. -## Querying the data from Azure Blob Storage {#querying-the-data-from-azure-blob-storage} +## Querying the data from Azure blob storage {#querying-the-data-from-azure-blob-storage} Open your preferred ClickHouse query console — this can be the ClickHouse Cloud web interface, the ClickHouse CLI client, or any other tool you use to run @@ -78,7 +78,7 @@ SELECT * FROM azureBlobStorage( This allows you to efficiently pull external data into ClickHouse without needing intermediate ETL steps. -## A simple example using the Environmental sensors dataset {#simple-example-using-the-environmental-sensors-dataset} +## A simple example using the environmental sensors dataset {#simple-example-using-the-environmental-sensors-dataset} As an example we will download a single file from the Environmental Sensors Dataset. diff --git a/docs/integrations/data-ingestion/azure-data-factory/using_http_interface.md b/docs/integrations/data-ingestion/azure-data-factory/using_http_interface.md index a64060de294..847229ce136 100644 --- a/docs/integrations/data-ingestion/azure-data-factory/using_http_interface.md +++ b/docs/integrations/data-ingestion/azure-data-factory/using_http_interface.md @@ -146,7 +146,7 @@ and move to the next one using your existing service. Once the deployment completes successfully, you can start using your new Azure Data Factory instance. -## Creating a new REST-Based linked service {#-creating-new-rest-based-linked-service} +## Creating a new rest-based linked service {#-creating-new-rest-based-linked-service} 1. Log in to the Microsoft Azure Portal and open your Data Factory instance. Azure Portal Home Page with Data Factory @@ -197,7 +197,7 @@ Data Factory instance. You should now see your newly registered REST-based linked service in the list. -## Creating a new dataset for the ClickHouse HTTP Interface {#creating-a-new-dataset-for-the-clickhouse-http-interface} +## Creating a new dataset for the ClickHouse HTTP interface {#creating-a-new-dataset-for-the-clickhouse-http-interface} Now that we have a linked service configured for the ClickHouse HTTP interface, we can create a dataset that Azure Data Factory will use to send data to @@ -280,7 +280,7 @@ service), Then, in Azure Data Factory Studio, go to the Factory Resources pane. Create a new dataset that points to the uploaded data. Click Publish all to save your changes. -## Creating a Copy Activity to transfer data to ClickHouse {#creating-the-copy-activity-to-transfer-data-to-clickhouse} +## Creating a copy activity to transfer data to ClickHouse {#creating-the-copy-activity-to-transfer-data-to-clickhouse} Now that we've configured both the input and output datasets, we can set up a **Copy Data** activity to transfer data from our example dataset into the diff --git a/docs/integrations/data-ingestion/azure-synapse/index.md b/docs/integrations/data-ingestion/azure-synapse/index.md index 6f0385b9a4f..f0f26cc1c6a 100644 --- a/docs/integrations/data-ingestion/azure-synapse/index.md +++ b/docs/integrations/data-ingestion/azure-synapse/index.md @@ -13,7 +13,7 @@ import sparkConfigViaNotebook from '@site/static/images/integrations/data-ingest import sparkUICHSettings from '@site/static/images/integrations/data-ingestion/azure-synapse/spark_ui_ch_settings.png'; import ClickHouseSupportedBadge from '@theme/badges/ClickHouseSupported'; -# Integrating Azure Synapse with ClickHouse +# Integrating Azure synapse with ClickHouse diff --git a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md index 881a92a43e9..bc609d03c0d 100644 --- a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md +++ b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md @@ -18,7 +18,7 @@ import cp_rpe_settings0 from '@site/static/images/integrations/data-ingestion/cl import cp_rpe_settings1 from '@site/static/images/integrations/data-ingestion/clickpipes/cp_rpe_settings1.png'; import Image from '@theme/IdealImage'; -# AWS PrivateLink for ClickPipes +# AWS privatelink for ClickPipes You can use [AWS PrivateLink](https://aws.amazon.com/privatelink/) to establish secure connectivity between VPCs, AWS services, your on-premises systems, and ClickHouse Cloud without exposing traffic to the public Internet. @@ -35,7 +35,7 @@ data source types: - MySQL - MongoDB -## Supported AWS PrivateLink endpoint types {#aws-privatelink-endpoint-types} +## Supported AWS privatelink endpoint types {#aws-privatelink-endpoint-types} ClickPipes reverse private endpoint can be configured with one of the following AWS PrivateLink approaches: @@ -93,7 +93,7 @@ aws vpc-lattice get-resource-gateway \ --resource-gateway-identifier ``` -#### Create a VPC Resource-Configuration {#create-resource-configuration} +#### Create a VPC resource-configuration {#create-resource-configuration} Resource-Configuration is associated with resource gateway to make your resource accessible. @@ -129,7 +129,7 @@ For more information, see the [AWS documentation](https://docs.aws.amazon.com/vp The output will contain a Resource-Configuration ARN, which you will need for the next step. It will also contain a Resource-Configuration ID, which you will need to set up a ClickPipe connection with VPC resource. -#### Create a Resource-Share {#create-resource-share} +#### Create a resource-share {#create-resource-share} Sharing your resource requires a Resource-Share. This is facilitated through the Resource Access Manager (RAM). @@ -153,7 +153,7 @@ For more details on PrivateLink with VPC resource, see [AWS documentation](https -### MSK multi-VPC connectivity {#msk-multi-vpc} +### Msk multi-vpc connectivity {#msk-multi-vpc} The [Multi-VPC connectivity](https://docs.aws.amazon.com/msk/latest/developerguide/aws-access-mult-vpc.html) is a built-in feature of AWS MSK that allows you to connect multiple VPCs to a single MSK cluster. Private DNS support is out of the box and does not require any additional configuration. diff --git a/docs/integrations/data-ingestion/clickpipes/index.md b/docs/integrations/data-ingestion/clickpipes/index.md index 643ca5a902e..076ecc11ba9 100644 --- a/docs/integrations/data-ingestion/clickpipes/index.md +++ b/docs/integrations/data-ingestion/clickpipes/index.md @@ -48,14 +48,14 @@ import Image from '@theme/IdealImage'; | Google Cloud Storage | |Object Storage| Stable | Configure ClickPipes to ingest large volumes of data from object storage. | | DigitalOcean Spaces | | Object Storage | Stable | Configure ClickPipes to ingest large volumes of data from object storage. | Azure Blob Storage | | Object Storage | Stable | Configure ClickPipes to ingest large volumes of data from object storage. -| [Amazon Kinesis](/integrations/clickpipes/kinesis) | |Streaming| Stable | Configure ClickPipes and start ingesting streaming data from Amazon Kinesis into ClickHouse cloud. | +| [Amazon Kinesis](/integrations/clickpipes/kinesis) | |Streaming| Stable | Configure ClickPipes and start ingesting streaming data from Amazon Kinesis into ClickHouse Cloud. | | [Postgres](/integrations/clickpipes/postgres) | |DBMS| Stable | Configure ClickPipes and start ingesting data from Postgres into ClickHouse Cloud. | | [MySQL](/integrations/clickpipes/mysql) | |DBMS| Public Beta | Configure ClickPipes and start ingesting data from MySQL into ClickHouse Cloud. | | [MongoDB](/integrations/clickpipes/mongodb) | |DBMS| Private Preview | Configure ClickPipes and start ingesting data from MongoDB into ClickHouse Cloud. | More connectors will get added to ClickPipes, you can find out more by [contacting us](https://clickhouse.com/company/contact?loc=clickpipes). -## List of Static IPs {#list-of-static-ips} +## List of static ips {#list-of-static-ips} The following are the static NAT IPs (separated by region) that ClickPipes uses to connect to your external services. Add your related instance region IPs to your IP allow list to allow traffic. @@ -90,7 +90,7 @@ Steps: ## Adjusting ClickPipes advanced settings {#clickpipes-advanced-settings} ClickPipes provides sensible defaults that cover the requirements of most use cases. If your use case requires additional fine-tuning, you can adjust the following settings: -### Object Storage ClickPipes {#clickpipes-advanced-settings-object-storage} +### Object storage ClickPipes {#clickpipes-advanced-settings-object-storage} | Setting | Default value | Description | |------------------------------------|---------------|---------------------------------------------------------------------------------------| @@ -115,9 +115,9 @@ ClickPipes provides sensible defaults that cover the requirements of most use ca ## Error reporting {#error-reporting} ClickPipes will store errors in two separate tables depending on the type of error encountered during the ingestion process. -### Record Errors {#record-errors} +### Record errors {#record-errors} ClickPipes will create a table next to your destination table with the postfix `_clickpipes_error`. This table will contain any errors from malformed data or mismatched schema and will include the entirety of the invalid message. This table has a [TTL](/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-ttl) of 7 days. -### System Errors {#system-errors} +### System errors {#system-errors} Errors related to the operation of the ClickPipe will be stored in the `system.clickpipes_log` table. This will store all other errors related to the operation of your ClickPipe (network, connectivity, etc.). This table has a [TTL](/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-ttl) of 7 days. If ClickPipes cannot connect to a data source after 15 min or to a destination after 1 hr, the ClickPipes instance stops and stores an appropriate message in the system error table (provided the ClickHouse instance is available). diff --git a/docs/integrations/data-ingestion/clickpipes/kafka/03_reference.md b/docs/integrations/data-ingestion/clickpipes/kafka/03_reference.md index b46c6217805..69a216928d9 100644 --- a/docs/integrations/data-ingestion/clickpipes/kafka/03_reference.md +++ b/docs/integrations/data-ingestion/clickpipes/kafka/03_reference.md @@ -61,7 +61,7 @@ The following standard ClickHouse data types are currently supported in ClickPip ### Avro {#avro} -#### Supported Avro Data Types {#supported-avro-data-types} +#### Supported Avro data types {#supported-avro-data-types} ClickPipes supports all Avro Primitive and Complex types, and all Avro Logical types except `time-millis`, `time-micros`, `local-timestamp-millis`, `local_timestamp-micros`, and `duration`. Avro `record` types are converted to Tuple, `array` types to Array, and `map` to Map (string keys only). In general the conversions listed [here](/interfaces/formats/Avro#data-type-mapping) are available. We recommend using exact type matching for Avro numeric types, as ClickPipes does not check for overflow or precision loss on type conversion. Alternatively, all Avro types can be inserted into a `String` column, and will be represented as a valid JSON string in that case. diff --git a/docs/integrations/data-ingestion/clickpipes/kafka/04_best_practices.md b/docs/integrations/data-ingestion/clickpipes/kafka/04_best_practices.md index 75e9b9227f0..d6ad8de3d21 100644 --- a/docs/integrations/data-ingestion/clickpipes/kafka/04_best_practices.md +++ b/docs/integrations/data-ingestion/clickpipes/kafka/04_best_practices.md @@ -10,7 +10,7 @@ keywords: ['kafka best practices', 'clickpipes', 'compression', 'authentication' # Best practices {#best-practices} -## Message Compression {#compression} +## Message compression {#compression} We strongly recommend using compression for your Kafka topics. Compression can result in a significant saving in data transfer costs with virtually no performance hit. To learn more about message compression in Kafka, we recommend starting with this [guide](https://www.confluent.io/blog/apache-kafka-message-compression/). @@ -25,7 +25,7 @@ ClickPipes for Kafka provides `at-least-once` delivery semantics (as one of the ## Authentication {#authentication} For Apache Kafka protocol data sources, ClickPipes supports [SASL/PLAIN](https://docs.confluent.io/platform/current/kafka/authentication_sasl/authentication_sasl_plain.html) authentication with TLS encryption, as well as `SASL/SCRAM-SHA-256` and `SASL/SCRAM-SHA-512`. Depending on the streaming source (Redpanda, MSK, etc) will enable all or a subset of these auth mechanisms based on compatibility. If you auth needs differ please [give us feedback](https://clickhouse.com/company/contact?loc=clickpipes). -### IAM {#iam} +### Iam {#iam} :::info IAM Authentication for the MSK ClickPipe is a beta feature. @@ -100,7 +100,7 @@ Role-based access only works for ClickHouse Cloud instances deployed to AWS. } ``` -### Custom Certificates {#custom-certificates} +### Custom certificates {#custom-certificates} ClickPipes for Kafka supports the upload of custom certificates for Kafka brokers which use non-public server certificates. Upload of client certificates and keys is also supported for mutual TLS (mTLS) based authentication. diff --git a/docs/integrations/data-ingestion/clickpipes/kafka/05_faq.md b/docs/integrations/data-ingestion/clickpipes/kafka/05_faq.md index 9d475a0723e..0ab66b2e209 100644 --- a/docs/integrations/data-ingestion/clickpipes/kafka/05_faq.md +++ b/docs/integrations/data-ingestion/clickpipes/kafka/05_faq.md @@ -65,7 +65,7 @@ Horizontal scaling adds more replicas to increase throughput, while vertical sca This can be configured during ClickPipe creation, or at any other point under **Settings** -> **Advanced Settings** -> **Scaling**. -### Azure Event Hubs {#azure-eventhubs} +### Azure event hubs {#azure-eventhubs}
diff --git a/docs/integrations/data-ingestion/clickpipes/kinesis.md b/docs/integrations/data-ingestion/clickpipes/kinesis.md index 8c1775fdcb3..7ef0d663632 100644 --- a/docs/integrations/data-ingestion/clickpipes/kinesis.md +++ b/docs/integrations/data-ingestion/clickpipes/kinesis.md @@ -25,7 +25,7 @@ import cp_destination from '@site/static/images/integrations/data-ingestion/clic import cp_overview from '@site/static/images/integrations/data-ingestion/clickpipes/cp_overview.png'; import Image from '@theme/IdealImage'; -# Integrating Amazon Kinesis with ClickHouse Cloud +# Integrating Amazon kinesis with ClickHouse Cloud ## Prerequisite {#prerequisite} You have familiarized yourself with the [ClickPipes intro](./index.md) and setup [IAM credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) or an [IAM Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html). Follow the [Kinesis Role-Based Access guide](./secure-kinesis.md) for information on how to setup a role that works with ClickHouse Cloud. diff --git a/docs/integrations/data-ingestion/clickpipes/mongodb/faq.md b/docs/integrations/data-ingestion/clickpipes/mongodb/faq.md index ebff7685865..5f145a9f582 100644 --- a/docs/integrations/data-ingestion/clickpipes/mongodb/faq.md +++ b/docs/integrations/data-ingestion/clickpipes/mongodb/faq.md @@ -10,7 +10,7 @@ keywords: ['clickpipes', 'mongodb', 'cdc', 'data ingestion', 'real-time sync'] # ClickPipes for MongoDB FAQ -### Can I query for individual fields in the JSON datatype? {#can-i-query-for-individual-fields-in-the-json-datatype} +### Can i query for individual fields in the JSON datatype? {#can-i-query-for-individual-fields-in-the-json-datatype} For direct field access, such as `{"user_id": 123}`, you can use **dot notation**: ```sql @@ -26,7 +26,7 @@ SELECT sum(doc.shipping.cost::Float32) AS total_shipping_cost FROM t1; ``` To learn more about working with JSON, see our [Working with JSON guide](./quickstart). -### How do I flatten the nested MongoDB documents in ClickHouse? {#how-do-i-flatten-the-nested-mongodb-documents-in-clickhouse} +### How do i flatten the nested MongoDB documents in ClickHouse? {#how-do-i-flatten-the-nested-mongodb-documents-in-clickhouse} MongoDB documents are replicated as JSON type in ClickHouse by default, preserving the nested structure. You have several options to flatten this data. If you want to flatten the data to columns, you can use normal views, materialized views, or query-time access. @@ -36,21 +36,21 @@ MongoDB documents are replicated as JSON type in ClickHouse by default, preservi For detailed examples, see our [Working with JSON guide](./quickstart). -### Can I connect MongoDB databases that don't have a public IP or are in private networks? {#can-i-connect-mongodb-databases-that-dont-have-a-public-ip-or-are-in-private-networks} +### Can i connect MongoDB databases that don't have a public ip or are in private networks? {#can-i-connect-mongodb-databases-that-dont-have-a-public-ip-or-are-in-private-networks} We support AWS PrivateLink for connecting to MongoDB databases that don't have a public IP or are in private networks. Azure Private Link and GCP Private Service Connect are currently not supported. -### What happens if I delete a database/table from my MongoDB database? {#what-happens-if-i-delete-a-database-table-from-my-mongodb-database} +### What happens if i delete a database/table from my MongoDB database? {#what-happens-if-i-delete-a-database-table-from-my-mongodb-database} When you delete a database/table from MongoDB, ClickPipes will continue running but the dropped database/table will stop replicating changes. The corresponding tables in ClickHouse is preserved. -### How does MongoDB CDC Connector handle transactions? {#how-does-mongodb-cdc-connector-handle-transactions} +### How does MongoDB CDC connector handle transactions? {#how-does-mongodb-cdc-connector-handle-transactions} Each document change within a transaction is processed individually to ClickHouse. Changes are applied in the order they appear in the oplog; and only committed changes are replicated to ClickHouse. If a MongoDB transaction is rolled back, those changes won't appear in the change stream. For more examples, see our [Working with JSON guide](./quickstart). -### How do I handle `resume of change stream was not possible, as the resume point may no longer be in the oplog.` error? {#resume-point-may-no-longer-be-in-the-oplog-error} +### How do i handle `resume of change stream was not possible, as the resume point may no longer be in the oplog.` error? {#resume-point-may-no-longer-be-in-the-oplog-error} This error typically occurs when the oplog is truncated and ClickPipe is unable to resume the change stream at the expected point. To resolve this issue, [resync the ClickPipe](./resync.md). To avoid this issue from recurring, we recommend increasing the oplog retention period. See instructions for [MongoDB Atlas](./source/atlas#enable-oplog-retention), [self-managed MongoDB](./source/generic#enable-oplog-retention), or [Amazon DocumentDB](./source/documentdb#configure-change-stream-log-retention). @@ -58,19 +58,19 @@ This error typically occurs when the oplog is truncated and ClickPipe is unable We use MongoDB's native Change Streams API to track changes in the database. Change Streams API provides a resumable stream of database changes by leveraging MongoDB's oplog (operations log). ClickPipe uses MongoDB's resume tokens to track the position in the oplog and ensure every change is replicated to ClickHouse. -### Which read preference should I use? {#which-read-preference-should-i-use} +### Which read preference should i use? {#which-read-preference-should-i-use} Which read preference to use depends on your specific use case. If you want to minimize the load on your primary node, we recommend using `secondaryPreferred` read preference. If you want to optimize ingestion latency, we recommend using `primaryPreferred` read preference. For more details, see [MongoDB documentation](https://www.mongodb.com/docs/manual/core/read-preference/#read-preference-modes-1). -### Does the MongoDB ClickPipe support Sharded Cluster? {#does-the-mongodb-clickpipe-support-sharded-cluster} +### Does the MongoDB ClickPipe support sharded cluster? {#does-the-mongodb-clickpipe-support-sharded-cluster} Yes, the MongoDB ClickPipe supports both Replica Set and Sharded Cluster. -### Does MongoDB ClickPipe support Amazon DocumentDB? {#documentdb-support} +### Does MongoDB ClickPipe support Amazon documentdb? {#documentdb-support} Yes, MongoDB ClickPipe supports Amazon DocumentDB 5.0. See [Amazon DocumentDB source setup guide](./source/documentdb.md) for details. -### Does MongoDB ClickPipe support PrivateLink? {#privatelink-support} +### Does MongoDB ClickPipe support privatelink? {#privatelink-support} We support PrivateLink for MongoDB (and DocumentDB) cluster in AWS only. diff --git a/docs/integrations/data-ingestion/clickpipes/mongodb/index.md b/docs/integrations/data-ingestion/clickpipes/mongodb/index.md index 4584474cf1e..d5bfb44d987 100644 --- a/docs/integrations/data-ingestion/clickpipes/mongodb/index.md +++ b/docs/integrations/data-ingestion/clickpipes/mongodb/index.md @@ -70,7 +70,7 @@ Make sure you are logged in to your ClickHouse Cloud account. If you don't have Fill in connection details -#### (Optional) Set up SSH Tunneling {#optional-set-up-ssh-tunneling} +#### (Optional) set up ssh tunneling {#optional-set-up-ssh-tunneling} You can specify SSH tunneling details if your source MongoDB database is not publicly accessible. diff --git a/docs/integrations/data-ingestion/clickpipes/mongodb/resync.md b/docs/integrations/data-ingestion/clickpipes/mongodb/resync.md index 0abb64cf13a..4cd84443049 100644 --- a/docs/integrations/data-ingestion/clickpipes/mongodb/resync.md +++ b/docs/integrations/data-ingestion/clickpipes/mongodb/resync.md @@ -10,7 +10,7 @@ keywords: ['clickpipes', 'mongodb', 'cdc', 'data ingestion', 'real-time sync'] import resync_button from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/resync_button.png' import Image from '@theme/IdealImage'; -### What does Resync do? {#what-mongodb-resync-do} +### What does resync do? {#what-mongodb-resync-do} Resync involves the following operations in order: @@ -28,7 +28,7 @@ Here are a few scenarios: 1. You may need to perform major schema changes on the source tables which would break the existing ClickPipe and you would need to restart. You can just click Resync after performing the changes. 2. Specifically for Clickhouse, maybe you needed to change the ORDER BY keys on the target tables. You can Resync to re-populate data into the new table with the right sorting key. -### Resync ClickPipe Guide {#guide-mongodb-resync} +### Resync ClickPipe guide {#guide-mongodb-resync} 1. In the Data Sources tab, click on the MongoDB ClickPipe you wish to resync. 2. Head over to the **Settings** tab. diff --git a/docs/integrations/data-ingestion/clickpipes/mongodb/source/atlas.md b/docs/integrations/data-ingestion/clickpipes/mongodb/source/atlas.md index b2e491c894f..5638f2d0270 100644 --- a/docs/integrations/data-ingestion/clickpipes/mongodb/source/atlas.md +++ b/docs/integrations/data-ingestion/clickpipes/mongodb/source/atlas.md @@ -15,7 +15,7 @@ import mongo_atlas_add_roles from '@site/static/images/integrations/data-ingesti import mongo_atlas_restrict_access from '@site/static/images/integrations/data-ingestion/clickpipes/mongodb/mongo-atlas-restrict-access.png' import Image from '@theme/IdealImage'; -# MongoDB Atlas source setup guide +# MongoDB atlas source setup guide ## Configure oplog retention {#enable-oplog-retention} diff --git a/docs/integrations/data-ingestion/clickpipes/mongodb/source/documentdb.md b/docs/integrations/data-ingestion/clickpipes/mongodb/source/documentdb.md index 1a74a7fd730..bfc3a088aef 100644 --- a/docs/integrations/data-ingestion/clickpipes/mongodb/source/documentdb.md +++ b/docs/integrations/data-ingestion/clickpipes/mongodb/source/documentdb.md @@ -13,9 +13,9 @@ import docdb_apply_parameter_group from '@site/static/images/integrations/data-i import docdb_parameter_group_status from '@site/static/images/integrations/data-ingestion/clickpipes/mongodb/docdb-parameter-group-status.png' import Image from '@theme/IdealImage'; -# Amazon DocumentDB source setup guide +# Amazon documentdb source setup guide -## Supported DocumentDB versions {#supported-documentdb-versions} +## Supported documentdb versions {#supported-documentdb-versions} ClickPipes supports DocumentDB version 5.0. @@ -23,7 +23,7 @@ ClickPipes supports DocumentDB version 5.0. By default, Amazon DocumentDB has a 3-hour change stream log retention period, while initial load may take much longer depending on existing data volume in your DocumentDB. We recommend setting the change stream log retention to 72 hours or longer to ensure that it is not truncated before the initial snapshot is completed. -### Update change stream log retention via AWS Console {#update-change-stream-log-retention-via-aws-console} +### Update change stream log retention via AWS console {#update-change-stream-log-retention-via-aws-console} 1. Click `Parameter groups` in the left panel, find the parameter group used by your DocumentDB cluster (if you are using the default parameter group, you will need to create a new parameter group first in order to modify it). Select parameter group diff --git a/docs/integrations/data-ingestion/clickpipes/mongodb/table_resync.md b/docs/integrations/data-ingestion/clickpipes/mongodb/table_resync.md index 96766f592a7..1bad442ac3d 100644 --- a/docs/integrations/data-ingestion/clickpipes/mongodb/table_resync.md +++ b/docs/integrations/data-ingestion/clickpipes/mongodb/table_resync.md @@ -13,15 +13,15 @@ There are scenarios where it would be useful to have specific tables of a pipe b While resyncing individual tables with a button click is a work-in-progress, this guide will share steps on how you can achieve this today in the MongoDB ClickPipe. -### 1. Remove the table from the pipe {#removing-table} +### 1. remove the table from the pipe {#removing-table} This can be followed by following the [table removal guide](./removing_tables). -### 2. Truncate or drop the table on ClickHouse {#truncate-drop-table} +### 2. truncate or drop the table on ClickHouse {#truncate-drop-table} This step is to avoid data duplication when we add this table again in the next step. You can do this by heading over to the **SQL Console** tab in ClickHouse Cloud and running a query. Note that we have validation to block table addition if the table already exists in ClickHouse and is not empty. -### 3. Add the table to the ClickPipe again {#add-table-again} +### 3. add the table to the ClickPipe again {#add-table-again} This can be followed by following the [table addition guide](./add_table). diff --git a/docs/integrations/data-ingestion/clickpipes/mysql/controlling_sync.md b/docs/integrations/data-ingestion/clickpipes/mysql/controlling_sync.md index ed3de2f4c28..740751462e8 100644 --- a/docs/integrations/data-ingestion/clickpipes/mysql/controlling_sync.md +++ b/docs/integrations/data-ingestion/clickpipes/mysql/controlling_sync.md @@ -35,7 +35,7 @@ The pull batch size is the number of records that the ClickPipe will pull from t The default is **100,000** records. A safe maximum is 10 million. -### An exception: Long-running transactions on source {#transactions} +### An exception: long-running transactions on source {#transactions} When a transaction is run on the source database, the ClickPipe waits until it receives the COMMIT of the transaction before it moves forward. This with **overrides** both the sync interval and the pull batch size. diff --git a/docs/integrations/data-ingestion/clickpipes/mysql/faq.md b/docs/integrations/data-ingestion/clickpipes/mysql/faq.md index e3d110efc21..88c36a90dc4 100644 --- a/docs/integrations/data-ingestion/clickpipes/mysql/faq.md +++ b/docs/integrations/data-ingestion/clickpipes/mysql/faq.md @@ -10,10 +10,10 @@ keywords: ['MySQL ClickPipes FAQ', 'ClickPipes MySQL troubleshooting', 'MySQL Cl # ClickPipes for MySQL FAQ -### Does the MySQL ClickPipe support MariaDB? {#does-the-clickpipe-support-mariadb} +### Does the MySQL ClickPipe support mariadb? {#does-the-clickpipe-support-mariadb} Yes, the MySQL ClickPipe supports MariaDB 10.0 and above. The configuration for it is very similar to MySQL, using GTID replication by default. -### Does the MySQL ClickPipe support PlanetScale, Vitess, or TiDB? {#does-the-clickpipe-support-planetscale-vitess} +### Does the MySQL ClickPipe support planetscale, vitess, or tidb? {#does-the-clickpipe-support-planetscale-vitess} No, these do not support MySQL's binlog API. ### How is replication managed? {#how-is-replication-managed} @@ -23,7 +23,7 @@ It's also possible for an inactive database to rotate the log file without allow At the start of an initial load we record the binlog offset to start at. This offset must still be valid when the initial load finishes in order for CDC to progress. If you are ingesting a large amount of data be sure to configure an appropriate binlog retention period. While setting up tables you can speed up initial load by configuring *Use a custom partitioning key for initial load* for large tables under advanced settings so that we can load a single table in parallel. -### Why am I getting a TLS certificate validation error when connecting to MySQL? {#tls-certificate-validation-error} +### Why am i getting a TLS certificate validation error when connecting to MySQL? {#tls-certificate-validation-error} When connecting to MySQL, you may encounter certificate errors like `x509: certificate is not valid for any names` or `x509: certificate signed by unknown authority`. These occur because ClickPipes enables TLS encryption by default. @@ -45,6 +45,6 @@ Please refer to the [ClickPipes for MySQL: Schema Changes Propagation Support](. Due to how MySQL [handles cascading deletes](https://dev.mysql.com/doc/refman/8.0/en/innodb-and-mysql-replication.html), they are not written to the binlog. Therefore it's not possible for ClickPipes (or any CDC tool) to replicate them. This can lead to inconsistent data. It's advised to use triggers instead for supporting cascading deletes. -### Why can I not replicate my table which has a dot in it? {#replicate-table-dot} +### Why can i not replicate my table which has a dot in it? {#replicate-table-dot} PeerDB has a limitation currently where dots in source table identifiers - aka either schema name or table name - is not supported for replication as PeerDB cannot discern, in that case, what is the schema and what is the table as it splits on dot. Effort is being made to support input of schema and table separately to get around this limitation. \ No newline at end of file diff --git a/docs/integrations/data-ingestion/clickpipes/mysql/index.md b/docs/integrations/data-ingestion/clickpipes/mysql/index.md index c25a501ba8c..b426d03b77a 100644 --- a/docs/integrations/data-ingestion/clickpipes/mysql/index.md +++ b/docs/integrations/data-ingestion/clickpipes/mysql/index.md @@ -73,7 +73,7 @@ Make sure you are logged in to your ClickHouse Cloud account. If you don't have Fill in connection details -#### (Optional) Set up SSH Tunneling {#optional-set-up-ssh-tunneling} +#### (Optional) set up ssh tunneling {#optional-set-up-ssh-tunneling} You can specify SSH tunneling details if your source MySQL database is not publicly accessible. diff --git a/docs/integrations/data-ingestion/clickpipes/mysql/resync.md b/docs/integrations/data-ingestion/clickpipes/mysql/resync.md index 2b31f4442b8..70d07226766 100644 --- a/docs/integrations/data-ingestion/clickpipes/mysql/resync.md +++ b/docs/integrations/data-ingestion/clickpipes/mysql/resync.md @@ -10,7 +10,7 @@ keywords: ['clickpipes', 'mysql', 'cdc', 'data ingestion', 'real-time sync'] import resync_button from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/resync_button.png' import Image from '@theme/IdealImage'; -### What does Resync do? {#what-mysql-resync-do} +### What does resync do? {#what-mysql-resync-do} Resync involves the following operations in order: @@ -32,7 +32,7 @@ Here are a few scenarios: You can resync multiple times, however please account for the load on the source database when you resync. ::: -### Resync ClickPipe Guide {#guide-mysql-resync} +### Resync ClickPipe guide {#guide-mysql-resync} 1. In the Data Sources tab, click on the MySQL ClickPipe you wish to resync. 2. Head over to the **Settings** tab. diff --git a/docs/integrations/data-ingestion/clickpipes/mysql/source/aurora.md b/docs/integrations/data-ingestion/clickpipes/mysql/source/aurora.md index d6917525b61..1eeb91e976c 100644 --- a/docs/integrations/data-ingestion/clickpipes/mysql/source/aurora.md +++ b/docs/integrations/data-ingestion/clickpipes/mysql/source/aurora.md @@ -27,7 +27,7 @@ This step-by-step guide shows you how to configure Amazon Aurora MySQL to replic The binary log is a set of log files that contain information about data modifications made to a MySQL server instance, and binary log files are required for replication. To configure binary log retention in Aurora MySQL, you must [enable binary logging](#enable-binlog-logging) and [increase the binlog retention interval](#binlog-retention-interval). -### 1. Enable binary logging via automated backup {#enable-binlog-logging} +### 1. enable binary logging via automated backup {#enable-binlog-logging} The automated backups feature determines whether binary logging is turned on or off for MySQL. Automated backups can be configured for your instance in the RDS Console by navigating to **Modify** > **Additional configuration** > **Backup** and selecting the **Enable automated backups** checkbox (if not selected already). @@ -35,7 +35,7 @@ The automated backups feature determines whether binary logging is turned on or We recommend setting the **Backup retention period** to a reasonably long value, depending on the replication use case. -### 2. Increase the binlog retention interval {#binlog-retention-interval} +### 2. increase the binlog retention interval {#binlog-retention-interval} :::warning If ClickPipes tries to resume replication and the required binlog files have been purged due to the configured binlog retention value, the ClickPipe will enter an errored state and a resync is required. @@ -84,7 +84,7 @@ The following parameters need to be set as follows:
Then, click on **Save Changes** in the top right corner. You may need to reboot your instance for the changes to take effect — a way of knowing this is if you see `Pending reboot` next to the parameter group link in the **Configuration** tab of the Aurora instance. -## Enable GTID mode (recommended) {#gtid-mode} +## Enable gtid mode (recommended) {#gtid-mode} :::tip The MySQL ClickPipe also supports replication without GTID mode. However, enabling GTID mode is recommended for better performance and easier troubleshooting. @@ -130,7 +130,7 @@ Connect to your Aurora MySQL instance as an admin user and execute the following ## Configure network access {#configure-network-access} -### IP-based access control {#ip-based-access-control} +### Ip-based access control {#ip-based-access-control} To restrict traffic to your Aurora MySQL instance, add the [documented static NAT IPs](../../index.md#list-of-static-ips) to the **Inbound rules** of your Aurora security group. @@ -138,7 +138,7 @@ To restrict traffic to your Aurora MySQL instance, add the [documented static NA Edit inbound rules for the above security group -### Private access via AWS PrivateLink {#private-access-via-aws-privatelink} +### Private access via AWS privatelink {#private-access-via-aws-privatelink} To connect to your Aurora MySQL instance through a private network, you can use AWS PrivateLink. Follow the [AWS PrivateLink setup guide for ClickPipes](/knowledgebase/aws-privatelink-setup-for-clickpipes) to set up the connection. diff --git a/docs/integrations/data-ingestion/clickpipes/mysql/source/gcp.md b/docs/integrations/data-ingestion/clickpipes/mysql/source/gcp.md index 811e3e916c9..0c15a215735 100644 --- a/docs/integrations/data-ingestion/clickpipes/mysql/source/gcp.md +++ b/docs/integrations/data-ingestion/clickpipes/mysql/source/gcp.md @@ -22,7 +22,7 @@ This is a step-by-step guide on how to configure your Cloud SQL for MySQL instan ## Enable binary log retention {#enable-binlog-retention-gcp} The binary log is a set of log files that contain information about data modifications made to an MySQL server instance, and binary log files are required for replication. -### Enable binary logging via PITR{#enable-binlog-logging-gcp} +### Enable binary logging via pitr{#enable-binlog-logging-gcp} The PITR feature determines whether binary logging is turned on or off for MySQL in Google Cloud. It can be set in the Cloud console, by editing your Cloud SQL instance and scrolling down to the below section. Enabling PITR in Cloud SQL @@ -71,7 +71,7 @@ This can be done either by editing the instance or by heading over to the `Conne IP allowlisting in GCP MySQL -## Download and use root CA certificate {#download-root-ca-certificate-gcp-mysql} +## Download and use root ca certificate {#download-root-ca-certificate-gcp-mysql} To connect to your Cloud SQL instance, you need to download the root CA certificate. 1. Go to your Cloud SQL instance in the Cloud console. diff --git a/docs/integrations/data-ingestion/clickpipes/mysql/source/generic_maria.md b/docs/integrations/data-ingestion/clickpipes/mysql/source/generic_maria.md index f83e123bf02..dd92764ea06 100644 --- a/docs/integrations/data-ingestion/clickpipes/mysql/source/generic_maria.md +++ b/docs/integrations/data-ingestion/clickpipes/mysql/source/generic_maria.md @@ -7,7 +7,7 @@ doc_type: 'guide' keywords: ['generic mariadb', 'clickpipes', 'binary logging', 'ssl tls', 'self hosted'] --- -# Generic MariaDB source setup guide +# Generic mariadb source setup guide :::info diff --git a/docs/integrations/data-ingestion/clickpipes/mysql/source/rds.md b/docs/integrations/data-ingestion/clickpipes/mysql/source/rds.md index 5a238013c3f..51015a3060b 100644 --- a/docs/integrations/data-ingestion/clickpipes/mysql/source/rds.md +++ b/docs/integrations/data-ingestion/clickpipes/mysql/source/rds.md @@ -19,7 +19,7 @@ import edit_button from '@site/static/images/integrations/data-ingestion/clickpi import enable_gtid from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/enable_gtid.png'; import Image from '@theme/IdealImage'; -# RDS MySQL source setup guide +# Rds MySQL source setup guide This step-by-step guide shows you how to configure Amazon RDS MySQL to replicate data into ClickHouse Cloud using the [MySQL ClickPipe](../index.md). For common questions around MySQL CDC, see the [MySQL FAQs page](/integrations/data-ingestion/clickpipes/mysql/faq.md). @@ -27,7 +27,7 @@ This step-by-step guide shows you how to configure Amazon RDS MySQL to replicate The binary log is a set of log files that contain information about data modifications made to an MySQL server instance, and binary log files are required for replication. To configure binary log retention in RDS MySQL, you must [enable binary logging](#enable-binlog-logging) and [increase the binlog retention interval](#binlog-retention-interval). -### 1. Enable binary logging via automated backup {#enable-binlog-logging} +### 1. enable binary logging via automated backup {#enable-binlog-logging} The automated backups feature determines whether binary logging is turned on or off for MySQL. Automated backups can be configured for your instance in the RDS Console by navigating to **Modify** > **Additional configuration** > **Backup** and selecting the **Enable automated backups** checkbox (if not selected already). @@ -35,7 +35,7 @@ The automated backups feature determines whether binary logging is turned on or We recommend setting the **Backup retention period** to a reasonably long value, depending on the replication use case. -### 2. Increase the binlog retention interval {#binlog-retention-interval} +### 2. increase the binlog retention interval {#binlog-retention-interval} :::warning If ClickPipes tries to resume replication and the required binlog files have been purged due to the configured binlog retention value, the ClickPipe will enter an errored state and a resync is required. @@ -83,7 +83,7 @@ The following parameters need to be set as follows:
Then, click on **Save Changes** in the top right corner. You may need to reboot your instance for the changes to take effect — a way of knowing this is if you see `Pending reboot` next to the parameter group link in the **Configuration** tab of the RDS instance. -## Enable GTID Mode {#gtid-mode} +## Enable gtid mode {#gtid-mode} :::tip The MySQL ClickPipe also supports replication without GTID mode. However, enabling GTID mode is recommended for better performance and easier troubleshooting. @@ -134,7 +134,7 @@ Connect to your RDS MySQL instance as an admin user and execute the following co ## Configure network access {#configure-network-access} -### IP-based access control {#ip-based-access-control} +### Ip-based access control {#ip-based-access-control} To restrict traffic to your Aurora MySQL instance, add the [documented static NAT IPs](../../index.md#list-of-static-ips) to the **Inbound rules** of your RDS security group. @@ -142,7 +142,7 @@ To restrict traffic to your Aurora MySQL instance, add the [documented static NA Edit inbound rules for the above security group -### Private access via AWS PrivateLink {#private-access-via-aws-privatelink} +### Private access via AWS privatelink {#private-access-via-aws-privatelink} To connect to your RDS instance through a private network, you can use AWS PrivateLink. Follow the [AWS PrivateLink setup guide for ClickPipes](/knowledgebase/aws-privatelink-setup-for-clickpipes) to set up the connection. diff --git a/docs/integrations/data-ingestion/clickpipes/mysql/source/rds_maria.md b/docs/integrations/data-ingestion/clickpipes/mysql/source/rds_maria.md index f585e145711..e9a24d36fad 100644 --- a/docs/integrations/data-ingestion/clickpipes/mysql/source/rds_maria.md +++ b/docs/integrations/data-ingestion/clickpipes/mysql/source/rds_maria.md @@ -17,7 +17,7 @@ import security_group_in_rds_mysql from '@site/static/images/integrations/data-i import edit_inbound_rules from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/rds/edit_inbound_rules.png'; import Image from '@theme/IdealImage'; -# RDS MariaDB source setup guide +# Rds mariadb source setup guide This is a step-by-step guide on how to configure your RDS MariaDB instance for replicating its data via the MySQL ClickPipe.
@@ -28,7 +28,7 @@ We also recommend going through the MySQL FAQs [here](/integrations/data-ingesti ## Enable binary log retention {#enable-binlog-retention-rds} The binary log is a set of log files that contain information about data modifications made to a MySQL server instance. Binary log files are required for replication. Both of the steps below must be followed: -### 1. Enable binary logging via automated backup{#enable-binlog-logging-rds} +### 1. enable binary logging via automated backup{#enable-binlog-logging-rds} The automated backups feature determines whether binary logging is turned on or off for MySQL. It can be set in the AWS console: @@ -36,7 +36,7 @@ The automated backups feature determines whether binary logging is turned on or Setting backup retention to a reasonably long value depending on the replication use-case is advisable. -### 2. Binlog retention hours{#binlog-retention-hours-rds} +### 2. binlog retention hours{#binlog-retention-hours-rds} Amazon RDS for MariaDB has a different method of setting binlog retention duration, which is the amount of time a binlog file containing changes is kept. If some changes are not read before the binlog file is removed, replication will be unable to continue. The default value of binlog retention hours is NULL, which means binary logs aren't retained. To specify the number of hours to retain binary logs on a DB instance, use the mysql.rds_set_configuration function with a binlog retention period long enough for replication to occur. `24 hours` is the recommended minimum. @@ -76,7 +76,7 @@ Next, click on `Save Changes` in the top-right. You may need to reboot your inst If you have a MariaDB cluster, the above parameters would be found in a [DB Cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithParamGroups.CreatingCluster.html) parameter group and not the DB instance group. ::: -## Enabling GTID Mode {#gtid-mode-rds} +## Enabling gtid mode {#gtid-mode-rds} Global Transaction Identifiers (GTIDs) are unique IDs assigned to each committed transaction in MySQL/MariaDB. They simplify binlog replication and make troubleshooting more straightforward. MariaDB enables GTID mode by default, so no user action is needed to use it. ## Configure a database user {#configure-database-user-rds} diff --git a/docs/integrations/data-ingestion/clickpipes/mysql/table_resync.md b/docs/integrations/data-ingestion/clickpipes/mysql/table_resync.md index d3708cbb315..a8827da071a 100644 --- a/docs/integrations/data-ingestion/clickpipes/mysql/table_resync.md +++ b/docs/integrations/data-ingestion/clickpipes/mysql/table_resync.md @@ -13,15 +13,15 @@ There are scenarios where it would be useful to have specific tables of a pipe b While resyncing individual tables with a button click is a work-in-progress, this guide will share steps on how you can achieve this today in the MySQL ClickPipe. -### 1. Remove the table from the pipe {#removing-table} +### 1. remove the table from the pipe {#removing-table} This can be followed by following the [table removal guide](./removing_tables). -### 2. Truncate or drop the table on ClickHouse {#truncate-drop-table} +### 2. truncate or drop the table on ClickHouse {#truncate-drop-table} This step is to avoid data duplication when we add this table again in the next step. You can do this by heading over to the **SQL Console** tab in ClickHouse Cloud and running a query. Note that we have validation to block table addition if the table already exists in ClickHouse and is not empty. -### 3. Add the table to the ClickPipe again {#add-table-again} +### 3. add the table to the ClickPipe again {#add-table-again} This can be followed by following the [table addition guide](./add_table). diff --git a/docs/integrations/data-ingestion/clickpipes/object-storage/02_reference.md b/docs/integrations/data-ingestion/clickpipes/object-storage/02_reference.md index f3116cacf74..5f1541cb188 100644 --- a/docs/integrations/data-ingestion/clickpipes/object-storage/02_reference.md +++ b/docs/integrations/data-ingestion/clickpipes/object-storage/02_reference.md @@ -72,7 +72,7 @@ This is not a valid path. `{N..M}` are not supported in ClickPipes. https://datasets-documentation.s3.eu-west-3.amazonaws.com/http/{documents-01,documents-02}.ndjson.gz ::: -## Continuous Ingest {#continuous-ingest} +## Continuous ingest {#continuous-ingest} ClickPipes supports continuous ingestion from S3, GCS, Azure Blob Storage, and DigitalOcean Spaces. When enabled, ClickPipes continuously ingests data from the specified path, and polls for new files at a rate of once every 30 seconds. However, new files must be lexically greater than the last ingested file. This means that they must be named in a way that defines the ingestion order. For instance, files named `file1`, `file2`, `file3`, etc., will be ingested sequentially. If a new file is added with a name like `file0`, ClickPipes will not ingest it because it is not lexically greater than the last ingested file. ## Tracking ingested files {#tracking-ingested-files} @@ -89,14 +89,14 @@ Public buckets need to allow both the `s3:GetObject` and the `s3:ListBucket` act Protected buckets can be accessed using either [IAM credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) or an [IAM Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html). To use an IAM Role, you will need to create the IAM Role as specified [in this guide](/cloud/data-sources/secure-s3). Copy the new IAM Role Arn after creation and paste it into the ClickPipe configuration as the "IAM ARN role". -### GCS {#gcs} +### Gcs {#gcs} Like S3, you can access public buckets with no configuration, and with protected buckets you can use [HMAC Keys](https://cloud.google.com/storage/docs/authentication/managing-hmackeys) in place of the AWS IAM credentials. You can read this guide from Google Cloud on [how to setup such keys](https://cloud.google.com/storage/docs/authentication/hmackeys). Service Accounts for GCS aren't directly supported. HMAC (IAM) Credentials must be used when authenticating with non-public buckets. The Service Account permissions attached to the HMAC credentials should be `storage.objects.list` and `storage.objects.get`. -### DigitalOcean Spaces {#dospaces} +### DigitalOcean spaces {#dospaces} Currently only protected buckets are supported for DigitalOcean spaces. You require an "Access Key" and a "Secret Key" to access the bucket and its files. You can read [this guide](https://docs.digitalocean.com/products/spaces/how-to/manage-access/) on how to create access keys. -### Azure Blob Storage {#azureblobstorage} +### Azure blob storage {#azureblobstorage} Currently only protected buckets are supported for Azure Blob Storage. Authentication is done via a connection string, which supports access keys and shared keys. For more information, read [this guide](https://learn.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string). diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/controlling_sync.md b/docs/integrations/data-ingestion/clickpipes/postgres/controlling_sync.md index 005ffc3398c..f243c52999b 100644 --- a/docs/integrations/data-ingestion/clickpipes/postgres/controlling_sync.md +++ b/docs/integrations/data-ingestion/clickpipes/postgres/controlling_sync.md @@ -35,7 +35,7 @@ The pull batch size is the number of records that the ClickPipe will pull from t The default is **100,000** records. A safe maximum is 10 million. -### An exception: Long-running transactions on source {#transactions} +### An exception: long-running transactions on source {#transactions} When a transaction is run on the source database, the ClickPipe waits until it receives the COMMIT of the transaction before it moves forward. This with **overrides** both the sync interval and the pull batch size. diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/deduplication.md b/docs/integrations/data-ingestion/clickpipes/postgres/deduplication.md index 0ae51114812..0f276638ca3 100644 --- a/docs/integrations/data-ingestion/clickpipes/postgres/deduplication.md +++ b/docs/integrations/data-ingestion/clickpipes/postgres/deduplication.md @@ -66,7 +66,7 @@ As a result, running a simple query like `SELECT count(*) FROM users;` may produ How can we ensure identical query results in both ClickHouse and PostgreSQL? -### Deduplicate using FINAL Keyword {#deduplicate-using-final-keyword} +### Deduplicate using final keyword {#deduplicate-using-final-keyword} The recommended way to deduplicate data in ClickHouse queries is to use the [FINAL modifier.](/sql-reference/statements/select/from#final-modifier) This ensures only the deduplicated rows are returned. @@ -122,7 +122,7 @@ ORDER BY viewcount DESC LIMIT 10 ``` -#### FINAL setting {#final-setting} +#### Final setting {#final-setting} Rather than adding the FINAL modifier to each table name in the query, you can use the [FINAL setting](/operations/settings/settings#final) to apply it automatically to all tables in the query. @@ -137,7 +137,7 @@ SET final = 1; SELECT count(*) FROM posts; ``` -#### ROW policy {#row-policy} +#### Row policy {#row-policy} An easy way to hide the redundant `_peerdb_is_deleted = 0` filter is to use [ROW policy.](/docs/operations/access-rights#row-policy-management) Below is an example that creates a row policy to exclude the deleted rows from all queries on the table votes. diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/faq.md b/docs/integrations/data-ingestion/clickpipes/postgres/faq.md index c422af4bfaf..955452316a3 100644 --- a/docs/integrations/data-ingestion/clickpipes/postgres/faq.md +++ b/docs/integrations/data-ingestion/clickpipes/postgres/faq.md @@ -19,7 +19,7 @@ If your ClickHouse Cloud service is idling, your Postgres CDC ClickPipe will con As an example, if your sync interval is set to 30 mins and your service idle time is set to 10 mins, Your service will wake-up every 30 mins and be active for 10 mins, then go back to idling. -### How are TOAST columns handled in ClickPipes for Postgres? {#how-are-toast-columns-handled-in-clickpipes-for-postgres} +### How are toast columns handled in ClickPipes for Postgres? {#how-are-toast-columns-handled-in-clickpipes-for-postgres} Please refer to the [Handling TOAST Columns](./toast) page for more information. @@ -47,7 +47,7 @@ It's important to note that if neither a primary key nor a replica identity is d Yes, partitioned tables are supported out of the box, as long as they have a PRIMARY KEY or REPLICA IDENTITY defined. The PRIMARY KEY and REPLICA IDENTITY must be present on both the parent table and its partitions. You can read more about it [here](https://blog.peerdb.io/real-time-change-data-capture-for-postgres-partitioned-tables). -### Can I connect Postgres databases that don't have a public IP or are in private networks? {#can-i-connect-postgres-databases-that-dont-have-a-public-ip-or-are-in-private-networks} +### Can i connect Postgres databases that don't have a public ip or are in private networks? {#can-i-connect-postgres-databases-that-dont-have-a-public-ip-or-are-in-private-networks} Yes! ClickPipes for Postgres offers two ways to connect to databases in private networks: @@ -64,7 +64,7 @@ Yes! ClickPipes for Postgres offers two ways to connect to databases in private - For detailed setup instructions, see our [PrivateLink documentation](/knowledgebase/aws-privatelink-setup-for-clickpipes) - For regions where PrivateLink is not available, please use SSH tunneling -### How do you handle UPDATEs and DELETEs? {#how-do-you-handle-updates-and-deletes} +### How do you handle updates and deletes? {#how-do-you-handle-updates-and-deletes} ClickPipes for Postgres captures both INSERTs and UPDATEs from Postgres as new rows with different versions (using the `_peerdb_` version column) in ClickHouse. The ReplacingMergeTree table engine periodically performs deduplication in the background based on the ordering key (ORDER BY columns), retaining only the row with the latest `_peerdb_` version. @@ -77,7 +77,7 @@ For more details, refer to: * [ReplacingMergeTree table engine best practices](https://docs.peerdb.io/bestpractices/clickhouse_datamodeling#replacingmergetree-table-engine) * [Postgres-to-ClickHouse CDC internals blog](https://clickhouse.com/blog/postgres-to-clickhouse-data-modeling-tips) -### Can I update primary key columns in PostgreSQL? {#can-i-update-primary-key-columns-in-postgresql} +### Can i update primary key columns in PostgreSQL? {#can-i-update-primary-key-columns-in-postgresql} :::warning Primary key updates in PostgreSQL cannot be properly replayed in ClickHouse by default. @@ -143,11 +143,11 @@ For an excellent deep dive into this topic, check out our blog post: [Overcoming ClickPipes for Postgres aims to map Postgres data types as natively as possible on the ClickHouse side. This document provides a comprehensive list of each data type and its mapping: [Data Type Matrix](https://docs.peerdb.io/datatypes/datatype-matrix). -### Can I define my own data type mapping while replicating data from Postgres to ClickHouse? {#can-i-define-my-own-data-type-mapping-while-replicating-data-from-postgres-to-clickhouse} +### Can i define my own data type mapping while replicating data from Postgres to ClickHouse? {#can-i-define-my-own-data-type-mapping-while-replicating-data-from-postgres-to-clickhouse} Currently, we don't support defining custom data type mappings as part of the pipe. However, note that the default data type mapping used by ClickPipes is highly native. Most column types in Postgres are replicated as closely as possible to their native equivalents on ClickHouse. Integer array types in Postgres, for instance, are replicated as integer array types on ClickHouse. -### How are JSON and JSONB columns replicated from Postgres? {#how-are-json-and-jsonb-columns-replicated-from-postgres} +### How are JSON and jsonb columns replicated from Postgres? {#how-are-json-and-jsonb-columns-replicated-from-postgres} JSON and JSONB columns are replicated as String type in ClickHouse. Since ClickHouse supports a native [JSON type](/sql-reference/data-types/newjson), you can create a materialized view over the ClickPipes tables to perform the translation if needed. Alternatively, you can use [JSON functions](/sql-reference/functions/json-functions) directly on the String column(s). We are actively working on a feature that replicates JSON and JSONB columns directly to the JSON type in ClickHouse. This feature is expected to be available in a few months. @@ -166,7 +166,7 @@ In summary, while sync and normalize processes are terminated during a pause, it A Postgres ClickPipe can also be created and managed via [OpenAPI](https://clickhouse.com/docs/cloud/manage/openapi) endpoints. This feature is in beta, and the API reference can be found [here](https://clickhouse.com/docs/cloud/manage/api/swagger#tag/beta). We are actively working on Terraform support to create Postgres ClickPipes as well. -### How do I speed up my initial load? {#how-do-i-speed-up-my-initial-load} +### How do i speed up my initial load? {#how-do-i-speed-up-my-initial-load} You cannot speed up an already running initial load. However, you can optimize future initial loads by adjusting certain settings. By default, the settings are configured with 4 parallel threads and a snapshot number of rows per partition set to 100,000. These are advanced settings and are generally sufficient for most use cases. @@ -178,7 +178,7 @@ For Postgres versions 13 or lower, CTID range scans are slower, and these settin These adjustments should significantly enhance the performance of the initial load, especially for older Postgres versions. If you are using Postgres 14 or later, these settings are less impactful due to improved support for CTID range scans. -### How should I scope my publications when setting up replication? {#how-should-i-scope-my-publications-when-setting-up-replication} +### How should i scope my publications when setting up replication? {#how-should-i-scope-my-publications-when-setting-up-replication} You can let ClickPipes manage your publications (requires additional permissions) or create them yourself. With ClickPipes-managed publications, we automatically handle table additions and removals as you edit the pipe. If self-managing, carefully scope your publications to only include tables you need to replicate - including unnecessary tables will slow down Postgres WAL decoding. @@ -257,7 +257,7 @@ If your database generates 100 GB of WAL per day, set: max_slot_wal_keep_size = 200GB ``` -### I'm seeing a ReceiveMessage EOF error in the logs. What does it mean? {#im-seeing-a-receivemessage-eof-error-in-the-logs-what-does-it-mean} +### I'm seeing a receivemessage eof error in the logs. what does it mean? {#im-seeing-a-receivemessage-eof-error-in-the-logs-what-does-it-mean} `ReceiveMessage` is a function in the Postgres logical decoding protocol that reads messages from the replication stream. An EOF (End of File) error indicates that the connection to the Postgres server was unexpectedly closed while trying to read from the replication stream. @@ -268,7 +268,7 @@ It can happen for a few reasons: - **Network Issues:** Temporary network disruptions can cause the connection to drop. - **Postgres Server Restart:** If the Postgres server is restarted or crashes, the connection will be lost. -### My replication slot is invalidated. What should I do? {#my-replication-slot-is-invalidated-what-should-i-do} +### My replication slot is invalidated. what should i do? {#my-replication-slot-is-invalidated-what-should-i-do} The only way to recover ClickPipe is by triggering a resync, which you can do in the Settings page. @@ -276,7 +276,7 @@ The most common cause of replication slot invalidation is a low `max_slot_wal_ke In rare cases, we have seen this issue occur even when `max_slot_wal_keep_size` is not configured. This could be due to an intricate and rare bug in PostgreSQL, although the cause remains unclear. -### I am seeing out of memory (OOMs) on ClickHouse while my ClickPipe is ingesting data. Can you help? {#i-am-seeing-out-of-memory-ooms-on-clickhouse-while-my-clickpipe-is-ingesting-data-can-you-help} +### I am seeing out of memory (ooms) on ClickHouse while my ClickPipe is ingesting data. can you help? {#i-am-seeing-out-of-memory-ooms-on-clickhouse-while-my-clickpipe-is-ingesting-data-can-you-help} One common reason for OOMs on ClickHouse is that your service is undersized. This means that your current service configuration doesn't have enough resources (e.g., memory or CPU) to handle the ingestion load effectively. We strongly recommend scaling up the service to meet the demands of your ClickPipe data ingestion. @@ -286,7 +286,7 @@ Another reason we've observed is the presence of downstream Materialized Views w - Another optimization for JOINs is to explicitly filter the tables through `subqueries` or `CTEs` and then perform the `JOIN` across these subqueries. This provides the planner with hints on how to efficiently filter rows and perform the `JOIN`. -### I am seeing an `invalid snapshot identifier` during the initial load. What should I do? {#i-am-seeing-an-invalid-snapshot-identifier-during-the-initial-load-what-should-i-do} +### I am seeing an `invalid snapshot identifier` during the initial load. what should i do? {#i-am-seeing-an-invalid-snapshot-identifier-during-the-initial-load-what-should-i-do} The `invalid snapshot identifier` error occurs when there is a connection drop between ClickPipes and your Postgres database. This can happen due to gateway timeouts, database restarts, or other transient issues. @@ -294,7 +294,7 @@ It is recommended that you do not carry out any disruptive operations like upgra To resolve this issue, you can trigger a resync from the ClickPipes UI. This will restart the initial load process from the beginning. -### What happens if I drop a publication in Postgres? {#what-happens-if-i-drop-a-publication-in-postgres} +### What happens if i drop a publication in Postgres? {#what-happens-if-i-drop-a-publication-in-postgres} Dropping a publication in Postgres will break your ClickPipe connection since the publication is required for the ClickPipe to pull changes from the source. When this happens, you'll typically receive an error alert indicating that the publication no longer exists. @@ -315,7 +315,7 @@ FOR TABLE <...>, <...> WITH (publish_via_partition_root = true); ``` -### What if I am seeing `Unexpected Datatype` errors or `Cannot parse type XX ...` {#what-if-i-am-seeing-unexpected-datatype-errors} +### What if i am seeing `Unexpected Datatype` errors or `Cannot parse type XX ...` {#what-if-i-am-seeing-unexpected-datatype-errors} This error typically occurs when the source Postgres database has a datatype which cannot be mapped during ingestion. For more specific issue, refer to the possibilities below. @@ -324,7 +324,7 @@ For more specific issue, refer to the possibilities below. There was a bug introduced in Postgres patch versions 17.5/16.9/15.13/14.18/13.21 due to which certain workloads can cause an exponential increase in memory usage, leading to a memory allocation request >1GB which Postgres considers invalid. This bug [has been fixed](https://github.com/postgres/postgres/commit/d87d07b7ad3b782cb74566cd771ecdb2823adf6a) and will be in the next Postgres patch series (17.6...). Please check with your Postgres provider when this patch version will be available for upgrade. If an upgrade isn't immediately possible, a resync of the pipe will be needed as it hits the error. -### I need to maintain a complete historical record in ClickHouse, even when the data is deleted from the source Postgres database. Can I completely ignore DELETE and TRUNCATE operations from Postgres in ClickPipes? {#ignore-delete-truncate} +### I need to maintain a complete historical record in ClickHouse, even when the data is deleted from the source Postgres database. can i completely ignore delete and truncate operations from Postgres in ClickPipes? {#ignore-delete-truncate} Yes! Before creating your Postgres ClickPipe, create a publication without DELETE operations. For example: ```sql @@ -334,17 +334,17 @@ Then when [setting up](https://clickhouse.com/docs/integrations/clickpipes/postg Note that TRUNCATE operations are ignored by ClickPipes and will not be replicated to ClickHouse. -### Why can I not replicate my table which has a dot in it? {#replicate-table-dot} +### Why can i not replicate my table which has a dot in it? {#replicate-table-dot} PeerDB has a limitation currently where dots in source table identifiers - aka either schema name or table name - is not supported for replication as PeerDB cannot discern, in that case, what is the schema and what is the table as it splits on dot. Effort is being made to support input of schema and table separately to get around this limitation. -### Initial load completed but there is no/missing data on ClickHouse. What could be the issue? {#initial-load-issue} +### Initial load completed but there is no/missing data on ClickHouse. what could be the issue? {#initial-load-issue} If your initial load has completed without error but your destination ClickHouse table is missing data, it might be that you have RLS (Row Level Security) policies enabled on your source Postgres tables. Also worth checking: - If the user has sufficient permissions to read the source tables. - If there are any row policies on ClickHouse side which might be filtering out rows. -### Can I have the ClickPipe create a replication slot with failover enabled? {#failover-slot} +### Can i have the ClickPipe create a replication slot with failover enabled? {#failover-slot} Yes, for a Postgres ClickPipe with replication mode as CDC or Snapshot + CDC, you can have ClickPipes create a replication slot with failover enabled, by toggling the below switch in the `Advanced Settings` section while creating the ClickPipe. Note that your Postgres version must be 17 or above to use this feature. diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/index.md b/docs/integrations/data-ingestion/clickpipes/postgres/index.md index 7b636118c92..198ff414582 100644 --- a/docs/integrations/data-ingestion/clickpipes/postgres/index.md +++ b/docs/integrations/data-ingestion/clickpipes/postgres/index.md @@ -85,13 +85,13 @@ Make sure you are logged in to your ClickHouse Cloud account. If you don't have Fill in connection details -#### (Optional) Setting up AWS Private Link {#optional-setting-up-aws-private-link} +#### (Optional) setting up AWS private link {#optional-setting-up-aws-private-link} You can use AWS Private Link to connect to your source Postgres database if it is hosted on AWS. This is useful if you want to keep your data transfer private. You can follow the [setup guide to set up the connection](/integrations/clickpipes/aws-privatelink). -#### (Optional) Setting up SSH tunneling {#optional-setting-up-ssh-tunneling} +#### (Optional) setting up ssh tunneling {#optional-setting-up-ssh-tunneling} You can specify SSH tunneling details if your source Postgres database is not publicly accessible. diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/maintenance.md b/docs/integrations/data-ingestion/clickpipes/postgres/maintenance.md index 32387b90a7e..bfe1e1f28b5 100644 --- a/docs/integrations/data-ingestion/clickpipes/postgres/maintenance.md +++ b/docs/integrations/data-ingestion/clickpipes/postgres/maintenance.md @@ -7,7 +7,7 @@ doc_type: 'reference' keywords: ['clickpipes', 'postgresql', 'cdc', 'data ingestion', 'real-time sync'] --- -# Maintenance windows for ClickPipes for Postgres +# Maintenance Windows for ClickPipes for Postgres There is an upcoming maintenance window for Postgres ClickPipes scheduled on: - **Date:** 17 April 2025 diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/ordering_keys.md b/docs/integrations/data-ingestion/clickpipes/postgres/ordering_keys.md index 44d6bbac4a4..dbf5f7e9efd 100644 --- a/docs/integrations/data-ingestion/clickpipes/postgres/ordering_keys.md +++ b/docs/integrations/data-ingestion/clickpipes/postgres/ordering_keys.md @@ -38,7 +38,7 @@ When including additional columns in the ordering key for ClickHouse (besides th For example, in a multi-tenant SaaS application, using (`tenant_id`, `id`) as the ordering key is a good choice. These columns uniquely identify each row, and `tenant_id` remains constant for an `id` even if other columns change. Since deduplication by id aligns with deduplication by (tenant_id, id), it helps avoid data [deduplication issues](https://docs.peerdb.io/mirror/ordering-key-different) that could arise if tenant_id were to change. -### Set Replica Identity on Postgres tables to custom ordering key {#set-replica-identity-on-postgres-tables-to-custom-ordering-key} +### Set replica identity on Postgres tables to custom ordering key {#set-replica-identity-on-postgres-tables-to-custom-ordering-key} For Postgres CDC to function as expected, it is important to modify the `REPLICA IDENTITY` on tables to include the ordering key columns. This is essential for handling DELETEs accurately. diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/parallel_initial_load.md b/docs/integrations/data-ingestion/clickpipes/postgres/parallel_initial_load.md index e3c9e4606bb..f310b05e85d 100644 --- a/docs/integrations/data-ingestion/clickpipes/postgres/parallel_initial_load.md +++ b/docs/integrations/data-ingestion/clickpipes/postgres/parallel_initial_load.md @@ -17,7 +17,7 @@ This document explains parallelized snapshot/initial load in the Postgres ClickP Initial load is the first phase of a CDC ClickPipe, where the ClickPipe syncs the historical data of the tables in the source database over to ClickHouse, before then starting CDC. A lot of the times, developers do this in a single-threaded manner - such as using pg_dump or pg_restore, or using a single thread to read from the source database and write to ClickHouse. However, the Postgres ClickPipe can parallelize this process, which can significantly speed up the initial load. -### CTID column in Postgres {#ctid-pg-snapshot} +### Ctid column in Postgres {#ctid-pg-snapshot} In Postgres, every row in a table has a unique identifier called the CTID. This is a system column that is not visible to users by default, but it can be used to uniquely identify rows in a table. The CTID is a combination of the block number and the offset within the block, which allows for efficient access to rows. ### Logical partitioning {#logical-partitioning-pg-snapshot} diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/resync.md b/docs/integrations/data-ingestion/clickpipes/postgres/resync.md index f582ef5ab46..f9e0c4e05d0 100644 --- a/docs/integrations/data-ingestion/clickpipes/postgres/resync.md +++ b/docs/integrations/data-ingestion/clickpipes/postgres/resync.md @@ -10,7 +10,7 @@ keywords: ['clickpipes', 'postgresql', 'cdc', 'data ingestion', 'real-time sync' import resync_button from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/resync_button.png' import Image from '@theme/IdealImage'; -### What does Resync do? {#what-postgres-resync-do} +### What does resync do? {#what-postgres-resync-do} Resync involves the following operations in order: 1. The existing ClickPipe is dropped, and a new "resync" ClickPipe is kicked off. Thus, changes to source table structures will be picked up when you resync. @@ -33,7 +33,7 @@ You can resync multiple times, however please account for the load on the source since initial load with parallel threads is involved each time. ::: -### Resync ClickPipe Guide {#guide-postgres-resync} +### Resync ClickPipe guide {#guide-postgres-resync} 1. In the Data Sources tab, click on the Postgres ClickPipe you wish to resync. 2. Head over to the **Settings** tab. diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/source/aurora.md b/docs/integrations/data-ingestion/clickpipes/postgres/source/aurora.md index 25262d5843b..0d70132ae1f 100644 --- a/docs/integrations/data-ingestion/clickpipes/postgres/source/aurora.md +++ b/docs/integrations/data-ingestion/clickpipes/postgres/source/aurora.md @@ -96,7 +96,7 @@ Connect to your Aurora PostgreSQL writer instance as an admin user and execute t ## Configure network access {#configure-network-access} -### IP-based access control {#ip-based-access-control} +### Ip-based access control {#ip-based-access-control} If you want to restrict traffic to your Aurora cluster, please add the [documented static NAT IPs](../../index.md#list-of-static-ips) to the `Inbound rules` of your Aurora security group. @@ -104,7 +104,7 @@ If you want to restrict traffic to your Aurora cluster, please add the [document Edit inbound rules for the above security group -### Private access via AWS PrivateLink {#private-access-via-aws-privatelink} +### Private access via AWS privatelink {#private-access-via-aws-privatelink} To connect to your Aurora cluster through a private network, you can use AWS PrivateLink. Follow our [AWS PrivateLink setup guide for ClickPipes](/knowledgebase/aws-privatelink-setup-for-clickpipes) to set up the connection. diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/source/azure-flexible-server-postgres.md b/docs/integrations/data-ingestion/clickpipes/postgres/source/azure-flexible-server-postgres.md index 499a40e657f..5afa568938e 100644 --- a/docs/integrations/data-ingestion/clickpipes/postgres/source/azure-flexible-server-postgres.md +++ b/docs/integrations/data-ingestion/clickpipes/postgres/source/azure-flexible-server-postgres.md @@ -69,7 +69,7 @@ Connect to your Azure Flexible Server Postgres through the admin user and run th ALTER ROLE clickpipes_user SET wal_sender_timeout to 0; ``` -## Add ClickPipes IPs to Firewall {#add-clickpipes-ips-to-firewall} +## Add ClickPipes ips to firewall {#add-clickpipes-ips-to-firewall} Please follow the below steps to add [ClickPipes IPs](../../index.md#list-of-static-ips) to your network. diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/source/crunchy-postgres.md b/docs/integrations/data-ingestion/clickpipes/postgres/source/crunchy-postgres.md index d0997be959b..f3f7eee015f 100644 --- a/docs/integrations/data-ingestion/clickpipes/postgres/source/crunchy-postgres.md +++ b/docs/integrations/data-ingestion/clickpipes/postgres/source/crunchy-postgres.md @@ -11,7 +11,7 @@ import firewall_rules_crunchy_bridge from '@site/static/images/integrations/data import add_firewall_rules_crunchy_bridge from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/setup/crunchy-postgres/add_firewall_rules_crunchy_bridge.png' import Image from '@theme/IdealImage'; -# Crunchy Bridge Postgres source setup guide +# Crunchy bridge Postgres source setup guide ClickPipes supports Postgres version 12 and later. @@ -55,7 +55,7 @@ Connect to your Crunchy Bridge Postgres through the `postgres` user and run the CREATE PUBLICATION clickpipes_publication FOR ALL TABLES; ``` -## Safe list ClickPipes IPs {#safe-list-clickpipes-ips} +## Safe list ClickPipes ips {#safe-list-clickpipes-ips} Safelist [ClickPipes IPs](../../index.md#list-of-static-ips) by adding the Firewall Rules in Crunchy Bridge. diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/source/generic.md b/docs/integrations/data-ingestion/clickpipes/postgres/source/generic.md index 3c51e6f5b7d..1632e56865a 100644 --- a/docs/integrations/data-ingestion/clickpipes/postgres/source/generic.md +++ b/docs/integrations/data-ingestion/clickpipes/postgres/source/generic.md @@ -76,7 +76,7 @@ Make sure to replace `clickpipes_user` and `clickpipes_password` with your desir ::: -## Enabling connections in pg_hba.conf to the ClickPipes User {#enabling-connections-in-pg_hbaconf-to-the-clickpipes-user} +## Enabling connections in pg_hba.conf to the ClickPipes user {#enabling-connections-in-pg_hbaconf-to-the-clickpipes-user} If you are self serving, you need to allow connections to the ClickPipes user from the ClickPipes IP addresses by following the below steps. If you are using a managed service, you can do the same by following the provider's documentation. diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/source/google-cloudsql.md b/docs/integrations/data-ingestion/clickpipes/postgres/source/google-cloudsql.md index 7ec4f4a9ab4..33fcf582896 100644 --- a/docs/integrations/data-ingestion/clickpipes/postgres/source/google-cloudsql.md +++ b/docs/integrations/data-ingestion/clickpipes/postgres/source/google-cloudsql.md @@ -17,7 +17,7 @@ import firewall1 from '@site/static/images/integrations/data-ingestion/clickpipe import firewall2 from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/google-cloudsql/firewall2.png'; import Image from '@theme/IdealImage'; -# Google Cloud SQL Postgres source setup guide +# Google cloud SQL Postgres source setup guide :::info @@ -75,7 +75,7 @@ Connect to your Cloud SQL Postgres through the admin user and run the below comm [//]: # (TODO Add SSH Tunneling) -## Add ClickPipes IPs to Firewall {#add-clickpipes-ips-to-firewall} +## Add ClickPipes ips to firewall {#add-clickpipes-ips-to-firewall} Please follow the below steps to add ClickPipes IPs to your network. diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/source/neon-postgres.md b/docs/integrations/data-ingestion/clickpipes/postgres/source/neon-postgres.md index cd153125343..31a56f8c328 100644 --- a/docs/integrations/data-ingestion/clickpipes/postgres/source/neon-postgres.md +++ b/docs/integrations/data-ingestion/clickpipes/postgres/source/neon-postgres.md @@ -61,7 +61,7 @@ SHOW max_wal_senders; -- should be 10 SHOW max_replication_slots; -- should be 10 ``` -## IP whitelisting (for Neon enterprise plan) {#ip-whitelisting-for-neon-enterprise-plan} +## Ip whitelisting (for neon enterprise plan) {#ip-whitelisting-for-neon-enterprise-plan} If you have Neon Enterprise plan, you can whitelist the [ClickPipes IPs](../../index.md#list-of-static-ips) to allow replication from ClickPipes to your Neon Postgres instance. To do this you can click on the **Settings** tab and go to the **IP Allow** section. diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/source/planetscale.md b/docs/integrations/data-ingestion/clickpipes/postgres/source/planetscale.md index 3001ee4f6ed..233036d4c52 100644 --- a/docs/integrations/data-ingestion/clickpipes/postgres/source/planetscale.md +++ b/docs/integrations/data-ingestion/clickpipes/postgres/source/planetscale.md @@ -11,7 +11,7 @@ import planetscale_wal_level_logical from '@site/static/images/integrations/data import planetscale_max_slot_wal_keep_size from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/planetscale/planetscale_max_slot_wal_keep_size.png'; import Image from '@theme/IdealImage'; -# PlanetScale for Postgres source setup guide +# Planetscale for Postgres source setup guide :::info PlanetScale for Postgres is currently in [early access](https://planetscale.com/postgres). diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/source/rds.md b/docs/integrations/data-ingestion/clickpipes/postgres/source/rds.md index 63fc91449d2..4d76e84d09e 100644 --- a/docs/integrations/data-ingestion/clickpipes/postgres/source/rds.md +++ b/docs/integrations/data-ingestion/clickpipes/postgres/source/rds.md @@ -16,7 +16,7 @@ import security_group_in_rds_postgres from '@site/static/images/integrations/dat import edit_inbound_rules from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/rds/edit_inbound_rules.png'; import Image from '@theme/IdealImage'; -# RDS Postgres source setup guide +# Rds Postgres source setup guide ## Supported Postgres versions {#supported-postgres-versions} @@ -96,7 +96,7 @@ Connect to your RDS Postgres instance as an admin user and execute the following ## Configure network access {#configure-network-access} -### IP-based access control {#ip-based-access-control} +### Ip-based access control {#ip-based-access-control} If you want to restrict traffic to your RDS instance, please add the [documented static NAT IPs](../../index.md#list-of-static-ips) to the `Inbound rules` of your RDS security group. @@ -104,11 +104,11 @@ If you want to restrict traffic to your RDS instance, please add the [documented Edit inbound rules for the above security group -### Private Access via AWS PrivateLink {#private-access-via-aws-privatelink} +### Private access via AWS privatelink {#private-access-via-aws-privatelink} To connect to your RDS instance through a private network, you can use AWS PrivateLink. Follow our [AWS PrivateLink setup guide for ClickPipes](/knowledgebase/aws-privatelink-setup-for-clickpipes) to set up the connection. -### Workarounds for RDS Proxy {#workarounds-for-rds-proxy} +### Workarounds for rds proxy {#workarounds-for-rds-proxy} RDS Proxy does not support logical replication connections. If you have dynamic IP addresses in RDS and cannot use DNS name or a lambda, here are some alternatives: 1. Using a cron job, resolve the RDS endpoint's IP periodically and update the NLB if it has changed. diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/source/supabase.md b/docs/integrations/data-ingestion/clickpipes/postgres/source/supabase.md index 0ddc83163c5..b75bf29dc58 100644 --- a/docs/integrations/data-ingestion/clickpipes/postgres/source/supabase.md +++ b/docs/integrations/data-ingestion/clickpipes/postgres/source/supabase.md @@ -65,7 +65,7 @@ For better recommendation of this value you can contact the ClickPipes team. ::: -## Connection details to use for Supabase {#connection-details-to-use-for-supabase} +## Connection details to use for supabase {#connection-details-to-use-for-supabase} Head over to your Supabase Project's `Project Settings` -> `Database` (under `Configuration`). @@ -79,7 +79,7 @@ The connection pooler is not supported for CDC based replication, hence it needs ::: -## Note on RLS {#note-on-rls} +## Note on rls {#note-on-rls} The ClickPipes Postgres user must not be restricted by RLS policies, as it can lead to missing data. You can disable RLS policies for the user by running the below command: ```sql ALTER USER clickpipes_user BYPASSRLS; diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/source/timescale.md b/docs/integrations/data-ingestion/clickpipes/postgres/source/timescale.md index bd8309176b6..0e7010e61e6 100644 --- a/docs/integrations/data-ingestion/clickpipes/postgres/source/timescale.md +++ b/docs/integrations/data-ingestion/clickpipes/postgres/source/timescale.md @@ -9,7 +9,7 @@ doc_type: 'guide' import BetaBadge from '@theme/badges/BetaBadge'; -# Postgres with TimescaleDB source setup guide +# Postgres with timescaledb source setup guide diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/table_resync.md b/docs/integrations/data-ingestion/clickpipes/postgres/table_resync.md index 4bab6ec754e..147d6d13e4e 100644 --- a/docs/integrations/data-ingestion/clickpipes/postgres/table_resync.md +++ b/docs/integrations/data-ingestion/clickpipes/postgres/table_resync.md @@ -13,15 +13,15 @@ There are scenarios where it would be useful to have specific tables of a pipe b While resyncing individual tables with a button click is a work-in-progress, this guide will share steps on how you can achieve this today in the Postgres ClickPipe. -### 1. Remove the table from the pipe {#removing-table} +### 1. remove the table from the pipe {#removing-table} This can be followed by following the [table removal guide](./removing_tables). -### 2. Truncate or drop the table on ClickHouse {#truncate-drop-table} +### 2. truncate or drop the table on ClickHouse {#truncate-drop-table} This step is to avoid data duplication when we add this table again in the next step. You can do this by heading over to the **SQL Console** tab in ClickHouse Cloud and running a query. Note that we have validation to block table addition if the table already exists in ClickHouse and is not empty. -### 3. Add the table to the ClickPipe again {#add-table-again} +### 3. add the table to the ClickPipe again {#add-table-again} This can be followed by following the [table addition guide](./add_table). diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/toast.md b/docs/integrations/data-ingestion/clickpipes/postgres/toast.md index fcf07081300..82eb123277b 100644 --- a/docs/integrations/data-ingestion/clickpipes/postgres/toast.md +++ b/docs/integrations/data-ingestion/clickpipes/postgres/toast.md @@ -8,7 +8,7 @@ keywords: ['clickpipes', 'postgresql', 'cdc', 'data ingestion', 'real-time sync' When replicating data from PostgreSQL to ClickHouse, it's important to understand the limitations and special considerations for TOAST (The Oversized-Attribute Storage Technique) columns. This guide will help you identify and properly handle TOAST columns in your replication process. -## What are TOAST columns in PostgreSQL? {#what-are-toast-columns-in-postgresql} +## What are toast columns in PostgreSQL? {#what-are-toast-columns-in-postgresql} TOAST (The Oversized-Attribute Storage Technique) is PostgreSQL's mechanism for handling large field values. When a row exceeds the maximum row size (typically 2KB, but this can vary depending on the PostgreSQL version and exact settings), PostgreSQL automatically moves large field values into a separate TOAST table, storing only a pointer in the main table. @@ -18,7 +18,7 @@ During the initial load (snapshot), all column values, including TOAST columns, You can read more about TOAST and its implementation in PostgreSQL here: https://www.postgresql.org/docs/current/storage-toast.html -## Identifying TOAST columns in a table {#identifying-toast-columns-in-a-table} +## Identifying toast columns in a table {#identifying-toast-columns-in-a-table} To identify if a table has TOAST columns, you can use the following SQL query: @@ -34,7 +34,7 @@ WHERE c.relname = 'your_table_name' This query will return the names and data types of columns that could potentially be TOASTed. However, it's important to note that this query only identifies columns that are eligible for TOAST storage based on their data type and storage attributes. To determine if these columns actually contain TOASTed data, you'll need to consider whether the values in these columns exceed the size. The actual TOASTing of data depends on the specific content stored in these columns. -## Ensuring proper handling of TOAST columns {#ensuring-proper-handling-of-toast-columns} +## Ensuring proper handling of toast columns {#ensuring-proper-handling-of-toast-columns} To ensure that TOAST columns are handled correctly during replication, you should set the `REPLICA IDENTITY` of the table to `FULL`. This tells PostgreSQL to include the full old row in the WAL for UPDATE and DELETE operations, ensuring that all column values (including TOAST columns) are available for replication. @@ -46,7 +46,7 @@ ALTER TABLE your_table_name REPLICA IDENTITY FULL; Refer to [this blog post](https://xata.io/blog/replica-identity-full-performance) for performance considerations when setting `REPLICA IDENTITY FULL`. -## Replication behavior when REPLICA IDENTITY FULL is not set {#replication-behavior-when-replica-identity-full-is-not-set} +## Replication behavior when replica identity full is not set {#replication-behavior-when-replica-identity-full-is-not-set} If `REPLICA IDENTITY FULL` is not set for a table with TOAST columns, you may encounter the following issues when replicating to ClickHouse: diff --git a/docs/integrations/data-ingestion/clickpipes/secure-kinesis.md b/docs/integrations/data-ingestion/clickpipes/secure-kinesis.md index d0fd6ae273f..b3db2bfdd96 100644 --- a/docs/integrations/data-ingestion/clickpipes/secure-kinesis.md +++ b/docs/integrations/data-ingestion/clickpipes/secure-kinesis.md @@ -31,9 +31,9 @@ Using this approach, customers can manage all access to their Kinesis data strea -### Obtaining the ClickHouse service IAM role Arn {#obtaining-the-clickhouse-service-iam-role-arn} +### Obtaining the ClickHouse service iam role arn {#obtaining-the-clickhouse-service-iam-role-arn} -- 1. Login to your ClickHouse cloud account. +- 1. Login to your ClickHouse Cloud account. - 2. Select the ClickHouse service you want to create the integration - 3. Select the **Settings** tab - 4. Scroll down to the **Network security information** section at the bottom of the page @@ -41,9 +41,9 @@ Using this approach, customers can manage all access to their Kinesis data strea Secure S3 ARN -### Setting up IAM assume role {#setting-up-iam-assume-role} +### Setting up iam assume role {#setting-up-iam-assume-role} -#### Manually create IAM role. {#manually-create-iam-role} +#### Manually create iam role. {#manually-create-iam-role} - 1. Login to your AWS Account in the web browser with an IAM user that has permission to create & manage IAM role. - 2. Browse to IAM Service Console diff --git a/docs/integrations/data-ingestion/clickpipes/secure-rds.md b/docs/integrations/data-ingestion/clickpipes/secure-rds.md index f1cdb8db949..4536b371b17 100644 --- a/docs/integrations/data-ingestion/clickpipes/secure-rds.md +++ b/docs/integrations/data-ingestion/clickpipes/secure-rds.md @@ -20,9 +20,9 @@ For MySQL and MariaDB, this limitation does not apply, and you can run both `Ini ## Setup {#setup} -### Obtaining the ClickHouse service IAM role Arn {#obtaining-the-clickhouse-service-iam-role-arn} +### Obtaining the ClickHouse service iam role arn {#obtaining-the-clickhouse-service-iam-role-arn} -1 - Login to your ClickHouse cloud account. +1 - Login to your ClickHouse Cloud account. 2 - Select the ClickHouse service you want to create the integration @@ -38,7 +38,7 @@ Let's call this value `{ClickHouse_IAM_ARN}`. This is the IAM role that will be ### Configuring the RDS/Aurora instance {#configuring-the-rds-aurora-instance} -#### Enabling IAM DB Authentication {#enabling-iam-db-authentication} +#### Enabling iam DB authentication {#enabling-iam-db-authentication} 1. Login to your AWS Account and navigate to the RDS instance you want to configure. 2. Click on the **Modify** button. 3. Scroll down to the **Database authentication** section. @@ -46,13 +46,13 @@ Let's call this value `{ClickHouse_IAM_ARN}`. This is the IAM role that will be 5. Click on the **Continue** button. 6. Review the changes and click on the **Apply immediately** option. -#### Obtaining the RDS/Aurora Resource ID {#obtaining-the-rds-resource-id} +#### Obtaining the RDS/Aurora resource id {#obtaining-the-rds-resource-id} 1. Login to your AWS Account and navigate to the RDS instance/Aurora Cluster you want to configure. 2. Click on the **Configuration** tab. 3. Note the **Resource ID** value. It should look like `db-xxxxxxxxxxxxxx` for RDS or `cluster-xxxxxxxxxxxxxx` for Aurora cluster. Let's call this value `{RDS_RESOURCE_ID}`. This is the resource ID that will be used in the IAM policy to allow access to the RDS instance. -#### Setting up the Database User {#setting-up-the-database-user} +#### Setting up the database user {#setting-up-the-database-user} ##### PostgreSQL {#setting-up-the-database-user-postgres} @@ -63,7 +63,7 @@ Let's call this value `{ClickHouse_IAM_ARN}`. This is the IAM role that will be ``` 2. Follow the rest of the steps in the [PostgreSQL source setup guide](postgres/source/rds) to configure your RDS instance for ClickPipes. -##### MySQL / MariaDB {#setting-up-the-database-user-mysql} +##### MySQL / mariadb {#setting-up-the-database-user-mysql} 1. Connect to your RDS/Aurora instance and create a new database user with the following command: ```sql @@ -71,9 +71,9 @@ Let's call this value `{ClickHouse_IAM_ARN}`. This is the IAM role that will be ``` 2. Follow the rest of the steps in the [MySQL source setup guide](mysql/source/rds) to configure your RDS/Aurora instance for ClickPipes. -### Setting up the IAM role {#setting-up-iam-role} +### Setting up the iam role {#setting-up-iam-role} -#### Manually create IAM role. {#manually-create-iam-role} +#### Manually create iam role. {#manually-create-iam-role} 1 - Login to your AWS Account in the web browser with an IAM user that has permission to create & manage IAM role. diff --git a/docs/integrations/data-ingestion/data-formats/binary.md b/docs/integrations/data-ingestion/data-formats/binary.md index 3160e09845a..c369f8dd47c 100644 --- a/docs/integrations/data-ingestion/data-formats/binary.md +++ b/docs/integrations/data-ingestion/data-formats/binary.md @@ -15,7 +15,7 @@ ClickHouse supports multiple binary formats, which result in better performance We're going to use some_data [table](assets/some_data.sql) and [data](assets/some_data.tsv) for demonstration, feel free to reproduce that on your ClickHouse instance. -## Exporting in a Native ClickHouse format {#exporting-in-a-native-clickhouse-format} +## Exporting in a native ClickHouse format {#exporting-in-a-native-clickhouse-format} The most efficient data format to export and import data between ClickHouse nodes is [Native](/interfaces/formats/Native) format. Exporting is done using `INTO OUTFILE` clause: @@ -26,7 +26,7 @@ INTO OUTFILE 'data.clickhouse' FORMAT Native This will create [data.clickhouse](assets/data.clickhouse) file in a native format. -### Importing from a Native format {#importing-from-a-native-format} +### Importing from a native format {#importing-from-a-native-format} To import data, we can use [file()](/sql-reference/table-functions/file.md) for smaller files or exploration purposes: @@ -73,7 +73,7 @@ COMPRESSION 'lz4' FORMAT Native ``` -## Exporting to RowBinary {#exporting-to-rowbinary} +## Exporting to rowbinary {#exporting-to-rowbinary} Another binary format supported is [RowBinary](/interfaces/formats/RowBinary), which allows importing and exporting data in binary-represented rows: @@ -84,7 +84,7 @@ INTO OUTFILE 'data.binary' FORMAT RowBinary This will generate [data.binary](assets/data.binary) file in a binary rows format. -### Exploring RowBinary files {#exploring-rowbinary-files} +### Exploring rowbinary files {#exploring-rowbinary-files} Automatic schema inference is not supported for this format, so to explore before loading, we have to define schema explicitly: ```sql @@ -104,7 +104,7 @@ LIMIT 5 Consider using [RowBinaryWithNames](/interfaces/formats/RowBinaryWithNames), which also adds a header row with a columns list. [RowBinaryWithNamesAndTypes](/interfaces/formats/RowBinaryWithNamesAndTypes) will also add an additional header row with column types. -### Importing from RowBinary files {#importing-from-rowbinary-files} +### Importing from rowbinary files {#importing-from-rowbinary-files} To load data from a RowBinary file, we can use a `FROM INFILE` clause: ```sql @@ -113,7 +113,7 @@ FROM INFILE 'data.binary' FORMAT RowBinary ``` -## Importing single binary value using RawBLOB {#importing-single-binary-value-using-rawblob} +## Importing single binary value using rawblob {#importing-single-binary-value-using-rawblob} Suppose we want to read an entire binary file and save it into a field in a table. This is the case when the [RawBLOB format](/interfaces/formats/RawBLOB) can be used. This format can be directly used with a single-column table only: @@ -139,7 +139,7 @@ SELECT length(data) FROM images └──────────────┘ ``` -### Exporting RawBLOB data {#exporting-rawblob-data} +### Exporting rawblob data {#exporting-rawblob-data} This format can also be used to export data using an `INTO OUTFILE` clause: @@ -151,7 +151,7 @@ FORMAT RawBLOB Note that we had to use `LIMIT 1` because exporting more than a single value will create a corrupted file. -## MessagePack {#messagepack} +## Messagepack {#messagepack} ClickHouse supports importing and exporting to [MessagePack](https://msgpack.org/) using the [MsgPack](/interfaces/formats/MsgPack). To export to MessagePack format: @@ -170,7 +170,7 @@ FROM INFILE 'data.msgpk' FORMAT MsgPack ``` -## Protocol Buffers {#protocol-buffers} +## Protocol buffers {#protocol-buffers} @@ -197,7 +197,7 @@ SETTINGS format_schema = 'schema:MessageType' This saves data to the [proto.bin](assets/proto.bin) file. ClickHouse also supports importing Protobuf data as well as nested messages. Consider using [ProtobufSingle](/interfaces/formats/ProtobufSingle) to work with a single Protocol Buffer message (length delimiters will be omitted in this case). -## Cap'n Proto {#capn-proto} +## Cap'n proto {#capn-proto} diff --git a/docs/integrations/data-ingestion/data-formats/csv-tsv.md b/docs/integrations/data-ingestion/data-formats/csv-tsv.md index 00bef9d1257..8c4c20b1c3a 100644 --- a/docs/integrations/data-ingestion/data-formats/csv-tsv.md +++ b/docs/integrations/data-ingestion/data-formats/csv-tsv.md @@ -107,7 +107,7 @@ The [file](assets/data_small.csv) has 1k rows, but ClickHouse loaded only 990 si When using the `file()` function, with ClickHouse Cloud you will need to run the commands in `clickhouse client` on the machine where the file resides. Another option is to use [`clickhouse-local`](/operations/utilities/clickhouse-local.md) to explore files locally. ::: -### Treating NULL values in CSV files {#treating-null-values-in-csv-files} +### Treating null values in CSV files {#treating-null-values-in-csv-files} Null values can be encoded differently depending on the application that generated the file. By default, ClickHouse uses `\N` as a Null value in CSV. But we can change that using the [format_csv_null_representation](/operations/settings/settings-formats.md/#format_tsv_null_representation) option. diff --git a/docs/integrations/data-ingestion/data-formats/json/formats.md b/docs/integrations/data-ingestion/data-formats/json/formats.md index 3ebca753362..dfcfe7619ba 100644 --- a/docs/integrations/data-ingestion/data-formats/json/formats.md +++ b/docs/integrations/data-ingestion/data-formats/json/formats.md @@ -427,7 +427,7 @@ Code: 117. DB::Exception: Unknown field found while parsing JSONEachRow format: ClickHouse will throw exceptions in cases of inconsistent JSON and table columns structure. -## BSON {#bson} +## Bson {#bson} ClickHouse allows exporting to and importing data from [BSON](https://bsonspec.org/) encoded files. This format is used by some DBMSs, e.g. [MongoDB](https://github.com/mongodb/mongo) database. diff --git a/docs/integrations/data-ingestion/data-formats/json/intro.md b/docs/integrations/data-ingestion/data-formats/json/intro.md index 8f33bbd2933..bc1f8f4521a 100644 --- a/docs/integrations/data-ingestion/data-formats/json/intro.md +++ b/docs/integrations/data-ingestion/data-formats/json/intro.md @@ -9,7 +9,7 @@ score: 10 doc_type: 'guide' --- -# JSON Overview +# JSON overview -## 1. Gather your connection details {#1-gather-your-connection-details} +## 1. gather your connection details {#1-gather-your-connection-details} -## 2. Making a read-only user {#2-making-a-read-only-user} +## 2. making a read-only user {#2-making-a-read-only-user} When connecting ClickHouse to a data visualization tool like Grafana, it is recommended to make a read-only user to protect your data from unwanted modifications. @@ -55,7 +55,7 @@ To configure a read-only user, follow these steps: 2. Ensure the `readonly` user has enough permission to modify the `max_execution_time` setting required by the underlying [clickhouse-go client](https://github.com/ClickHouse/clickhouse-go). 3. If you're using a public ClickHouse instance, it is not recommended to set `readonly=2` in the `readonly` profile. Instead, leave `readonly=1` and set the constraint type of `max_execution_time` to [changeable_in_readonly](/operations/settings/constraints-on-settings) to allow modification of this setting. -## 3. Install the ClickHouse plugin for Grafana {#3--install-the-clickhouse-plugin-for-grafana} +## 3. install the ClickHouse plugin for Grafana {#3--install-the-clickhouse-plugin-for-grafana} Before Grafana can connect to ClickHouse, you need to install the appropriate Grafana plugin. Assuming you are logged in to Grafana, follow these steps: @@ -69,7 +69,7 @@ Before Grafana can connect to ClickHouse, you need to install the appropriate Gr Install the ClickHouse plugin -## 4. Define a ClickHouse data source {#4-define-a-clickhouse-data-source} +## 4. define a ClickHouse data source {#4-define-a-clickhouse-data-source} 1. Once the installation is complete, click the **Add new data source** button. (You can also add a data source from the **Data sources** tab on the **Connections** page.) @@ -93,7 +93,7 @@ For more settings, check the [plugin configuration](./config.md) documentation. Select Save & test -## 5. Next steps {#5-next-steps} +## 5. next steps {#5-next-steps} Your data source is now ready to use! Learn more about how to build queries with the [query builder](./query-builder.md). diff --git a/docs/integrations/data-visualization/grafana/query-builder.md b/docs/integrations/data-visualization/grafana/query-builder.md index 665af81ddd3..5478d4e036e 100644 --- a/docs/integrations/data-visualization/grafana/query-builder.md +++ b/docs/integrations/data-visualization/grafana/query-builder.md @@ -20,7 +20,7 @@ import trace_id_in_logs from '@site/static/images/integrations/data-visualizatio import demo_data_links from '@site/static/images/integrations/data-visualization/grafana/demo_data_links.png'; import ClickHouseSupportedBadge from '@theme/badges/ClickHouseSupported'; -# Query Builder +# Query builder diff --git a/docs/integrations/data-visualization/lightdash-and-clickhouse.md b/docs/integrations/data-visualization/lightdash-and-clickhouse.md index a86d1922b56..28b6aa284b0 100644 --- a/docs/integrations/data-visualization/lightdash-and-clickhouse.md +++ b/docs/integrations/data-visualization/lightdash-and-clickhouse.md @@ -31,7 +31,7 @@ Developers love Lightdash for its open architecture, version-controlled YAML mod This partnership brings together **ClickHouse’s speed** and **Lightdash’s developer experience**, making it easier than ever to explore, visualize, and automate insights with AI. -## Build an interactive dashboard with Lightdash and ClickHouse {#build-an-interactive-dashboard} +## Build an interactive dashboard with lightdash and ClickHouse {#build-an-interactive-dashboard} In this guide, you’ll see how **Lightdash** connects to **ClickHouse** to explore your dbt models and build interactive dashboards. The example below shows a finished dashboard powered by data from ClickHouse. @@ -68,7 +68,7 @@ For example: Example profiles.yml configuration for a lightdash-clickhouse project
-### Create a Lightdash project connected to ClickHouse {#creating-a-lightdash-project-connected-to-clickhouse} +### Create a lightdash project connected to ClickHouse {#creating-a-lightdash-project-connected-to-clickhouse} Once your dbt profile is configured for ClickHouse, you’ll also need to connect your **dbt project** to Lightdash. @@ -80,7 +80,7 @@ After connecting your dbt project, Lightdash will automatically detect your Clic --- -### Explore your ClickHouse data in Lightdash {#exploring-your-clickhouse-data-in-lightdash} +### Explore your ClickHouse data in lightdash {#exploring-your-clickhouse-data-in-lightdash} Once connected, Lightdash automatically syncs your dbt models and exposes: @@ -92,7 +92,7 @@ You can now build dashboards, share insights, and even use **Ask AI** to generat --- -### Define metrics and dimensions in Lightdash {#defining-metrics-and-dimensions-in-lightdash} +### Define metrics and dimensions in lightdash {#defining-metrics-and-dimensions-in-lightdash} In Lightdash, all **metrics** and **dimensions** are defined directly in your dbt model `.yml` files. This makes your business logic version-controlled, consistent, and fully transparent. @@ -150,7 +150,7 @@ Learn more about how to create and organize dashboards here: --- -### Ask AI: self-serve analytics powered by dbt {#ask-ai} +### Ask ai: self-serve analytics powered by dbt {#ask-ai} **AI Agents** in Lightdash make data exploration truly self-serve. Instead of writing queries, users can simply ask questions in plain language — like *“What was our monthly revenue growth?”* — and the AI Agent automatically generates the right visualization, referencing your dbt-defined metrics and models to ensure accuracy and consistency. diff --git a/docs/integrations/data-visualization/looker-and-clickhouse.md b/docs/integrations/data-visualization/looker-and-clickhouse.md index 9fba107b3b8..3b1abefc9d2 100644 --- a/docs/integrations/data-visualization/looker-and-clickhouse.md +++ b/docs/integrations/data-visualization/looker-and-clickhouse.md @@ -24,10 +24,10 @@ import PartnerBadge from '@theme/badges/PartnerBadge'; Looker can connect to ClickHouse Cloud or on-premise deployment via the official ClickHouse data source. -## 1. Gather your connection details {#1-gather-your-connection-details} +## 1. gather your connection details {#1-gather-your-connection-details} -## 2. Create a ClickHouse data source {#2-create-a-clickhouse-data-source} +## 2. create a ClickHouse data source {#2-create-a-clickhouse-data-source} Navigate to Admin -> Database -> Connections and click the "Add Connection" button in the top right corner. @@ -51,7 +51,7 @@ Test your connection first, and, once it is done, connect to your new ClickHouse Now you should be able to attach ClickHouse Datasource to your Looker project. -## 3. Known limitations {#3-known-limitations} +## 3. known limitations {#3-known-limitations} 1. The following data types are handled as strings by default: * Array - serialization does not work as expected due to the JDBC driver limitations diff --git a/docs/integrations/data-visualization/looker-studio-and-clickhouse.md b/docs/integrations/data-visualization/looker-studio-and-clickhouse.md index a1ab313eb2d..4332412db68 100644 --- a/docs/integrations/data-visualization/looker-studio-and-clickhouse.md +++ b/docs/integrations/data-visualization/looker-studio-and-clickhouse.md @@ -23,7 +23,7 @@ import looker_studio_enable_mysql from '@site/static/images/integrations/data-vi import looker_studio_mysql_cloud from '@site/static/images/integrations/data-visualization/looker_studio_mysql_cloud.png'; import PartnerBadge from '@theme/badges/PartnerBadge'; -# Looker Studio +# Looker studio @@ -35,7 +35,7 @@ Looker Studio can connect to ClickHouse via the MySQL interface using the offici ## On-premise ClickHouse server setup {#on-premise-clickhouse-server-setup} -## Connecting Looker Studio to ClickHouse {#connecting-looker-studio-to-clickhouse} +## Connecting Looker studio to ClickHouse {#connecting-looker-studio-to-clickhouse} First, login to https://lookerstudio.google.com using your Google account and create a new Data Source: @@ -70,7 +70,7 @@ Finally, you should be able to see the introspected table structure and adjust t Now you can proceed with exploring your data or creating a new report! -## Using Looker Studio with ClickHouse Cloud {#using-looker-studio-with-clickhouse-cloud} +## Using Looker studio with ClickHouse Cloud {#using-looker-studio-with-clickhouse-cloud} When using ClickHouse Cloud, you need to enable MySQL interface first. You can do that in connection dialog, "MySQL" tab. diff --git a/docs/integrations/data-visualization/metabase-and-clickhouse.md b/docs/integrations/data-visualization/metabase-and-clickhouse.md index ee7c76a7cfe..93a110a4b55 100644 --- a/docs/integrations/data-visualization/metabase-and-clickhouse.md +++ b/docs/integrations/data-visualization/metabase-and-clickhouse.md @@ -41,10 +41,10 @@ In this guide you will ask some questions of your ClickHouse data with Metabase If you do not have a dataset to work with you can add one of the examples. This guide uses the [UK Price Paid](/getting-started/example-datasets/uk-price-paid.md) dataset, so you might choose that one. There are several others to look at in the same documentation category. ::: -## 1. Gather your connection details {#1-gather-your-connection-details} +## 1. gather your connection details {#1-gather-your-connection-details} -## 2. Download the ClickHouse plugin for Metabase {#2--download-the-clickhouse-plugin-for-metabase} +## 2. download the ClickHouse plugin for Metabase {#2--download-the-clickhouse-plugin-for-metabase} 1. If you do not have a `plugins` folder, create one as a subfolder of where you have `metabase.jar` saved. @@ -56,7 +56,7 @@ If you do not have a dataset to work with you can add one of the examples. This 5. Access Metabase at http://hostname:3000. On the initial startup, you will see a welcome screen and have to work your way through a list of questions. If prompted to select a database, select "**I'll add my data later**": -## 3. Connect Metabase to ClickHouse {#3--connect-metabase-to-clickhouse} +## 3. connect Metabase to ClickHouse {#3--connect-metabase-to-clickhouse} 1. Click on the gear icon in the top-right corner and select **Admin Settings** to visit your Metabase admin page. @@ -74,7 +74,7 @@ If you do not have a dataset to work with you can add one of the examples. This 6. Click the **Save** button and Metabase will scan your database for tables. -## 4. Run a SQL query {#4-run-a-sql-query} +## 4. run a SQL query {#4-run-a-sql-query} 1. Exit the **Admin settings** by clicking the **Exit admin** button in the top-right corner. @@ -86,7 +86,7 @@ If you do not have a dataset to work with you can add one of the examples. This Metabase SQL editor showing a query on UK price paid data -## 5. Ask a question {#5-ask-a-question} +## 5. ask a question {#5-ask-a-question} 1. Click on **+ New** and select **Question**. Notice you can build a question by starting with a database and table. For example, the following question is being asked of a table named `uk_price_paid` in the `default` database. Here is a simple question that calculates the average price by town, within the county of Greater Manchester: diff --git a/docs/integrations/data-visualization/omni-and-clickhouse.md b/docs/integrations/data-visualization/omni-and-clickhouse.md index f9b62831663..370ef4c81e6 100644 --- a/docs/integrations/data-visualization/omni-and-clickhouse.md +++ b/docs/integrations/data-visualization/omni-and-clickhouse.md @@ -19,11 +19,11 @@ import PartnerBadge from '@theme/badges/PartnerBadge'; Omni can connect to ClickHouse Cloud or on-premise deployment via the official ClickHouse data source. -## 1. Gather your connection details {#1-gather-your-connection-details} +## 1. gather your connection details {#1-gather-your-connection-details} -## 2. Create a ClickHouse data source {#2-create-a-clickhouse-data-source} +## 2. create a ClickHouse data source {#2-create-a-clickhouse-data-source} Navigate to Admin -> Connections and click the "Add Connection" button in the top right corner. diff --git a/docs/integrations/data-visualization/powerbi-and-clickhouse.md b/docs/integrations/data-visualization/powerbi-and-clickhouse.md index 69972eb593b..db320bb9a45 100644 --- a/docs/integrations/data-visualization/powerbi-and-clickhouse.md +++ b/docs/integrations/data-visualization/powerbi-and-clickhouse.md @@ -29,7 +29,7 @@ import powerbi_dsn_credentials from '@site/static/images/integrations/data-visua import powerbi_16 from '@site/static/images/integrations/data-visualization/powerbi_16.png'; import ClickHouseSupportedBadge from '@theme/badges/ClickHouseSupported'; -# Power BI +# Power bi @@ -51,7 +51,7 @@ This tutorial will guide you through the process of: ## Prerequisites {#prerequisites} -### Power BI Installation {#power-bi-installation} +### Power bi installation {#power-bi-installation} This tutorial assumes you have Microsoft Power BI Desktop installed on your Windows machine. You can download and install Power BI Desktop [here](https://www.microsoft.com/en-us/download/details.aspx?id=58494) @@ -66,7 +66,7 @@ You'll need the following details for connecting to your ClickHouse instance: * Password - Password of the user * Database - Name of the database on the instance you want to connect to -## Power BI desktop {#power-bi-desktop} +## Power bi desktop {#power-bi-desktop} To get started with querying data in Power BI Desktop, you'll need to complete the following steps: @@ -75,7 +75,7 @@ To get started with querying data in Power BI Desktop, you'll need to complete t 3. Connect to ClickHouse 4. Query and Visualize you data -### Install the ODBC Driver {#install-the-odbc-driver} +### Install the odbc driver {#install-the-odbc-driver} Download the most recent [ClickHouse ODBC release](https://github.com/ClickHouse/clickhouse-odbc/releases). @@ -88,7 +88,7 @@ Execute the supplied `.msi` installer and follow the wizard. `Debug symbols` are optional and not required ::: -#### Verify ODBC driver {#verify-odbc-driver} +#### Verify odbc driver {#verify-odbc-driver} When the driver installation is completed, you can verify the installation was successful by: @@ -102,7 +102,7 @@ Verify the ClickHouse Driver is listed. ODBC Data Source Administrator showing ClickHouse drivers in the Drivers tab
-### Find the ClickHouse Connector {#find-the-clickhouse-connector} +### Find the ClickHouse connector {#find-the-clickhouse-connector} :::note Available in version `2.137.751.0` of Power BI Desktop @@ -142,7 +142,7 @@ If you have a use case that has a small amount of data, you can choose import mo ClickHouse connection credentials dialog for username and password
-### Query and Visualise Data {#query-and-visualise-data} +### Query and visualise data {#query-and-visualise-data} Finally, you should see the databases and tables in the Navigator view. Select the desired table and click "Load" to import the data from ClickHouse. @@ -153,19 +153,19 @@ import the data from ClickHouse. Once the import is complete, your ClickHouse Data should be accessible in Power BI as usual.
-## Power BI service {#power-bi-service} +## Power bi service {#power-bi-service} In order to use Microsoft Power BI Service, you need to create an [on-premise data gateway](https://learn.microsoft.com/en-us/power-bi/connect-data/service-gateway-onprem). For more details on how to setup custom connectors, please refer to Microsoft's documentation on how to [use custom data connectors with an on-premises data gateway](https://learn.microsoft.com/en-us/power-bi/connect-data/service-gateway-custom-connectors). -## ODBC driver (import only) {#odbc-driver-import-only} +## Odbc driver (import only) {#odbc-driver-import-only} We recommend using the ClickHouse Connector that uses DirectQuery. Install the [ODBC Driver](#install-the-odbc-driver) onto the on-premise data gateway instance and [verify](#verify-odbc-driver) as outlined above. -### Create a new User DSN {#create-a-new-user-dsn} +### Create a new user dsn {#create-a-new-user-dsn} When the driver installation is complete, an ODBC data source can be created. Search for ODBC in the Start menu and select "ODBC Data Sources (64-bit)". @@ -194,7 +194,7 @@ If you are using a deployment that has SSL enabled (e.g. ClickHouse Cloud or a s - `Timeout` is an integer representing seconds. Default value: `30 seconds`. ::: -### Get data into Power BI {#get-data-into-power-bi} +### Get data into power bi {#get-data-into-power-bi} In case you don't have Power BI installed yet, [download and install Power BI Desktop](https://www.microsoft.com/en-us/download/details.aspx?id=58494). @@ -230,7 +230,7 @@ Once the import is complete, your ClickHouse Data should be accessible in Power ## Known limitations {#known-limitations} -### UInt64 {#uint64} +### Uint64 {#uint64} Unsigned integer types such as UInt64 or bigger won't be loaded into the dataset automatically, as Int64 is the maximum whole number type support by Power BI. diff --git a/docs/integrations/data-visualization/quicksight-and-clickhouse.md b/docs/integrations/data-visualization/quicksight-and-clickhouse.md index bfb94c190c0..44f99299110 100644 --- a/docs/integrations/data-visualization/quicksight-and-clickhouse.md +++ b/docs/integrations/data-visualization/quicksight-and-clickhouse.md @@ -21,7 +21,7 @@ import quicksight_06 from '@site/static/images/integrations/data-visualization/q import quicksight_07 from '@site/static/images/integrations/data-visualization/quicksight_07.png'; import ClickHouseSupportedBadge from '@theme/badges/ClickHouseSupported'; -# QuickSight +# Quicksight @@ -119,7 +119,7 @@ mysql> show databases; Read 4 rows, 603.00 B in 0.00156 sec., 2564 rows/sec., 377.48 KiB/sec. ``` -## Connecting QuickSight to ClickHouse {#connecting-quicksight-to-clickhouse} +## Connecting quicksight to ClickHouse {#connecting-quicksight-to-clickhouse} First of all, go to [https://quicksight.aws.amazon.com](https://quicksight.aws.amazon.com), navigate to Datasets and click "New dataset": diff --git a/docs/integrations/data-visualization/splunk-and-clickhouse.md b/docs/integrations/data-visualization/splunk-and-clickhouse.md index 11f31e5edef..348be279d46 100644 --- a/docs/integrations/data-visualization/splunk-and-clickhouse.md +++ b/docs/integrations/data-visualization/splunk-and-clickhouse.md @@ -56,7 +56,7 @@ Before you get started you will need: - Admin or SSH access to your Splunk Enterprise OS Instance - ClickHouse connection details (see [here](/integrations/metabase#1-gather-your-connection-details) if you're using ClickHouse Cloud) -## Install and configure DB Connect on Splunk Enterprise {#install-and-configure-db-connect-on-splunk-enterprise} +## Install and configure DB connect on Splunk enterprise {#install-and-configure-db-connect-on-splunk-enterprise} You must first install the Java Runtime Environment on your Splunk Enterprise instance. If you're using Docker, you can use the command `microdnf install java-11-openjdk`. @@ -74,7 +74,7 @@ Once you've verified that the DB Connect App is installed, add the java_home pat Splunk DB Connect settings page showing Java Home configuration -## Configure JDBC for ClickHouse {#configure-jdbc-for-clickhouse} +## Configure jdbc for ClickHouse {#configure-jdbc-for-clickhouse} Download the [ClickHouse JDBC driver](https://github.com/ClickHouse/clickhouse-java) to the DB Connect Drivers folder such as: diff --git a/docs/integrations/data-visualization/superset-and-clickhouse.md b/docs/integrations/data-visualization/superset-and-clickhouse.md index 9287ad54b81..b7f387b36c0 100644 --- a/docs/integrations/data-visualization/superset-and-clickhouse.md +++ b/docs/integrations/data-visualization/superset-and-clickhouse.md @@ -45,10 +45,10 @@ In this guide you will build a dashboard in Superset with data from a ClickHouse If you do not have a dataset to work with you can add one of the examples. This guide uses the [UK Price Paid](/getting-started/example-datasets/uk-price-paid.md) dataset, so you might choose that one. There are several others to look at in the same documentation category. ::: -## 1. Gather your connection details {#1-gather-your-connection-details} +## 1. gather your connection details {#1-gather-your-connection-details} -## 2. Install the Driver {#2-install-the-driver} +## 2. install the driver {#2-install-the-driver} 1. Superset uses the `clickhouse-connect` driver to connect to ClickHouse. The details of `clickhouse-connect` are at https://pypi.org/project/clickhouse-connect/ and it can be installed with the following command: @@ -58,7 +58,7 @@ If you do not have a dataset to work with you can add one of the examples. This 2. Start (or restart) Superset. -## 3. Connect Superset to ClickHouse {#3-connect-superset-to-clickhouse} +## 3. connect Superset to ClickHouse {#3-connect-superset-to-clickhouse} 1. Within Superset, select **Data** from the top menu and then **Databases** from the drop-down menu. Add a new database by clicking the **+ Database** button: @@ -80,7 +80,7 @@ If you do not have a dataset to work with you can add one of the examples. This 4. Click the **CONNECT** and then **FINISH** buttons to complete the setup wizard, and you should see your database in the list of databases. -## 4. Add a Dataset {#4-add-a-dataset} +## 4. add a dataset {#4-add-a-dataset} 1. To interact with your ClickHouse data with Superset, you need to define a **_dataset_**. From the top menu in Superset, select **Data**, then **Datasets** from the drop-down menu. @@ -91,7 +91,7 @@ If you do not have a dataset to work with you can add one of the examples. This 3. Click the **ADD** button at the bottom of the dialog window and your table appears in the list of datasets. You are ready to build a dashboard and analyze your ClickHouse data! -## 5. Creating charts and a dashboard in Superset {#5--creating-charts-and-a-dashboard-in-superset} +## 5. creating charts and a dashboard in Superset {#5--creating-charts-and-a-dashboard-in-superset} If you are familiar with Superset, then you will feel right at home with this next section. If you are new to Superset, well...it's like a lot of the other cool visualization tools out there in the world - it doesn't take long to get started, but the details and nuances get learned over time as you use the tool. diff --git a/docs/integrations/data-visualization/tableau/tableau-analysis-tips.md b/docs/integrations/data-visualization/tableau/tableau-analysis-tips.md index 398df7b0e07..2ac8d65c809 100644 --- a/docs/integrations/data-visualization/tableau/tableau-analysis-tips.md +++ b/docs/integrations/data-visualization/tableau/tableau-analysis-tips.md @@ -9,10 +9,10 @@ doc_type: 'guide' --- # Analysis tips -## MEDIAN() and PERCENTILE() functions {#median-and-percentile-functions} +## Median() and percentile() functions {#median-and-percentile-functions} - In Live mode the MEDIAN() and PERCENTILE() functions (since connector v0.1.3 release) use the [ClickHouse quantile()() function](/sql-reference/aggregate-functions/reference/quantile/), which significantly speeds up the calculation, but uses sampling. If you want to get accurate calculation results, then use functions `MEDIAN_EXACT()` and `PERCENTILE_EXACT()` (based on [quantileExact()()](/sql-reference/aggregate-functions/reference/quantileexact/)). - In Extract mode you can't use MEDIAN_EXACT() and PERCENTILE_EXACT() because MEDIAN() and PERCENTILE() are always accurate (and slow). -## Additional functions for calculated fields in Live mode {#additional-functions-for-calculated-fields-in-live-mode} +## Additional functions for calculated fields in live mode {#additional-functions-for-calculated-fields-in-live-mode} ClickHouse has a huge number of functions that can be used for data analysis — much more than Tableau supports. For the convenience of users, we have added new functions that are available for use in Live mode when creating Calculated Fields. Unfortunately, it is not possible to add descriptions to these functions in the Tableau interface, so we will add a description for them right here. - **[`-If` Aggregation Combinator](/sql-reference/aggregate-functions/combinators/#-if)** *(added in v0.2.3)* - allows to have Row-Level Filters right in the Aggregate Calculation. `SUM_IF(), AVG_IF(), COUNT_IF(), MIN_IF() & MAX_IF()` functions have been added. - **`BAR([my_int], [min_val_int], [max_val_int], [bar_string_length_int])`** *(added in v0.2.1)* — Forget about boring bar charts! Use `BAR()` function instead (equivalent of [`bar()`](/sql-reference/functions/other-functions#bar) in ClickHouse). For example, this calculated field returns nice bars as String: diff --git a/docs/integrations/data-visualization/tableau/tableau-and-clickhouse.md b/docs/integrations/data-visualization/tableau/tableau-and-clickhouse.md index b051b07afb5..44f97694b2b 100644 --- a/docs/integrations/data-visualization/tableau/tableau-and-clickhouse.md +++ b/docs/integrations/data-visualization/tableau/tableau-and-clickhouse.md @@ -100,7 +100,7 @@ To solve that, consider upgrading your Tableau Desktop application, or [install | Password | *\***** | :::note -When working with ClickHouse cloud, it's required to enable the SSL checkbox for secured connections. +When working with ClickHouse Cloud, it's required to enable the SSL checkbox for secured connections. :::
@@ -122,7 +122,7 @@ could change, but for now you must use **default** as the database.) You are now ready to build some visualizations in Tableau! -## Building Visualizations in Tableau {#building-visualizations-in-tableau} +## Building visualizations in Tableau {#building-visualizations-in-tableau} Now that we have a ClickHouse data source configured in Tableau, let's visualize the data... diff --git a/docs/integrations/data-visualization/tableau/tableau-connection-tips.md b/docs/integrations/data-visualization/tableau/tableau-connection-tips.md index fc7b454ee56..60ec0159180 100644 --- a/docs/integrations/data-visualization/tableau/tableau-connection-tips.md +++ b/docs/integrations/data-visualization/tableau/tableau-connection-tips.md @@ -36,7 +36,7 @@ In 99% of cases you don't need the Advanced tab, for the remaining 1% you can us - **JDBC Driver URL Parameters**. You can pass the remaining [driver parameters](https://github.com/ClickHouse/clickhouse-jdbc#configuration), for example `jdbcCompliance`, in this field. Be careful, the parameter values must be passed in the URL Encoded format, and in the case of passing `custom_http_params` or `typeMappings` in this field and in the previous fields of the Advanced tab, the values of the preceding two fields on the Advanced tab have a higher priority - **Set Session ID** checkbox. It is needed to set session-level settings in Initial SQL tab, generates a `session_id` with a timestamp and a pseudo-random number in the format `"tableau-jdbc-connector-*{timestamp}*-*{number}*"` -## Limited support for UInt64, Int128, (U)Int256 data types {#limited-support-for-uint64-int128-uint256-data-types} +## Limited support for uint64, int128, (U)Int256 data types {#limited-support-for-uint64-int128-uint256-data-types} By default, the driver displays fields of types *UInt64, Int128, (U)Int256* as strings, **but it displays, not converts**. This means that when you try to write the next calculated field, you will get an error ```text LEFT([myUInt256], 2) // Error! diff --git a/docs/integrations/data-visualization/tableau/tableau-online-and-clickhouse.md b/docs/integrations/data-visualization/tableau/tableau-online-and-clickhouse.md index 7d3dcdd5636..5ada4b83b4f 100644 --- a/docs/integrations/data-visualization/tableau/tableau-online-and-clickhouse.md +++ b/docs/integrations/data-visualization/tableau/tableau-online-and-clickhouse.md @@ -21,7 +21,7 @@ import tableau_desktop_03 from '@site/static/images/integrations/data-visualizat import tableau_desktop_04 from '@site/static/images/integrations/data-visualization/tableau_desktop_04.png'; import tableau_desktop_05 from '@site/static/images/integrations/data-visualization/tableau_desktop_05.png'; -# Tableau Online +# Tableau online Tableau Online can connect to ClickHouse Cloud or on-premise ClickHouse setup via MySQL interface using the official MySQL data source. @@ -31,7 +31,7 @@ Tableau Online can connect to ClickHouse Cloud or on-premise ClickHouse setup vi ## On-premise ClickHouse server setup {#on-premise-clickhouse-server-setup} -## Connecting Tableau Online to ClickHouse (on-premise without SSL) {#connecting-tableau-online-to-clickhouse-on-premise-without-ssl} +## Connecting Tableau online to ClickHouse (on-premise without SSL) {#connecting-tableau-online-to-clickhouse-on-premise-without-ssl} Login to your Tableau Cloud site and add a new Published Data Source. @@ -57,7 +57,7 @@ After that, all that remains is to click "Publish As" in the top right corner, a NB: if you want to use Tableau Online in combination with Tableau Desktop and share ClickHouse datasets between them, make sure you use Tableau Desktop with the default MySQL connector as well, following the setup guide that is displayed [here](https://www.tableau.com/support/drivers) if you select MySQL from the Data Source drop-down. If you have an M1 Mac, check [this troubleshooting thread](https://community.tableau.com/s/question/0D58b0000Ar6OhvCQE/unable-to-install-mysql-driver-for-m1-mac) for a driver installation workaround. -## Connecting Tableau Online to ClickHouse (cloud or on-premise setup with SSL) {#connecting-tableau-online-to-clickhouse-cloud-or-on-premise-setup-with-ssl} +## Connecting Tableau online to ClickHouse (cloud or on-premise setup with SSL) {#connecting-tableau-online-to-clickhouse-cloud-or-on-premise-setup-with-ssl} As it is not possible to provide the SSL certificates via the Tableau Online MySQL connection setup wizard, the only way is to use Tableau Desktop to set the connection up, and then export it to Tableau Online. This process is, however, pretty straightforward. diff --git a/docs/integrations/language-clients/csharp.md b/docs/integrations/language-clients/csharp.md index 5c4f686189a..ee7163c1464 100644 --- a/docs/integrations/language-clients/csharp.md +++ b/docs/integrations/language-clients/csharp.md @@ -12,7 +12,7 @@ integration: - website: 'https://github.com/ClickHouse/clickhouse-cs' --- -# ClickHouse C# client +# ClickHouse c# client The official C# client for connecting to ClickHouse. The client source code is available in the [GitHub repository](https://github.com/ClickHouse/clickhouse-cs). @@ -25,7 +25,7 @@ Originally developed by [Oleg V. Kozlyuk](https://github.com/DarkWanderer). --- -## Supported .NET versions {#supported-net-versions} +## Supported .net versions {#supported-net-versions} `ClickHouse.Driver` supports the following .NET versions: @@ -216,7 +216,7 @@ Console.WriteLine($"Rows written: {bulkCopy.RowsWritten}"); --- -### Performing SELECT queries {#performing-select-queries} +### Performing select queries {#performing-select-queries} Execute SELECT queries and process results: @@ -417,7 +417,7 @@ The ClickHouse .NET client integrates with the `Microsoft.Extensions.Logging` ab ### Quick start {#logging-quick-start} -#### Using ClickHouseConnection {#logging-clickhouseconnection} +#### Using clickhouseconnection {#logging-clickhouseconnection} ```csharp using ClickHouse.Driver.ADO; @@ -516,7 +516,7 @@ The driver uses dedicated categories so that you can fine-tune log levels per co | `ClickHouse.Driver.Transport` | `ClickHouseConnection` | Low-level HTTP streaming requests, compression flags, response status codes, and transport failures. | | `ClickHouse.Driver.BulkCopy` | `ClickHouseBulkCopy` | Metadata loading, batch operations, row counts, and upload completions. | -#### Example: Diagnosing connection issues {#logging-config-example} +#### Example: diagnosing connection issues {#logging-config-example} ```json { @@ -559,7 +559,7 @@ var settings = new ClickHouseClientSettings() --- -### ORM & Dapper support {#orm-support} +### Orm & dapper support {#orm-support} `ClickHouse.Driver` supports Dapper (with limitations). diff --git a/docs/integrations/language-clients/go/index.md b/docs/integrations/language-clients/go/index.md index 41e898fc534..39695622bda 100644 --- a/docs/integrations/language-clients/go/index.md +++ b/docs/integrations/language-clients/go/index.md @@ -111,7 +111,7 @@ func connect() (driver.Conn, error) { } ``` -### Run go mod tidy {#run-go-mod-tidy} +### Run Go mod tidy {#run-go-mod-tidy} ```bash go mod tidy @@ -513,7 +513,7 @@ conn.Exec(context.Background(), "INSERT INTO example VALUES (1, 'test-1')") Note the ability to pass a Context to the query. This can be used to pass specific query level settings - see [Using Context](#using-context). -### Batch Insert {#batch-insert} +### Batch insert {#batch-insert} To insert a large number of rows, the client provides batch semantics. This requires the preparation of a batch to which rows can be appended. This is finally sent via the `Send()` method. Batches are held in memory until `Send` is executed. @@ -655,7 +655,7 @@ Similar to insertion, the Scan method requires the target variables to be of an Finally, note the ability to pass a `Context` to the `Query` and `QueryRow` methods. This can be used for query level settings - see [Using Context](#using-context) for further details. -### Async Insert {#async-insert} +### Async insert {#async-insert} Asynchronous inserts are supported through the Async method. This allows the user to specify whether the client should wait for the server to complete the insert or respond once the data has been received. This effectively controls the parameter [wait_for_async_insert](/operations/settings/settings#wait_for_async_insert). @@ -694,7 +694,7 @@ for i := 0; i < 100; i++ { [Full Example](https://github.com/ClickHouse/clickhouse-go/blob/main/examples/clickhouse_api/async.go) -### Columnar Insert {#columnar-insert} +### Columnar insert {#columnar-insert} Inserts can be inserted in column format. This can provide performance benefits if the data is already orientated in this structure by avoiding the need to pivot to rows. @@ -1238,7 +1238,7 @@ if err = conn.QueryRow(ctx, "SELECT * FROM example").Scan(&point, &ring, &polygo [Full Example](https://github.com/ClickHouse/clickhouse-go/blob/main/examples/clickhouse_api/geo.go) -#### UUID {#uuid} +#### Uuid {#uuid} The UUID type is supported by the [github.com/google/uuid](https://github.com/google/uuid) package. Users can also send and marshal a UUID as a string or any type which implements `sql.Scanner` or `Stringify`. @@ -1390,7 +1390,7 @@ if err = conn.QueryRow(ctx, "SELECT * FROM example").Scan(&col1, &col2, &col3, & The client additionally supports the `sql.Null*` types e.g. `sql.NullInt64`. These are compatible with their equivalent ClickHouse types. -#### Big Ints - Int128, Int256, UInt128, UInt256 {#big-ints---int128-int256-uint128-uint256} +#### Big ints - int128, int256, uint128, uint256 {#big-ints---int128-int256-uint128-uint256} Number types larger than 64 bits are represented using the native go [big](https://pkg.go.dev/math/big) package. @@ -2121,7 +2121,7 @@ _, err = conn.Exec("INSERT INTO example VALUES (1, 'test-1')") This method does not support receiving a context - by default, it executes with the background context. Users can use `ExecContext` if this is needed - see [Using Context](#using-context). -### Batch Insert {#batch-insert-1} +### Batch insert {#batch-insert-1} Batch semantics can be achieved by creating a `sql.Tx` via the `Being` method. From this, a batch can be obtained using the `Prepare` method with the `INSERT` statement. This returns a `sql.Stmt` to which rows can be appended using the `Exec` method. The batch will be accumulated in memory until `Commit` is executed on the original `sql.Tx`. @@ -2201,7 +2201,7 @@ for rows.Next() { [Full Example](https://github.com/ClickHouse/clickhouse-go/blob/main/examples/std/query_rows.go) -### Async Insert {#async-insert-1} +### Async insert {#async-insert-1} Asynchronous inserts can be achieved by executing an insert via the `ExecContext` method. This should be passed a context with asynchronous mode enabled, as shown below. This allows the user to specify whether the client should wait for the server to complete the insert or respond once the data has been received. This effectively controls the parameter [wait_for_async_insert](/operations/settings/settings#wait_for_async_insert). @@ -2232,7 +2232,7 @@ ctx := clickhouse.Context(context.Background(), clickhouse.WithStdAsync(false)) [Full Example](https://github.com/ClickHouse/clickhouse-go/blob/main/examples/std/async.go) -### Columnar Insert {#columnar-insert-1} +### Columnar insert {#columnar-insert-1} Not supported using the standard interface. diff --git a/docs/integrations/language-clients/java/r2dbc.md b/docs/integrations/language-clients/java/r2dbc.md index 6ad36d9f3bc..1fafb2f949f 100644 --- a/docs/integrations/language-clients/java/r2dbc.md +++ b/docs/integrations/language-clients/java/r2dbc.md @@ -12,9 +12,9 @@ import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; -# R2DBC driver +# R2dbc driver -## R2DBC driver {#r2dbc-driver} +## R2dbc driver {#r2dbc-driver} [R2DBC](https://r2dbc.io/) wrapper of async Java client for ClickHouse. diff --git a/docs/integrations/language-clients/js.md b/docs/integrations/language-clients/js.md index e497d87379c..41a064d606b 100644 --- a/docs/integrations/language-clients/js.md +++ b/docs/integrations/language-clients/js.md @@ -15,7 +15,7 @@ integration: import ConnectionDetails from '@site/docs/_snippets/_gather_your_details_http.mdx'; import ExperimentalBadge from '@theme/badges/ExperimentalBadge'; -# ClickHouse JS +# ClickHouse js The official JS client for connecting to ClickHouse. The client is written in TypeScript and provides typings for the client public API. @@ -221,7 +221,7 @@ There is no guarantee the same connection in a pool will be used for subsequent See also: [Keep-Alive configuration](./js.md#keep-alive-configuration-nodejs-only). -### Query ID {#query-id} +### Query id {#query-id} Every method that sends a query or a statement (`command`, `exec`, `insert`, `select`) will provide `query_id` in the result. This unique identifier is assigned by the client per query, and might be useful to fetch the data from `system.query_log`, if it is enabled in the [server configuration](/operations/server-configuration-parameters/settings), or cancel long-running queries (see [the example](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/cancel_query.ts)). If necessary, `query_id` can be overridden by the user in `command`/`query`/`exec`/`insert` methods params. @@ -936,7 +936,7 @@ await client.query({ See [this example](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/insert_decimals.ts) for more details. -### Integral types: Int64, Int128, Int256, UInt64, UInt128, UInt256 {#integral-types-int64-int128-int256-uint64-uint128-uint256} +### Integral types: int64, int128, int256, uint64, uint128, uint256 {#integral-types-int64-int128-int256-uint64-uint128-uint256} Though the server can accept it as a number, it is returned as a string in `JSON*` family output formats to avoid integer overflow as max values for these types are bigger than `Number.MAX_SAFE_INTEGER`. diff --git a/docs/integrations/language-clients/moose-olap.md b/docs/integrations/language-clients/moose-olap.md index 6d6458e0ca9..095c5868651 100644 --- a/docs/integrations/language-clients/moose-olap.md +++ b/docs/integrations/language-clients/moose-olap.md @@ -10,7 +10,7 @@ doc_type: 'guide' import CommunityMaintainedBadge from '@theme/badges/CommunityMaintained'; -# Developing on ClickHouse with Moose OLAP +# Developing on ClickHouse with moose OLAP @@ -18,7 +18,7 @@ import CommunityMaintainedBadge from '@theme/badges/CommunityMaintained'; Moose OLAP offers developer-friendly abstractions and ORM-like functionality, built natively for ClickHouse. -## Key features of Moose OLAP {#key-features} +## Key features of moose OLAP {#key-features} - **Schemas as code**: Define your ClickHouse tables in TypeScript or Python with type safety and IDE autocompletion - **Type-safe queries**: Write SQL queries with type checking and autocompletion support @@ -41,7 +41,7 @@ Or follow this guide to get up and running with Moose OLAP on an existing ClickH -### Install Moose {#step-1-install-moose} +### Install moose {#step-1-install-moose} Install the Moose CLI globally to your system: @@ -51,7 +51,7 @@ bash -i <(curl -fsSL https://fiveonefour.com/install.sh) moose ### Set up your project {#step-2-set-up-project} -#### Option A: Use your own existing ClickHouse deployment {#option-a-use-own-clickhouse} +#### Option a: use your own existing ClickHouse deployment {#option-a-use-own-clickhouse} **Important**: Your production ClickHouse will remain untouched. This will just initialize a new Moose OLAP project with data models derived from your ClickHouse tables. @@ -69,7 +69,7 @@ Your ClickHouse connection string should be in this format: https://username:password@host:port/?database=database_name ``` -#### Option B: use ClickHouse playground {#option-b-use-clickhouse-playground} +#### Option b: use ClickHouse playground {#option-b-use-clickhouse-playground} Don't have ClickHouse up and running yet? Use the ClickHouse Playground to try out Moose OLAP! @@ -129,7 +129,7 @@ moose seed --connection-string --limit 100 moose seed --connection-string https://explorer:@play.clickhouse.com:443/?database=default --limit 100 ``` -### Building with Moose OLAP {#step-7-building-with-moose-olap} +### Building with moose OLAP {#step-7-building-with-moose-olap} Now that you have your Tables defined in code, you get the same benefits as ORM data models in web apps - type safety and autocomplete when building APIs and Materialized Views on top of your analytical data. As a next step, you could try: * Building a REST API with [Moose API](https://docs.fiveonefour.com/moose/apis) diff --git a/docs/integrations/language-clients/python/advanced-inserting.md b/docs/integrations/language-clients/python/advanced-inserting.md index e5fb625c8b2..73dcf4f302d 100644 --- a/docs/integrations/language-clients/python/advanced-inserting.md +++ b/docs/integrations/language-clients/python/advanced-inserting.md @@ -8,9 +8,9 @@ title: 'Advanced Inserting' doc_type: 'reference' --- -## Inserting data with ClickHouse Connect: Advanced usage {#inserting-data-with-clickhouse-connect--advanced-usage} +## Inserting data with ClickHouse connect: advanced usage {#inserting-data-with-clickhouse-connect--advanced-usage} -### InsertContexts {#insertcontexts} +### Insertcontexts {#insertcontexts} ClickHouse Connect executes all inserts within an `InsertContext`. The `InsertContext` includes all the values sent as arguments to the client `insert` method. In addition, when an `InsertContext` is originally constructed, ClickHouse Connect retrieves the data types for the insert columns required for efficient Native format inserts. By reusing the `InsertContext` for multiple inserts, this "pre-query" is avoided and inserts are executed more quickly and efficiently. @@ -95,7 +95,7 @@ df = pd.DataFrame({ client.insert_df("users", df) ``` -#### PyArrow Table insert {#pyarrow-table-insert} +#### Pyarrow table insert {#pyarrow-table-insert} ```python import clickhouse_connect @@ -112,7 +112,7 @@ arrow_table = pa.table({ client.insert_arrow("users", arrow_table) ``` -#### Arrow-backed DataFrame insert (pandas 2.x) {#arrow-backed-dataframe-insert-pandas-2} +#### Arrow-backed DataFrame insert (Pandas 2.x) {#arrow-backed-dataframe-insert-pandas-2} ```python import clickhouse_connect @@ -134,7 +134,7 @@ client.insert_df_arrow("users", df) When inserting Python `datetime.datetime` objects into ClickHouse `DateTime` or `DateTime64` columns, ClickHouse Connect automatically handles timezone information. Since ClickHouse stores all DateTime values internally as timezone-naive Unix timestamps (seconds or fractional seconds since the epoch), timezone conversion happens automatically on the client side during insertion. -#### Timezone-aware datetime objects {#timezone-aware-datetime-objects} +#### Timezone-aware DateTime objects {#timezone-aware-datetime-objects} If you insert a timezone-aware Python `datetime.datetime` object, ClickHouse Connect will automatically call `.timestamp()` to convert it to a Unix timestamp, which correctly accounts for the timezone offset. This means you can insert datetime objects from any timezone, and they will be correctly stored as their UTC equivalent timestamp. @@ -171,7 +171,7 @@ In this example, all three datetime objects represent different points in time b When using pytz, you must use the `localize()` method to attach timezone information to a naive datetime. Passing `tzinfo=` directly to the datetime constructor will use incorrect historical offsets. For UTC, `tzinfo=pytz.UTC` works correctly. See [pytz docs](https://pythonhosted.org/pytz/#localized-times-and-date-arithmetic) for more info. ::: -#### Timezone-naive datetime objects {#timezone-naive-datetime-objects} +#### Timezone-naive DateTime objects {#timezone-naive-datetime-objects} If you insert a timezone-naive Python `datetime.datetime` object (one without `tzinfo`), the `.timestamp()` method will interpret it as being in the system's local timezone. To avoid ambiguity, it's recommended to: diff --git a/docs/integrations/language-clients/python/advanced-querying.md b/docs/integrations/language-clients/python/advanced-querying.md index de999c94e0f..a8900de28ca 100644 --- a/docs/integrations/language-clients/python/advanced-querying.md +++ b/docs/integrations/language-clients/python/advanced-querying.md @@ -8,9 +8,9 @@ title: 'Advanced Querying' doc_type: 'reference' --- -# Querying data with ClickHouse Connect: Advanced usage {#querying-data-with-clickhouse-connect--advanced-usage} +# Querying data with ClickHouse connect: advanced usage {#querying-data-with-clickhouse-connect--advanced-usage} -## QueryContexts {#querycontexts} +## Querycontexts {#querycontexts} ClickHouse Connect executes standard queries within a `QueryContext`. The `QueryContext` contains the key structures that are used to build queries against the ClickHouse database, and the configuration used to process the result into a `QueryResult` or other response data structure. That includes the query itself, parameters, settings, read formats, and other properties. @@ -59,7 +59,7 @@ When using one of the Client `query_*_stream` methods, results are returned on a Because of limitations in the HTTP protocol, if blocks are processed at a rate significantly slower than the ClickHouse server is streaming data, the ClickHouse server will close the connection, resulting in an Exception being thrown in the processing thread. Some of this can be mitigated by increasing the buffer size of the HTTP streaming buffer (which defaults to 10 megabytes) using the common `http_buffer_size` setting. Large `http_buffer_size` values should be okay in this situation if there is sufficient memory available to the application. Data in the buffer is stored compressed if using `lz4` or `zstd` compression, so using those compression types will increase the overall buffer available. -### StreamContexts {#streamcontexts} +### Streamcontexts {#streamcontexts} Each of the `query_*_stream` methods (like `query_row_block_stream`) returns a ClickHouse `StreamContext` object, which is a combined Python context/generator. This is the basic usage: @@ -134,7 +134,7 @@ with client.query_row_block_stream("SELECT number, number * 2 FROM system.number # Received block with 34591 rows ``` -#### Stream Pandas DataFrames {#stream-pandas-dataframes} +#### Stream Pandas dataframes {#stream-pandas-dataframes} ```python import clickhouse_connect @@ -177,11 +177,11 @@ with client.query_arrow_stream("SELECT * FROM large_table") as stream: # Received Arrow batch with 34591 rows ``` -## NumPy, Pandas, and Arrow queries {#numpy-pandas-and-arrow-queries} +## Numpy, Pandas, and Arrow queries {#numpy-pandas-and-arrow-queries} ClickHouse Connect provides specialized query methods for working with NumPy, Pandas, and Arrow data structures. These methods allow you to retrieve query results directly in these popular data formats without manual conversion. -### NumPy queries {#numpy-queries} +### Numpy queries {#numpy-queries} The `query_np` method returns query results as a NumPy array instead of a ClickHouse Connect `QueryResult`. @@ -230,7 +230,7 @@ print(df) # 4 4 8 ``` -### PyArrow queries {#pyarrow-queries} +### Pyarrow queries {#pyarrow-queries} The `query_arrow` method returns query results as a PyArrow Table. It utilizes the ClickHouse `Arrow` format directly, so it only accepts three arguments in common with the main `query` method: `query`, `parameters`, and `settings`. In addition, there is an additional argument, `use_strings`, which determines whether the Arrow Table will render ClickHouse String types as strings (if True) or bytes (if False). @@ -256,7 +256,7 @@ print(arrow_table) # str: [["0","1","2"]] ``` -### Arrow-backed DataFrames {#arrow-backed-dataframes} +### Arrow-backed dataframes {#arrow-backed-dataframes} ClickHouse Connect supports fast, memory‑efficient DataFrame creation from Arrow results via the `query_df_arrow` and `query_df_arrow_stream` methods. These are thin wrappers around the Arrow query methods and perform zero‑copy conversions to DataFrames where possible: @@ -265,7 +265,7 @@ ClickHouse Connect supports fast, memory‑efficient DataFrame creation from Arr - For `dataframe_library='polars'`, returns a Polars DataFrame created from the Arrow table (`pl.from_arrow`), which is similarly efficient and can be zero‑copy depending on the data. - `query_df_arrow_stream`: Streams results as a sequence of DataFrames (pandas 2.x or Polars) converted from Arrow stream batches. -#### Query to Arrow-backed DataFrame {#query-to-arrow-backed-dataframe} +#### Query to arrow-backed DataFrame {#query-to-arrow-backed-dataframe} ```python import clickhouse_connect diff --git a/docs/integrations/language-clients/python/advanced-usage.md b/docs/integrations/language-clients/python/advanced-usage.md index 5832d0b0100..26166dd144a 100644 --- a/docs/integrations/language-clients/python/advanced-usage.md +++ b/docs/integrations/language-clients/python/advanced-usage.md @@ -8,7 +8,7 @@ title: 'Advanced Usage' doc_type: 'reference' --- -# Advanced Usage {#advanced-usage} +# Advanced usage {#advanced-usage} ## Raw API {#raw-api} @@ -84,7 +84,7 @@ Because each query or insert executed maintains state in its own `QueryContext` Additionally, in an application that has two or more queries and/or inserts "in flight" at the same time, there are two further considerations to keep in mind. The first is the ClickHouse "session" associated with the query/insert, and the second is the HTTP connection pool used by ClickHouse Connect Client instances. -## AsyncClient wrapper {#asyncclient-wrapper} +## Asyncclient wrapper {#asyncclient-wrapper} ClickHouse Connect provides an async wrapper over the regular `Client`, so that it is possible to use the client in an `asyncio` environment. @@ -114,7 +114,7 @@ Note: Unlike the regular `Client`, the `AsyncClient` enforces `autogenerate_sess See also: [run_async example](https://github.com/ClickHouse/clickhouse-connect/blob/main/examples/run_async.py). -## Managing ClickHouse session IDs {#managing-clickhouse-session-ids} +## Managing ClickHouse session ids {#managing-clickhouse-session-ids} Each ClickHouse query occurs within the context of a ClickHouse "session". Sessions are currently used for two purposes: - To associate specific ClickHouse settings with multiple queries (see the [user settings](/operations/settings/settings.md)). The ClickHouse `SET` command is used to change the settings for the scope of a user session. diff --git a/docs/integrations/language-clients/python/driver-api.md b/docs/integrations/language-clients/python/driver-api.md index 0b5a22d2f7a..70ddb9cbd34 100644 --- a/docs/integrations/language-clients/python/driver-api.md +++ b/docs/integrations/language-clients/python/driver-api.md @@ -8,7 +8,7 @@ title: 'ClickHouse Connect Driver API' doc_type: 'reference' --- -# ClickHouse Connect driver API {#clickhouse-connect-driver-api} +# ClickHouse connect driver API {#clickhouse-connect-driver-api} :::note Passing keyword arguments is recommended for most api methods given the number of possible arguments, most of which are optional. @@ -16,7 +16,7 @@ Passing keyword arguments is recommended for most api methods given the number o *Methods not documented here are not considered part of the API, and may be removed or changed.* ::: -## Client Initialization {#client-initialization} +## Client initialization {#client-initialization} The `clickhouse_connect.driver.client` class provides the primary interface between a Python application and the ClickHouse database server. Use the `clickhouse_connect.get_client` function to obtain a Client instance, which accepts the following arguments: @@ -118,7 +118,7 @@ print(client.database) # Output: 'github' ``` -## Client Lifecycle and Best Practices {#client-lifecycle-and-best-practices} +## Client lifecycle and best practices {#client-lifecycle-and-best-practices} Creating a ClickHouse Connect client is an expensive operation that involves establishing a connection, retrieving server metadata, and initializing settings. Follow these best practices for optimal performance: @@ -356,7 +356,7 @@ settings = {'merge_tree_min_rows_for_concurrent_read': 65535, client.query("SELECT event_type, sum(timeout) FROM event_errors WHERE event_time > '2022-08-01'", settings=settings) ``` -## Client `command` Method {#client-command-method} +## Client `command` method {#client-command-method} Use the `Client.command` method to send SQL queries to the ClickHouse server that do not normally return data or that return a single primitive or array value rather than a full dataset. This method takes the following parameters: @@ -452,7 +452,7 @@ result = client.command( ) ``` -## Client `query` Method {#client-query-method} +## Client `query` method {#client-query-method} The `Client.query` method is the primary way to retrieve a single "batch" dataset from the ClickHouse server. It utilizes the Native ClickHouse format over HTTP to transmit large datasets (up to approximately one million rows) efficiently. This method takes the following parameters: @@ -603,7 +603,7 @@ The `*_stream` properties return a Python Context that can be used as an iterato The complete details of streaming query results (using StreamContext objects) are outlined in [Advanced Queries (Streaming Queries)](advanced-querying.md#streaming-queries). -## Consuming query results with NumPy, Pandas or Arrow {#consuming-query-results-with-numpy-pandas-or-arrow} +## Consuming query results with numpy, Pandas or Arrow {#consuming-query-results-with-numpy-pandas-or-arrow} ClickHouse Connect provides specialized query methods for NumPy, Pandas, and Arrow data formats. For detailed information on using these methods, including examples, streaming capabilities, and advanced type handling, see [Advanced Querying (NumPy, Pandas and Arrow Queries)](advanced-querying.md#numpy-pandas-and-arrow-queries). @@ -611,7 +611,7 @@ ClickHouse Connect provides specialized query methods for NumPy, Pandas, and Arr For streaming large result sets, ClickHouse Connect provides multiple streaming methods. See [Advanced Queries (Streaming Queries)](advanced-querying.md#streaming-queries) for details and examples. -## Client `insert` Method {#client-insert-method} +## Client `insert` method {#client-insert-method} For the common use case of inserting multiple records into ClickHouse, there is the `Client.insert` method. It takes the following parameters: @@ -717,7 +717,7 @@ client.insert( ) ``` -## File Inserts {#file-inserts} +## File inserts {#file-inserts} For inserting data directly from files into ClickHouse tables, see [Advanced Inserting (File Inserts)](advanced-inserting.md#file-inserts). @@ -741,11 +741,11 @@ The functions and the DT64Param class in the `clickhouse_connect.driver.binding` For information on using ClickHouse Connect in multithreaded, multiprocess, and async/event-driven applications, see [Advanced Usage (Multithreaded, multiprocess, and async/event driven use cases)](advanced-usage.md#multithreaded-multiprocess-and-asyncevent-driven-use-cases). -## AsyncClient wrapper {#asyncclient-wrapper} +## Asyncclient wrapper {#asyncclient-wrapper} For information on using the AsyncClient wrapper for asyncio environments, see [Advanced Usage (AsyncClient wrapper)](advanced-usage.md#asyncclient-wrapper). -## Managing ClickHouse Session IDs {#managing-clickhouse-session-ids} +## Managing ClickHouse session ids {#managing-clickhouse-session-ids} For information on managing ClickHouse session IDs in multi-threaded or concurrent applications, see [Advanced Usage (Managing ClickHouse Session IDs)](advanced-usage.md#managing-clickhouse-session-ids). diff --git a/docs/integrations/language-clients/python/index.md b/docs/integrations/language-clients/python/index.md index c8d3f550d45..9e4fb26d1d2 100644 --- a/docs/integrations/language-clients/python/index.md +++ b/docs/integrations/language-clients/python/index.md @@ -77,7 +77,7 @@ There are two examples shown for connecting to ClickHouse: - Connecting to a ClickHouse server on localhost. - Connecting to a ClickHouse Cloud service. -#### Use a ClickHouse Connect client instance to connect to a ClickHouse server on localhost: {#use-a-clickhouse-connect-client-instance-to-connect-to-a-clickhouse-server-on-localhost} +#### Use a ClickHouse connect client instance to connect to a ClickHouse server on localhost: {#use-a-clickhouse-connect-client-instance-to-connect-to-a-clickhouse-server-on-localhost} ```python import clickhouse_connect @@ -85,7 +85,7 @@ import clickhouse_connect client = clickhouse_connect.get_client(host='localhost', username='default', password='password') ``` -#### Use a ClickHouse Connect client instance to connect to a ClickHouse Cloud service: {#use-a-clickhouse-connect-client-instance-to-connect-to-a-clickhouse-cloud-service} +#### Use a ClickHouse connect client instance to connect to a ClickHouse Cloud service: {#use-a-clickhouse-connect-client-instance-to-connect-to-a-clickhouse-cloud-service} :::tip Use the connection details gathered earlier. ClickHouse Cloud services require TLS, so use port 8443. diff --git a/docs/integrations/language-clients/python/sqlalchemy.md b/docs/integrations/language-clients/python/sqlalchemy.md index 3252d336488..29dd0c9e9cf 100644 --- a/docs/integrations/language-clients/python/sqlalchemy.md +++ b/docs/integrations/language-clients/python/sqlalchemy.md @@ -10,7 +10,7 @@ doc_type: 'reference' ClickHouse Connect includes a SQLAlchemy dialect (`clickhousedb`) built on top of the core driver. It targets SQLAlchemy Core APIs and supports SQLAlchemy 1.4.40+ and 2.0.x. -## Connect with SQLAlchemy {#sqlalchemy-connect} +## Connect with sqlalchemy {#sqlalchemy-connect} Create an engine using either `clickhousedb://` or `clickhousedb+connect://` URLs. Query parameters map to ClickHouse settings, client options, and HTTP/TLS transport options. @@ -100,7 +100,7 @@ with engine.begin() as conn: Reflected columns include dialect-specific attributes such as `clickhousedb_default_type`, `clickhousedb_codec_expression`, and `clickhousedb_ttl_expression` when present on the server. -## Inserts (Core and basic ORM) {#sqlalchemy-inserts} +## Inserts (core and basic orm) {#sqlalchemy-inserts} Inserts work via SQLAlchemy Core as well as with simple ORM models for convenience. diff --git a/docs/integrations/language-clients/rust.md b/docs/integrations/language-clients/rust.md index 32fd185c431..81daa43003b 100644 --- a/docs/integrations/language-clients/rust.md +++ b/docs/integrations/language-clients/rust.md @@ -223,7 +223,7 @@ inserter.end().await?; ``` ::: -### Executing DDLs {#executing-ddls} +### Executing ddls {#executing-ddls} With a single-node deployment, it is enough to execute DDLs like this: @@ -257,7 +257,7 @@ let numbers = client Besides `query`, it works similarly with `insert` and `inserter` methods; additionally, the same method can be called on the `Client` instance to set global settings for all queries. -### Query ID {#query-id} +### Query id {#query-id} Using `.with_option`, you can set the `query_id` option to identify queries in the ClickHouse query log. @@ -277,7 +277,7 @@ If you set `query_id` manually, make sure that it is unique. UUIDs are a good ch See also: [query_id example](https://github.com/ClickHouse/clickhouse-rs/blob/main/examples/query_id.rs) in the client repo. -### Session ID {#session-id} +### Session id {#session-id} Similarly to `query_id`, you can set the `session_id` to execute the statements in the same session. `session_id` can be set either globally on the client level, or per `query`, `insert`, or `inserter` call. @@ -507,7 +507,7 @@ See [the example](https://github.com/ClickHouse/clickhouse-rs/tree/main/examples ## Troubleshooting {#troubleshooting} -### CANNOT_READ_ALL_DATA {#cannot_read_all_data} +### Cannot_read_all_data {#cannot_read_all_data} The most common cause for the `CANNOT_READ_ALL_DATA` error is that the row definition on the application side does match that in ClickHouse. diff --git a/docs/integrations/sql-clients/datagrip.md b/docs/integrations/sql-clients/datagrip.md index 4418a9bb6d5..9814260b89e 100644 --- a/docs/integrations/sql-clients/datagrip.md +++ b/docs/integrations/sql-clients/datagrip.md @@ -27,10 +27,10 @@ import CommunityMaintainedBadge from '@theme/badges/CommunityMaintained'; DataGrip is available at https://www.jetbrains.com/datagrip/ -## 1. Gather your connection details {#1-gather-your-connection-details} +## 1. gather your connection details {#1-gather-your-connection-details} -## 2. Load the ClickHouse driver {#2-load-the-clickhouse-driver} +## 2. load the ClickHouse driver {#2-load-the-clickhouse-driver} 1. Launch DataGrip, and on the **Data Sources** tab in the **Data Sources and Drivers** dialog, click the **+** icon @@ -51,7 +51,7 @@ DataGrip is available at https://www.jetbrains.com/datagrip/ DataGrip Drivers tab showing ClickHouse driver installation -## 3. Connect to ClickHouse {#3-connect-to-clickhouse} +## 3. connect to ClickHouse {#3-connect-to-clickhouse} - Specify your database connection details, and click **Test Connection**. In step one you gathered your connection details - fill in the host URL, port, username, password, and database name, then test the connection. diff --git a/docs/integrations/sql-clients/dbeaver.md b/docs/integrations/sql-clients/dbeaver.md index 11ad493936c..0ab22b0c8cc 100644 --- a/docs/integrations/sql-clients/dbeaver.md +++ b/docs/integrations/sql-clients/dbeaver.md @@ -21,7 +21,7 @@ import dbeaver_sql_editor from '@site/static/images/integrations/sql-clients/dbe import dbeaver_query_log_select from '@site/static/images/integrations/sql-clients/dbeaver-query-log-select.png'; import ClickHouseSupportedBadge from '@theme/badges/ClickHouseSupported'; -# Connect DBeaver to ClickHouse +# Connect dbeaver to ClickHouse @@ -31,7 +31,7 @@ DBeaver is available in multiple offerings. In this guide [DBeaver Community](ht Please use DBeaver version 23.1.0 or above for improved support of `Nullable` columns in ClickHouse. ::: -## 1. Gather your ClickHouse details {#1-gather-your-clickhouse-details} +## 1. gather your ClickHouse details {#1-gather-your-clickhouse-details} DBeaver uses JDBC over HTTP(S) to connect to ClickHouse; you need: @@ -40,11 +40,11 @@ DBeaver uses JDBC over HTTP(S) to connect to ClickHouse; you need: - username - password -## 2. Download DBeaver {#2-download-dbeaver} +## 2. download dbeaver {#2-download-dbeaver} DBeaver is available at https://dbeaver.io/download/ -## 3. Add a database {#3-add-a-database} +## 3. add a database {#3-add-a-database} - Either use the **Database > New Database Connection** menu or the **New Database Connection** icon in the **Database Navigator** to bring up the **Connect to a database** dialog: @@ -72,7 +72,7 @@ If DBeaver detects that you do not have the ClickHouse driver installed it will Test the connection -## 4. Query ClickHouse {#4-query-clickhouse} +## 4. query ClickHouse {#4-query-clickhouse} Open a query editor and run a query. diff --git a/docs/integrations/sql-clients/dbvisualizer.md b/docs/integrations/sql-clients/dbvisualizer.md index afad69a1598..e934ce7781f 100644 --- a/docs/integrations/sql-clients/dbvisualizer.md +++ b/docs/integrations/sql-clients/dbvisualizer.md @@ -15,25 +15,25 @@ import Image from '@theme/IdealImage'; import dbvisualizer_driver_manager from '@site/static/images/integrations/sql-clients/dbvisualizer-driver-manager.png'; import CommunityMaintainedBadge from '@theme/badges/CommunityMaintained'; -# Connecting DbVisualizer to ClickHouse +# Connecting dbvisualizer to ClickHouse -## Start or download DbVisualizer {#start-or-download-dbvisualizer} +## Start or download dbvisualizer {#start-or-download-dbvisualizer} DbVisualizer is available at https://www.dbvis.com/download/ -## 1. Gather your connection details {#1-gather-your-connection-details} +## 1. gather your connection details {#1-gather-your-connection-details} -## 2. Built-in JDBC driver management {#2-built-in-jdbc-driver-management} +## 2. built-in jdbc driver management {#2-built-in-jdbc-driver-management} DbVisualizer has the most up-to-date JDBC drivers for ClickHouse included. It has full JDBC driver management built right in that points to the latest releases as well as historical versions for the drivers. DbVisualizer driver manager interface showing ClickHouse JDBC driver configuration -## 3. Connect to ClickHouse {#3-connect-to-clickhouse} +## 3. connect to ClickHouse {#3-connect-to-clickhouse} To connect a database with DbVisualizer, you must first create and setup a Database Connection. diff --git a/docs/integrations/sql-clients/marimo.md b/docs/integrations/sql-clients/marimo.md index b793d258767..64be5406a7b 100644 --- a/docs/integrations/sql-clients/marimo.md +++ b/docs/integrations/sql-clients/marimo.md @@ -26,7 +26,7 @@ import CommunityMaintainedBadge from '@theme/badges/CommunityMaintained'; Connect to ClickHouse -## 1. Install marimo with SQL support {#install-marimo-sql} +## 1. install marimo with SQL support {#install-marimo-sql} ```shell pip install "marimo[sql]" clickhouse_connect @@ -34,7 +34,7 @@ marimo edit clickhouse_demo.py ``` This should open up a web browser running on localhost. -## 2. Connecting to ClickHouse. {#connect-to-clickhouse} +## 2. connecting to ClickHouse. {#connect-to-clickhouse} Navigate to the datasources panel on the left side of the marimo editor and click on 'Add database'. @@ -48,7 +48,7 @@ You will then have a cell that can be run to establish a connection. Run the cell to connect to ClickHouse -## 3. Run SQL {#run-sql} +## 3. run SQL {#run-sql} Once you have set up a connection, you can create a new SQL cell and choose the clickhouse engine. diff --git a/docs/integrations/sql-clients/qstudio.md b/docs/integrations/sql-clients/qstudio.md index ee258fd6151..24cec58729d 100644 --- a/docs/integrations/sql-clients/qstudio.md +++ b/docs/integrations/sql-clients/qstudio.md @@ -13,7 +13,7 @@ import qstudio_running_query from '@site/static/images/integrations/sql-clients/ import Image from '@theme/IdealImage'; import CommunityMaintainedBadge from '@theme/badges/CommunityMaintained'; -# Connect QStudio to ClickHouse +# Connect qstudio to ClickHouse @@ -21,7 +21,7 @@ QStudio is a free SQL GUI, it allows running SQL scripts, easy browsing of table QStudio connects to ClickHouse using JDBC. -## 1. Gather your ClickHouse details {#1-gather-your-clickhouse-details} +## 1. gather your ClickHouse details {#1-gather-your-clickhouse-details} QStudio uses JDBC over HTTP(S) to connect to ClickHouse; you need: @@ -32,11 +32,11 @@ QStudio uses JDBC over HTTP(S) to connect to ClickHouse; you need: -## 2. Download QStudio {#2-download-qstudio} +## 2. download qstudio {#2-download-qstudio} QStudio is available at https://www.timestored.com/qstudio/download/ -## 3. Add a database {#3-add-a-database} +## 3. add a database {#3-add-a-database} - When you first open QStudio click on the menu options **Server->Add Server** or on the add server button on the toolbar. - Then set the details: @@ -53,7 +53,7 @@ QStudio is available at https://www.timestored.com/qstudio/download/ If QStudio detects that you do not have the ClickHouse JDBC driver installed, it will offer to download them for you: -## 4. Query ClickHouse {#4-query-clickhouse} +## 4. query ClickHouse {#4-query-clickhouse} - Open a query editor and run a query. You can run queries by - Ctrl + e - Runs highlighted text diff --git a/docs/integrations/sql-clients/sql-console.md b/docs/integrations/sql-clients/sql-console.md index 1ac317a4f4a..a95097e6bc8 100644 --- a/docs/integrations/sql-clients/sql-console.md +++ b/docs/integrations/sql-clients/sql-console.md @@ -47,7 +47,7 @@ import adjust_axis_scale from '@site/static/images/cloud/sqlconsole/adjust-axis- import give_a_query_a_name from '@site/static/images/cloud/sqlconsole/give-a-query-a-name.png' import save_the_query from '@site/static/images/cloud/sqlconsole/save-the-query.png' -# SQL Console +# SQL console SQL console is the fastest and easiest way to explore and query your databases in ClickHouse Cloud. You can use the SQL console to: @@ -175,7 +175,7 @@ You can also use the save button or `cmd / ctrl + s` keyboard shortcut to save a Save button in the query editor toolbar -## Using GenAI to manage queries {#using-genai-to-manage-queries} +## Using genai to manage queries {#using-genai-to-manage-queries} This feature allows users to write queries as natural language questions and have the query console create SQL queries based on the context of the available tables. GenAI can also help users debug their queries. diff --git a/docs/integrations/sql-clients/tablum.md b/docs/integrations/sql-clients/tablum.md index e4c39eda4d6..5f748536ae0 100644 --- a/docs/integrations/sql-clients/tablum.md +++ b/docs/integrations/sql-clients/tablum.md @@ -27,13 +27,13 @@ import CommunityMaintainedBadge from '@theme/badges/CommunityMaintained'; You can install a self-hosted version of TABLUM.IO on your Linux server in docker. ::: -## 1. Sign up or sign in to the service {#1-sign-up-or-sign-in-to-the-service} +## 1. sign up or sign in to the service {#1-sign-up-or-sign-in-to-the-service} First, sign up to TABLUM.IO using your email or use a quick-login via accounts in Google or Facebook. TABLUM.IO login page -## 2. Add a ClickHouse connector {#2-add-a-clickhouse-connector} +## 2. add a ClickHouse connector {#2-add-a-clickhouse-connector} Gather your ClickHouse connection details, navigate to the **Connector** tab, and fill in the host URL, port, username, password, database name, and connector's name. After completing these fields, click on **Test connection** button to validate the details and then click on **Save connector for me** to make it persistent. @@ -47,13 +47,13 @@ Typically, the port is 8443 when using TLS or 8123 when not using TLS. Adding a ClickHouse connector in TABLUM.IO -## 3. Select the connector {#3-select-the-connector} +## 3. select the connector {#3-select-the-connector} Navigate to the **Dataset** tab. Select recently created ClickHouse connector in the dropdown. In the right panel, you will see the list of available tables and schemas. Selecting the ClickHouse connector in TABLUM.IO -## 4. Input a SQL query and run it {#4-input-a-sql-query-and-run-it} +## 4. input a SQL query and run it {#4-input-a-sql-query-and-run-it} Type a query in the SQL Console and press **Run Query**. The results will be displayed as a spreadsheet. diff --git a/docs/integrations/tools/data-integration/easypanel/index.md b/docs/integrations/tools/data-integration/easypanel/index.md index f0ac1717b67..6e325f21a2e 100644 --- a/docs/integrations/tools/data-integration/easypanel/index.md +++ b/docs/integrations/tools/data-integration/easypanel/index.md @@ -9,7 +9,7 @@ doc_type: 'guide' import CommunityMaintainedBadge from '@theme/badges/CommunityMaintained'; -# Deploying ClickHouse on Easypanel +# Deploying ClickHouse on easypanel diff --git a/docs/integrations/tools/data-integration/index.md b/docs/integrations/tools/data-integration/index.md index 0b42e03cd7a..2c295c25052 100644 --- a/docs/integrations/tools/data-integration/index.md +++ b/docs/integrations/tools/data-integration/index.md @@ -6,7 +6,7 @@ description: 'Landing page for the data integrations section' doc_type: 'landing-page' --- -# Data Integrations +# Data integrations | Page | Description | |-----------|---------------------------------------------------------------------------------------------------------------------------------| diff --git a/docs/integrations/tools/data-integration/retool/index.md b/docs/integrations/tools/data-integration/retool/index.md index dfca46adbe5..15414e0d1bf 100644 --- a/docs/integrations/tools/data-integration/retool/index.md +++ b/docs/integrations/tools/data-integration/retool/index.md @@ -23,10 +23,10 @@ import PartnerBadge from '@theme/badges/PartnerBadge'; -## 1. Gather your connection details {#1-gather-your-connection-details} +## 1. gather your connection details {#1-gather-your-connection-details} -## 2. Create a ClickHouse resource {#2-create-a-clickhouse-resource} +## 2. create a ClickHouse resource {#2-create-a-clickhouse-resource} Login to your Retool account and navigate to the _Resources_ tab. Choose "Create New" -> "Resource": diff --git a/docs/integrations/tools/data-integration/splunk/index.md b/docs/integrations/tools/data-integration/splunk/index.md index 4ef6ade244a..490fe25841e 100644 --- a/docs/integrations/tools/data-integration/splunk/index.md +++ b/docs/integrations/tools/data-integration/splunk/index.md @@ -22,7 +22,7 @@ import splunk_011 from '@site/static/images/integrations/tools/data-integration/ import splunk_012 from '@site/static/images/integrations/tools/data-integration/splunk/splunk_012.png'; import PartnerBadge from '@theme/badges/PartnerBadge'; -# Storing ClickHouse Cloud Audit logs into Splunk +# Storing ClickHouse Cloud audit logs into Splunk @@ -34,7 +34,7 @@ This add-on contains only a modular input, no additional UI are provided with th # Installation -## For Splunk Enterprise {#for-splunk-enterprise} +## For Splunk enterprise {#for-splunk-enterprise} Download the ClickHouse Cloud Audit Add-on for Splunk from [Splunkbase](https://splunkbase.splunk.com/app/7709). diff --git a/docs/intro.md b/docs/intro.md index d7d98ff9d55..1e8de7066c0 100644 --- a/docs/intro.md +++ b/docs/intro.md @@ -71,7 +71,7 @@ Because the block-wise storage and transfer from disk to memory is aligned with ClickHouse uses an asynchronous multi-master replication scheme to ensure that data is stored redundantly on multiple nodes. After being written to any available replica, all the remaining replicas retrieve their copy in the background. The system maintains identical data on different replicas. Recovery after most failures is performed automatically, or semi-automatically in complex cases. -## Role-Based Access Control {#role-based-access-control} +## Role-based access control {#role-based-access-control} ClickHouse implements user account management using SQL queries and allows for role-based access control configuration similar to what can be found in ANSI SQL standard and popular relational database management systems. diff --git a/docs/materialized-view/incremental-materialized-view.md b/docs/materialized-view/incremental-materialized-view.md index 947c4434acf..499ec0d946b 100644 --- a/docs/materialized-view/incremental-materialized-view.md +++ b/docs/materialized-view/incremental-materialized-view.md @@ -372,7 +372,7 @@ WHERE PostId IN ( Materialized views can be chained (or cascaded), allowing complex workflows to be established. For more information see the guide ["Cascading materialized views"](https://clickhouse.com/docs/guides/developer/cascading-materialized-views). -## Materialized views and JOINs {#materialized-views-and-joins} +## Materialized views and joins {#materialized-views-and-joins} :::note Refreshable Materialized Views The following applies to Incremental Materialized Views only. Refreshable Materialized Views execute their query periodically over the full target dataset and fully support JOINs. Consider using them for complex JOINs if a reduction in result freshness can be tolerated. @@ -556,7 +556,7 @@ WHERE DisplayName = 'brand_new_user' Note, however, that this result is incorrect. -### Best practices for JOINs in materialized views {#join-best-practices} +### Best practices for joins in materialized views {#join-best-practices} - **Use the left-most table as the trigger.** Only the table on the left side of the `SELECT` statement triggers the materialized view. Changes to right-side tables will not trigger updates. @@ -1095,7 +1095,7 @@ Leave it disabled when: - You require predictable, ordered execution - You're debugging or auditing insert behavior and want deterministic replay -## Materialized views and Common Table Expressions (CTE) {#materialized-views-common-table-expressions-ctes} +## Materialized views and common table expressions (cte) {#materialized-views-common-table-expressions-ctes} **Non-recursive** Common Table Expressions (CTEs) are supported in Materialized Views. diff --git a/docs/materialized-view/refreshable-materialized-view.md b/docs/materialized-view/refreshable-materialized-view.md index a5229a42707..4232dad7ea5 100644 --- a/docs/materialized-view/refreshable-materialized-view.md +++ b/docs/materialized-view/refreshable-materialized-view.md @@ -64,7 +64,7 @@ FROM system.view_refreshes; └──────────┴──────────────────┴───────────┴─────────────────────┴─────────────────────┴─────────────────────┴───────────┴──────────────┘ ``` -## How can I change the refresh rate? {#how-can-i-change-the-refresh-rate} +## How can i change the refresh rate? {#how-can-i-change-the-refresh-rate} To change the refresh rate of a refreshable materialized view, use the [`ALTER TABLE...MODIFY REFRESH`](/sql-reference/statements/alter/view#alter-table--modify-refresh-statement) syntax. @@ -183,7 +183,7 @@ FORMAT PrettyCompactMonoBlock Lets now have a look at how to use refreshable materialized views with some example datasets. -### Stack Overflow {#stack-overflow} +### Stack overflow {#stack-overflow} The [denormalizing data guide](/data-modeling/denormalization) shows various techniques for denormalizing data using a Stack Overflow dataset. We populate data into the following tables: `votes`, `users`, `badges`, `posts`, and `postlinks`. @@ -233,7 +233,7 @@ The view will execute immediately and every hour thereafter as configured to ens The syntax here is identical to an incremental materialized view, except we include a [`REFRESH`](/sql-reference/statements/create/view#refreshable-materialized-view) clause: ::: -### IMDb {#imdb} +### Imdb {#imdb} In the [dbt and ClickHouse integration guide](/integrations/dbt) we populated an IMDb dataset with the following tables: `actors`, `directors`, `genres`, `movie_directors`, `movies`, and `roles`. diff --git a/docs/native-protocol/columns.md b/docs/native-protocol/columns.md index e4e8ca22b09..794fbec9235 100644 --- a/docs/native-protocol/columns.md +++ b/docs/native-protocol/columns.md @@ -37,7 +37,7 @@ Just an array of String, i.e. (len, value). An array of N-byte sequences. -## IP {#ip} +## Ip {#ip} IPv4 is alias of `UInt32` numeric type and represented as UInt32. @@ -71,7 +71,7 @@ Rows count in `Data` is last value from `Offsets`. // Nulls: [ 1, 0, 0, 1, 0] (len: 5) ``` -## UUID {#uuid} +## Uuid {#uuid} Alias of `FixedString(16)`, UUID value represented as binary. diff --git a/docs/native-protocol/hash.md b/docs/native-protocol/hash.md index df6d2ceaf85..ac564d4cab3 100644 --- a/docs/native-protocol/hash.md +++ b/docs/native-protocol/hash.md @@ -7,7 +7,7 @@ doc_type: 'reference' keywords: ['CityHash', 'native protocol hash', 'hash function', 'Google CityHash', 'protocol hashing'] --- -# CityHash +# Cityhash ClickHouse uses **one of the previous** versions of [CityHash from Google](https://github.com/google/cityhash). diff --git a/docs/operations_/backup_restore/01_local_disk.md b/docs/operations_/backup_restore/01_local_disk.md index ca844cf9d7e..03cdc1ba7f3 100644 --- a/docs/operations_/backup_restore/01_local_disk.md +++ b/docs/operations_/backup_restore/01_local_disk.md @@ -11,7 +11,7 @@ import S3Settings from '@site/docs/operations_/backup_restore/_snippets/_s3_sett import ExampleSetup from '@site/docs/operations_/backup_restore/_snippets/_example_setup.md'; import Syntax from '@site/docs/operations_/backup_restore/_snippets/_syntax.md'; -# BACKUP / RESTORE to disk {#backup-to-a-local-disk} +# Backup / restore to disk {#backup-to-a-local-disk} ## Syntax {#syntax} diff --git a/docs/operations_/backup_restore/02_s3_endpoint.md b/docs/operations_/backup_restore/02_s3_endpoint.md index 5c07dd6f390..a6be81a41d4 100644 --- a/docs/operations_/backup_restore/02_s3_endpoint.md +++ b/docs/operations_/backup_restore/02_s3_endpoint.md @@ -8,7 +8,7 @@ doc_type: 'guide' import Syntax from '@site/docs/operations_/backup_restore/_snippets/_syntax.md'; -# BACKUP / RESTORE to or from an S3 endpoint {#backup-to-a-local-disk} +# Backup / restore to or from an S3 endpoint {#backup-to-a-local-disk} This article covers backing up or restoring backups to/from an S3 bucket via an S3 endpoint. diff --git a/docs/operations_/backup_restore/03_azure_blob_storage.md b/docs/operations_/backup_restore/03_azure_blob_storage.md index 19d0ebc410f..7e08e23838e 100644 --- a/docs/operations_/backup_restore/03_azure_blob_storage.md +++ b/docs/operations_/backup_restore/03_azure_blob_storage.md @@ -8,13 +8,13 @@ doc_type: 'guide' import Syntax from '@site/docs/operations_/backup_restore/_snippets/_syntax.md'; -# BACKUP/RESTORE to or from Azure Blob Storage {#backup-to-azure-blob-storage} +# BACKUP/RESTORE to or from Azure blob storage {#backup-to-azure-blob-storage} ## Syntax {#syntax} -## Configuring BACKUP / RESTORE to use an AzureBlobStorage endpoint {#configuring-backuprestore-to-use-an-azureblobstorage-endpoint} +## Configuring backup / restore to use an azureblobstorage endpoint {#configuring-backuprestore-to-use-an-azureblobstorage-endpoint} To write backups to an AzureBlobStorage container you need the following pieces of information: - AzureBlobStorage endpoint connection string / url, diff --git a/docs/operations_/backup_restore/04_alternative_methods.md b/docs/operations_/backup_restore/04_alternative_methods.md index 3175c4fde61..97883ab07df 100644 --- a/docs/operations_/backup_restore/04_alternative_methods.md +++ b/docs/operations_/backup_restore/04_alternative_methods.md @@ -21,7 +21,7 @@ being written to ClickHouse and store it in cold storage somewhere. Most compani already have some default recommended cold storage, which could be an object store or a distributed filesystem like [HDFS](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html). -### Filesystem Snapshots {#filesystem-snapshots} +### Filesystem snapshots {#filesystem-snapshots} Some local filesystems provide snapshot functionality (for example, [ZFS](https://en.wikipedia.org/wiki/ZFS)), but they might not be the best choice for serving live queries. A possible solution @@ -34,7 +34,7 @@ disks attached per server, which would be cost-effective. For smaller volumes of data, a simple `INSERT INTO ... SELECT ...` to remote tables might work as well. -### Manipulations with Parts {#manipulations-with-parts} +### Manipulations with parts {#manipulations-with-parts} ClickHouse allows using the `ALTER TABLE ... FREEZE PARTITION ...` query to create a local copy of table partitions. This is implemented using hardlinks to the `/var/lib/clickhouse/shadow/` diff --git a/docs/tips-and-tricks/cost-optimization.md b/docs/tips-and-tricks/cost-optimization.md index c19e2c347bc..2bb190896cf 100644 --- a/docs/tips-and-tricks/cost-optimization.md +++ b/docs/tips-and-tricks/cost-optimization.md @@ -26,7 +26,7 @@ description: 'Cost optimization strategies from ClickHouse community meetups wit *Learn about how [ClickHouse Cloud can help manage operational costs](/cloud/overview)*. -## Compression strategy: LZ4 vs ZSTD in production {#compression-strategy} +## Compression strategy: lz4 vs zstd in production {#compression-strategy} When Microsoft Clarity needed to handle hundreds of terabytes of data, they discovered that compression choices have dramatic cost implications. At their scale, every bit of storage savings matters, and they faced a classic trade-off: performance versus storage costs. Microsoft Clarity handles massive volumes—two petabytes of uncompressed data per month across all accounts, processing around 60,000 queries per hour across eight nodes and serving billions of page views from millions of websites. At this scale, compression strategy becomes a critical cost factor. diff --git a/docs/tips-and-tricks/too-many-parts.md b/docs/tips-and-tricks/too-many-parts.md index 10f4c610f2f..404420f757a 100644 --- a/docs/tips-and-tricks/too-many-parts.md +++ b/docs/tips-and-tricks/too-many-parts.md @@ -70,7 +70,7 @@ ORDER BY total_parts DESC LIMIT 20; ``` -## Video Sources {#video-sources} +## Video sources {#video-sources} - [Fast, Concurrent, and Consistent Asynchronous INSERTS in ClickHouse](https://www.youtube.com/watch?v=AsMPEfN5QtM) - ClickHouse team member explains async inserts and the too many parts problem - [Production ClickHouse at Scale](https://www.youtube.com/watch?v=liTgGiTuhJE) - Real-world batching strategies from observability platforms \ No newline at end of file diff --git a/docs/tools-and-utilities/static-files-disk-uploader.md b/docs/tools-and-utilities/static-files-disk-uploader.md index b842be2ff83..05fea9ac512 100644 --- a/docs/tools-and-utilities/static-files-disk-uploader.md +++ b/docs/tools-and-utilities/static-files-disk-uploader.md @@ -6,7 +6,7 @@ description: 'Provides a description of the clickhouse-static-files-disk-uploade doc_type: 'guide' --- -# clickhouse-static-files-disk-uploader +# Clickhouse-static-files-disk-uploader Outputs a data directory containing metadata for a specified ClickHouse table. This metadata can be used to create a ClickHouse table on a different server containing a read-only dataset backed by a `web` disk. diff --git a/docs/troubleshooting/index.md b/docs/troubleshooting/index.md index 503fad2081c..ab34c236292 100644 --- a/docs/troubleshooting/index.md +++ b/docs/troubleshooting/index.md @@ -83,14 +83,14 @@ High memory usage, out-of-memory crashes, or need help sizing your ClickHouse de
-## Connections and Authentication {#connections-and-authentication} +## Connections and authentication {#connections-and-authentication} Can't connect to ClickHouse, authentication failures, SSL certificate errors, or client setup issues.
Show connection solutions -### Basic Connection issues {#basic-connection-issues} +### Basic connection issues {#basic-connection-issues} - [Fix HTTP interface issues](/docs/interfaces/http) - [Handle SSL certificate problems](/docs/knowledgebase/certificate_verify_failed_error) - [User authentication setup](/docs/operations/access-rights) diff --git a/docs/tutorial.md b/docs/tutorial.md index e6ca2551e3a..957d66e6c3f 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -9,7 +9,7 @@ show_related_blogs: true doc_type: 'guide' --- -# Advanced Tutorial +# Advanced tutorial ## Overview {#overview} diff --git a/docs/use-cases/AI_ML/AIChat/index.md b/docs/use-cases/AI_ML/AIChat/index.md index adfd2814c77..6c5f7d5c753 100644 --- a/docs/use-cases/AI_ML/AIChat/index.md +++ b/docs/use-cases/AI_ML/AIChat/index.md @@ -21,7 +21,7 @@ import img_history from '@site/static/images/use-cases/AI_ML/AIChat/5_history.pn import img_result_actions from '@site/static/images/use-cases/AI_ML/AIChat/6_result_actions.png'; import img_new_tab from '@site/static/images/use-cases/AI_ML/AIChat/7_open_in_editor.png'; -# Using AI Chat in ClickHouse Cloud +# Using ai chat in ClickHouse Cloud > This guide explains how to enable and use the AI Chat feature in the ClickHouse Cloud Console. @@ -31,7 +31,7 @@ import img_new_tab from '@site/static/images/use-cases/AI_ML/AIChat/7_open_in_ed 1. You must have access to a ClickHouse Cloud organization with AI features enabled (contact your org admin or support if unavailable). -## Open the AI Chat panel {#open-panel} +## Open the ai chat panel {#open-panel} 1. Navigate to a ClickHouse Cloud service. 2. In the left sidebar, click the sparkle icon labeled “Ask AI”. @@ -63,7 +63,7 @@ Use the mode selector at the bottom-left of the flyout to switch. 2. Press Enter to send (use Shift + Enter for a newline). 3. While the model is processing you can click “Stop” to interrupt. -## Understanding “Agent” thinking steps {#thinking-steps} +## Understanding “agent” thinking steps {#thinking-steps} In Agent mode you may see expandable intermediate “thinking” or planning steps. These provide transparency into how the assistant forms its answer. Collapse or expand as needed. diff --git a/docs/use-cases/AI_ML/MCP/01_remote_mcp.md b/docs/use-cases/AI_ML/MCP/01_remote_mcp.md index e7a79e5046c..5fc18ce3e05 100644 --- a/docs/use-cases/AI_ML/MCP/01_remote_mcp.md +++ b/docs/use-cases/AI_ML/MCP/01_remote_mcp.md @@ -22,13 +22,13 @@ import img5 from '@site/static/images/use-cases/AI_ML/MCP/5connected_mcp_claude. import img6 from '@site/static/images/use-cases/AI_ML/MCP/6slash_mcp_claude.png'; import img7 from '@site/static/images/use-cases/AI_ML/MCP/7usage_mcp.png'; -# Enabling the ClickHouse Cloud Remote MCP Server +# Enabling the ClickHouse Cloud remote mcp server > This guide explains how to enable and use the ClickHouse Cloud Remote MCP Server. We will use Claude Code as an MCP Client for this example, but any LLM client that supports MCP can be used. -## Enable the remote MCP server for your ClickHouse Cloud service {#enable-remote-mcp-server} +## Enable the remote mcp server for your ClickHouse Cloud service {#enable-remote-mcp-server} 1. Connect to your ClickHouse Cloud Service, click on the `Connect` button, and enable the Remote MCP Server for your Service @@ -42,7 +42,7 @@ import img7 from '@site/static/images/use-cases/AI_ML/MCP/7usage_mcp.png'; https://mcp.clickhouse.cloud/mcp ``` -## Add the ClickHouse MCP Server in Claude Code {#add-clickhouse-mcp-server-claude-code} +## Add the ClickHouse mcp server in claude code {#add-clickhouse-mcp-server-claude-code} 1. In your working directory, run the following command to add the ClickHouse Cloud MCP Server configuration to Claude Code. In this example, we named the MCP server in the Claude Code config `clickhouse_cloud` @@ -78,7 +78,7 @@ claude mcp add --transport http clickhouse_cloud https://mcp.clickhouse.cloud/mc OAuth Connect flow success -## Use the ClickHouse Cloud Remote MCP Server from Claude Code {#use-rempte-mcp-from-claude-code} +## Use the ClickHouse Cloud remote mcp server from claude code {#use-rempte-mcp-from-claude-code} 1. Verify in Claude Code that the remote MCP server is connected diff --git a/docs/use-cases/AI_ML/MCP/02_claude-desktop.md b/docs/use-cases/AI_ML/MCP/02_claude-desktop.md index 664900e0b36..664f40e9f41 100644 --- a/docs/use-cases/AI_ML/MCP/02_claude-desktop.md +++ b/docs/use-cases/AI_ML/MCP/02_claude-desktop.md @@ -18,7 +18,7 @@ import FindMCPServers from '@site/static/images/use-cases/AI_ML/MCP/find-mcp-ser import MCPPermission from '@site/static/images/use-cases/AI_ML/MCP/mcp-permission.png'; import ClaudeConversation from '@site/static/images/use-cases/AI_ML/MCP/claude-conversation.png'; -# Using ClickHouse MCP server with Claude Desktop +# Using ClickHouse mcp server with claude desktop > This guide explains how to set up Claude Desktop with a ClickHouse MCP server using uv > and connect it to the ClickHouse example datasets. @@ -32,11 +32,11 @@ import ClaudeConversation from '@site/static/images/use-cases/AI_ML/MCP/claude-c You will need to install [uv](https://docs.astral.sh/uv/) to follow the instructions in this guide. If you don't want to use uv, you will need to update the MCP Server config to use an alternative package manager. -## Download Claude Desktop {#download-claude-desktop} +## Download claude desktop {#download-claude-desktop} You'll also need to install the Claude Desktop app, which you can download from the [Claude Desktop website](https://claude.ai/desktop). -## Configuring ClickHouse MCP server {#configure-clickhouse-mcp-server} +## Configuring ClickHouse mcp server {#configure-clickhouse-mcp-server} Once you've got Claude Desktop installed, it's time to configure the [ClickHouse MCP server](https://github.com/ClickHouse/mcp-clickhouse). We can do this via the [Claude Desktop configuration file](https://claude.ai/docs/configuration). @@ -98,7 +98,7 @@ MCP mcp-clickhouse: spawn uv ENOENT If that happens, you'll need to update the `command` to have the full path to `uv`. e.g. if you've installed via Cargo, it will be `/Users//.cargo/bin/uv` ::: -## Using ClickHouse MCP server {#using-clickhouse-mcp-server} +## Using ClickHouse mcp server {#using-clickhouse-mcp-server} Once you've restarted Claude Desktop, you can find the ClickHouse MCP server by clicking on the `Search and tools` icon: diff --git a/docs/use-cases/AI_ML/MCP/03_librechat.md b/docs/use-cases/AI_ML/MCP/03_librechat.md index 90ec522f813..b2050d0ef1c 100644 --- a/docs/use-cases/AI_ML/MCP/03_librechat.md +++ b/docs/use-cases/AI_ML/MCP/03_librechat.md @@ -15,14 +15,14 @@ import Link from '@docusaurus/Link'; import Image from '@theme/IdealImage'; import LibreInterface from '@site/static/images/use-cases/AI_ML/MCP/librechat.png'; -# Using ClickHouse MCP server with LibreChat +# Using ClickHouse mcp server with librechat > This guide explains how to set up LibreChat with a ClickHouse MCP server using Docker > and connect it to the ClickHouse example datasets. -## Install docker {#install-docker} +## Install Docker {#install-docker} You will need Docker to run LibreChat and the MCP server. To get Docker: 1. Visit [docker.com](https://www.docker.com/products/docker-desktop) @@ -32,7 +32,7 @@ You will need Docker to run LibreChat and the MCP server. To get Docker:
For more information, see the [Docker documentation](https://docs.docker.com/get-docker/). -## Clone the LibreChat repository {#clone-librechat-repo} +## Clone the librechat repository {#clone-librechat-repo} Open a terminal (command prompt, terminal or PowerShell) and clone the LibreChat repository using the following command: @@ -82,7 +82,7 @@ cp librechat.example.yaml librechat.yaml This creates the main [configuration file](https://www.librechat.ai/docs/configuration/librechat_yaml) for LibreChat. -## Add ClickHouse MCP server to Docker compose {#add-clickhouse-mcp-server-to-docker-compose} +## Add ClickHouse mcp server to Docker compose {#add-clickhouse-mcp-server-to-docker-compose} Next we'll add the ClickHouse MCP server to the LibreChat Docker compose file so that the LLM can interact with the @@ -134,7 +134,7 @@ isSelected={true} /> -## Configure MCP server in librechat.yaml {#configure-mcp-server-in-librechat-yaml} +## Configure mcp server in librechat.yaml {#configure-mcp-server-in-librechat-yaml} Open `librechat.yaml` and place the following configuration at the end of the file: @@ -159,9 +159,9 @@ For simplicity, we will remove the need to authenticate for now: socialLogins: [] ``` -## Add a local LLM using Ollama (optional) {#add-local-llm-using-ollama} +## Add a local llm using ollama (optional) {#add-local-llm-using-ollama} -### Install Ollama {#install-ollama} +### Install ollama {#install-ollama} Go to the [Ollama website](https://ollama.com/download) and install Ollama for your system. @@ -175,7 +175,7 @@ This will pull the model to your local machine if it is not present. For a list of models see the [Ollama library](https://ollama.com/library) -### Configure Ollama in librechat.yaml {#configure-ollama-in-librechat-yaml} +### Configure ollama in librechat.yaml {#configure-ollama-in-librechat-yaml} Once the model has downloaded, configure it in `librechat.yaml`: @@ -208,7 +208,7 @@ docker compose up Wait until all services are fully running. -## Open LibreChat in your browser {#open-librechat-in-browser} +## Open librechat in your browser {#open-librechat-in-browser} Once all services are up and running, open your browser and go to `http://localhost:3080/` diff --git a/docs/use-cases/AI_ML/MCP/04_anythingllm.md b/docs/use-cases/AI_ML/MCP/04_anythingllm.md index bdc8894fd5a..bbbefc71a1f 100644 --- a/docs/use-cases/AI_ML/MCP/04_anythingllm.md +++ b/docs/use-cases/AI_ML/MCP/04_anythingllm.md @@ -18,7 +18,7 @@ import Conversation from '@site/static/images/use-cases/AI_ML/MCP/allm_conversat import MCPServers from '@site/static/images/use-cases/AI_ML/MCP/allm_mcp-servers.png'; import ToolIcon from '@site/static/images/use-cases/AI_ML/MCP/alm_tool-icon.png'; -# Using ClickHouse MCP server with AnythingLLM +# Using ClickHouse mcp server with anythingllm > This guide explains how to set up [AnythingLLM](https://anythingllm.com/) with a ClickHouse MCP server using Docker > and connect it to the ClickHouse example datasets. @@ -35,7 +35,7 @@ You will need Docker to run LibreChat and the MCP server. To get Docker:
For more information, see the [Docker documentation](https://docs.docker.com/get-docker/). -## Pull AnythingLLM Docker image {#pull-anythingllm-docker-image} +## Pull anythingllm Docker image {#pull-anythingllm-docker-image} Run the following command to pull the AnythingLLM Docker image to your machine: @@ -53,7 +53,7 @@ mkdir -p $STORAGE_LOCATION && \ touch "$STORAGE_LOCATION/.env" ``` -## Configure MCP Server config file {#configure-mcp-server-config-file} +## Configure mcp server config file {#configure-mcp-server-config-file} Create the `plugins` directory: @@ -90,7 +90,7 @@ If you want to explore your own data, you can do so by using the [host, username and password](https://clickhouse.com/docs/getting-started/quick-start/cloud#connect-with-your-app) of your own ClickHouse Cloud service. -## Start the AnythingLLM Docker container {#start-anythingllm-docker-container} +## Start the anythingllm Docker container {#start-anythingllm-docker-container} Run the following command to start the AnythingLLM Docker container: @@ -106,7 +106,7 @@ mintplexlabs/anythingllm Once that's started, navigate to `http://localhost:3001` in your browser. Select the model that you want to use and provide your API key. -## Wait for MCP Servers to start up {#wait-for-mcp-servers-to-start-up} +## Wait for mcp servers to start up {#wait-for-mcp-servers-to-start-up} Click on the tool icon in the bottom left-hand side of the UI: @@ -117,7 +117,7 @@ Wait until you see `Mcp ClickHouse` set to `On` MCP servers ready -## Chat with ClickHouse MCP Server with AnythingLLM {#chat-with-clickhouse-mcp-server-with-anythingllm} +## Chat with ClickHouse mcp server with anythingllm {#chat-with-clickhouse-mcp-server-with-anythingllm} We're now ready to start a chat. To make MCP Servers available to a chat, you'll need to prefix the first message in the conversation with `@agent`. diff --git a/docs/use-cases/AI_ML/MCP/05_open-webui.md b/docs/use-cases/AI_ML/MCP/05_open-webui.md index f11dbe535c5..7bc4a691d67 100644 --- a/docs/use-cases/AI_ML/MCP/05_open-webui.md +++ b/docs/use-cases/AI_ML/MCP/05_open-webui.md @@ -25,7 +25,7 @@ import AddConnection from '@site/static/images/use-cases/AI_ML/MCP/7_add_connect import OpenAIModels from '@site/static/images/use-cases/AI_ML/MCP/8_openai_models_more.png'; import Conversation from '@site/static/images/use-cases/AI_ML/MCP/9_conversation.png'; -# Using ClickHouse MCP server with Open WebUI +# Using ClickHouse mcp server with open webui > This guide explains how to set up [Open WebUI](https://github.com/open-webui/open-webui) with a ClickHouse MCP server > and connect it to the ClickHouse example datasets. @@ -37,7 +37,7 @@ import Conversation from '@site/static/images/use-cases/AI_ML/MCP/9_conversation You will need to install [uv](https://docs.astral.sh/uv/) to follow the instructions in this guide. If you don't want to use uv, you will need to update the MCP Server config to use an alternative package manager. -## Launch Open WebUI {#launch-open-webui} +## Launch open webui {#launch-open-webui} To launch Open WebUI, you can run the following command: @@ -47,7 +47,7 @@ uv run --with open-webui open-webui serve Navigate to http://localhost:8080/ to see the UI. -## Configure ClickHouse MCP Server {#configure-clickhouse-mcp-server} +## Configure ClickHouse mcp server {#configure-clickhouse-mcp-server} To setup the ClickHouse MCP Server, we'll need to convert the MCP Server to Open API endpoints. Let's first set environmental variables that will let us connect to the ClickHouse SQL Playground: @@ -88,7 +88,7 @@ If we click on the tool icon, we can then list the available tools: Open WebUI tool listing -## Configure OpenAI {#configure-openai} +## Configure openai {#configure-openai} By default, Open WebUI works with Ollama models, but we can add OpenAI compatible endpoints as well. These are configured via the settings menu, but this time we need to click on the `Connections` tab: @@ -103,7 +103,7 @@ The OpenAI models will then be available on the top menu: Open WebUI - Models -## Chat to ClickHouse MCP Server with Open WebUI {#chat-to-clickhouse-mcp-server} +## Chat to ClickHouse mcp server with open webui {#chat-to-clickhouse-mcp-server} We can then have a conversation and Open WebUI will call the MCP Server if necessary: diff --git a/docs/use-cases/AI_ML/MCP/06_ollama.md b/docs/use-cases/AI_ML/MCP/06_ollama.md index 8b286158b11..68f09d0bca5 100644 --- a/docs/use-cases/AI_ML/MCP/06_ollama.md +++ b/docs/use-cases/AI_ML/MCP/06_ollama.md @@ -14,13 +14,13 @@ import {CardHorizontal} from '@clickhouse/click-ui/bundled' import Link from '@docusaurus/Link'; import Image from '@theme/IdealImage'; -# Using ClickHouse MCP server with Ollama +# Using ClickHouse mcp server with ollama > This guide explains how to use the ClickHouse MCP Server with Ollama. -## Install Ollama {#install-ollama} +## Install ollama {#install-ollama} Ollama is a library for running Large Language Models (LLMs) on your own machine. It has a [wide range of models available](https://ollama.com/library) and is easy to use. @@ -91,7 +91,7 @@ ollama show qwen3 We can see from this output that the default qwen3 model has just over 8 billion parameters. -## Install MCPHost {#install-mcphost} +## Install mcphost {#install-mcphost} At the time of writing (July 2025) there is no native functionality for using Ollama with MCP Servers. However, we can use [MCPHost](https://github.com/mark3labs/mcphost) to run Ollama models with MCP Servers. @@ -105,7 +105,7 @@ go install github.com/mark3labs/mcphost@latest The binary will be installed under `~/go/bin` so we need to make sure that directory is on our path. -## Configuring ClickHouse MCP Server {#configure-clickhouse-mcp-server} +## Configuring ClickHouse mcp server {#configure-clickhouse-mcp-server} We can configure MCP Servers with MCPHost in YAML or JSON files. MCPHost will look for config files in your home directory the following order: @@ -153,7 +153,7 @@ export CLICKHOUSE_PASSWORD="" In theory, you should be able to provide these variables under the `environment` key in the MCP configuration file, but we've found that this doesn't work. ::: -## Running MCPHost {#running-mcphost} +## Running mcphost {#running-mcphost} Once you've configured the ClickHouse MCP server, you can run MCPHost by running the following command: diff --git a/docs/use-cases/AI_ML/MCP/07_janai.md b/docs/use-cases/AI_ML/MCP/07_janai.md index 5c6e2685fde..107dde8a9bb 100644 --- a/docs/use-cases/AI_ML/MCP/07_janai.md +++ b/docs/use-cases/AI_ML/MCP/07_janai.md @@ -26,7 +26,7 @@ import ToolsCalled from '@site/static/images/use-cases/AI_ML/MCP/8_janai_tools_c import ToolsCalledExpanded from '@site/static/images/use-cases/AI_ML/MCP/9_janai_tools_called_expanded.png'; import Result from '@site/static/images/use-cases/AI_ML/MCP/10_janai_result.png'; -# Using ClickHouse MCP server with Jan.ai +# Using ClickHouse mcp server with Jan.ai > This guide explains how to use the ClickHouse MCP Server with [Jan.ai](https://jan.ai/docs). @@ -39,7 +39,7 @@ You can download Jan.ai for [Mac](https://jan.ai/docs/desktop/mac), [Windows](ht It's a native app, so once it's downloaded, you can launch it. -## Add LLM to Jan.ai {#add-llm-to-janai} +## Add llm to Jan.ai {#add-llm-to-janai} We can enabled models via the settings menu. @@ -47,7 +47,7 @@ To enable OpenAI, we need to provide an API key, as shown below: Enable OpenAI models -## Enable MCP Servers {#enable-mcp-servers} +## Enable mcp servers {#enable-mcp-servers} At the time of writing, MCP Servers are an experimental feature in Jan.ai. We can enable them by toggling experimental features: @@ -56,7 +56,7 @@ We can enable them by toggling experimental features: Once that toggle is pressed, we'll see `MCP Servers` on the left menu. -## Configure ClickHouse MCP Server {#configure-clickhouse-mcp-server} +## Configure ClickHouse mcp server {#configure-clickhouse-mcp-server} If we click on the `MCP Servers` menu, we'll see a list of MCP servers that we can connect to: @@ -76,7 +76,7 @@ The ClickHouse MCP Server's tools will now be visible on the chat dialog: ClickHouse MCP Server tools -## Chat to ClickHouse MCP Server with Jan.ai {#chat-to-clickhouse-mcp-server} +## Chat to ClickHouse mcp server with Jan.ai {#chat-to-clickhouse-mcp-server} It's time to have a conversation about some data stored in ClickHouse! Let's ask a question: diff --git a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/agno.md b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/agno.md index b2e445c593f..ae8d3173434 100644 --- a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/agno.md +++ b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/agno.md @@ -10,7 +10,7 @@ show_related_blogs: true doc_type: 'guide' --- -# How to build an AI Agent with Agno and the ClickHouse MCP Server +# How to build an ai agent with agno and the ClickHouse mcp server In this guide you'll learn how to build an [Agno](https://github.com/agno-agi/agno) AI agent that can interact with [ClickHouse's SQL playground](https://sql.clickhouse.com/) using [ClickHouse's MCP Server](https://github.com/ClickHouse/mcp-clickhouse). @@ -68,7 +68,7 @@ env = { } ``` -## Initialize MCP Server and Agno agent {#initialize-mcp-and-agent} +## Initialize mcp server and agno agent {#initialize-mcp-and-agent} Now configure the ClickHouse MCP Server to point at the ClickHouse SQL playground and also initialize our Agno agent and ask it a question: diff --git a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/chainlit.md b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/chainlit.md index 63ae9e20930..6faa77272a0 100644 --- a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/chainlit.md +++ b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/chainlit.md @@ -10,7 +10,7 @@ show_related_blogs: true doc_type: 'guide' --- -# How to build an AI agent with Chainlit and the ClickHouse MCP Server +# How to build an ai agent with chainlit and the ClickHouse mcp server This guide explores how to combine Chainlit's powerful chat interface framework with the ClickHouse Model Context Protocol (MCP) Server to create interactive data @@ -22,7 +22,7 @@ integration with ClickHouse's high-performance columnar database. - You'll need an Anthropic API key - You'll need to have [`uv`](https://docs.astral.sh/uv/getting-started/installation/) installed -## Basic Chainlit app {#basic-chainlit-app} +## Basic chainlit app {#basic-chainlit-app} You can see an example of a basic chat app by running the following: @@ -32,7 +32,7 @@ uv run --with anthropic --with chainlit chainlit run chat_basic.py -w -h Then navigate to `http://localhost:8000` -## Adding ClickHouse MCP Server {#adding-clickhouse-mcp-server} +## Adding ClickHouse mcp server {#adding-clickhouse-mcp-server} Things get more interesting if we add the ClickHouse MCP Server. You'll need to update your `.chainlit/config.toml` file to allow the `uv` command diff --git a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/claude-agent-sdk.md b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/claude-agent-sdk.md index adbd640ea3a..2f56b404d8a 100644 --- a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/claude-agent-sdk.md +++ b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/claude-agent-sdk.md @@ -10,7 +10,7 @@ show_related_blogs: true doc_type: 'guide' --- -# How to build an AI Agent with Claude Agent SDK and the ClickHouse MCP Server +# How to build an ai agent with claude agent SDK and the ClickHouse mcp server In this guide you'll learn how to build a [Claude Agent SDK](https://docs.claude.com/en/api/agent-sdk/overview) AI agent that can interact with [ClickHouse's SQL playground](https://sql.clickhouse.com/) using [ClickHouse's MCP Server](https://github.com/ClickHouse/mcp-clickhouse). @@ -63,7 +63,7 @@ env = { } ``` -## Initialize MCP Server and Claude Agent SDK agent {#initialize-mcp-and-agent} +## Initialize mcp server and claude agent SDK agent {#initialize-mcp-and-agent} Now configure the ClickHouse MCP Server to point at the ClickHouse SQL playground and also initialize our agent and ask it a question: diff --git a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/copilotkit.md b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/copilotkit.md index bca912a4f28..ed1c6b6e3b1 100644 --- a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/copilotkit.md +++ b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/copilotkit.md @@ -10,7 +10,7 @@ show_related_blogs: true doc_type: 'guide' --- -# How to build an AI agent with CopilotKit and the ClickHouse MCP Server +# How to build an ai agent with copilotkit and the ClickHouse mcp server This is an example of how to build an agentic application using data stored in ClickHouse. It uses the [ClickHouse MCP Server](https://github.com/ClickHouse/mcp-clickhouse) @@ -63,7 +63,7 @@ uv add fastmcp Copy the `env.example` file to `.env` and edit it to provide your `ANTHROPIC_API_KEY`. -## Use your own LLM {#use-your-own-llm} +## Use your own llm {#use-your-own-llm} If you'd rather use another LLM provider than Anthropic, you can modify the Copilotkit runtime to use a different LLM adapter. diff --git a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/crewai.md b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/crewai.md index ed25de45ce5..acbd45480bb 100644 --- a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/crewai.md +++ b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/crewai.md @@ -10,7 +10,7 @@ show_related_blogs: true doc_type: 'guide' --- -# How to build an AI Agent with CrewAI and the ClickHouse MCP Server +# How to build an ai agent with crewai and the ClickHouse mcp server In this guide you'll learn how to build a [CrewAI](https://docs.crewai.com/) AI agent that can interact with [ClickHouse's SQL playground](https://sql.clickhouse.com/) using [ClickHouse's MCP Server](https://github.com/ClickHouse/mcp-clickhouse). @@ -63,7 +63,7 @@ env = { } ``` -## Initialize MCP Server and CrewAI agent {#initialize-mcp-and-agent} +## Initialize mcp server and crewai agent {#initialize-mcp-and-agent} Now configure the ClickHouse MCP Server to point at the ClickHouse SQL playground and also initialize our agent and ask it a question: diff --git a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/dspy.md b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/dspy.md index 53cc8adea54..6be2a46816b 100644 --- a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/dspy.md +++ b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/dspy.md @@ -10,7 +10,7 @@ show_related_blogs: true doc_type: 'guide' --- -# How to build an AI Agent with DSPy and the ClickHouse MCP Server +# How to build an ai agent with dspy and the ClickHouse mcp server In this guide, you'll learn how to build an AI agent with [DSPy](https://github.com/langchain-ai/langgraph) that can interact with [ClickHouse's SQL playground](https://sql.clickhouse.com/) using [ClickHouse's MCP Server](https://github.com/ClickHouse/mcp-clickhouse). @@ -65,7 +65,7 @@ env = { } ``` -## Initialize MCP Server {#initialize-mcp} +## Initialize mcp server {#initialize-mcp} Now configure the ClickHouse MCP Server to point at the ClickHouse SQL playground. @@ -86,7 +86,7 @@ server_params = StdioServerParameters( ) ``` -## Initialize LLM {#initialize-llm} +## Initialize llm {#initialize-llm} Next, initialize the LLM with the following line: diff --git a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/index.md b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/index.md index cfe4e0f3b3f..02fbb400f25 100644 --- a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/index.md +++ b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/index.md @@ -8,7 +8,7 @@ keywords: ['ClickHouse', 'Agno', 'Chainlit', 'MCP', 'DSPy', 'LangChain', 'LlamaI doc_type: 'guide' --- -# Guides for integrating AI agent libraries with ClickHouse MCP Server +# Guides for integrating ai agent libraries with ClickHouse mcp server | Page | Description | diff --git a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/langchain.md b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/langchain.md index 37735924b02..868bd52cf63 100644 --- a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/langchain.md +++ b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/langchain.md @@ -10,7 +10,7 @@ show_related_blogs: true doc_type: 'guide' --- -# How to build a LangChain/LangGraph AI agent using ClickHouse MCP Server +# How to build a LangChain/LangGraph ai agent using ClickHouse mcp server In this guide, you'll learn how to build a [LangChain/LangGraph](https://github.com/langchain-ai/langgraph) AI agent that can interact with [ClickHouse's SQL playground](https://sql.clickhouse.com/) using [ClickHouse's MCP Server](https://github.com/ClickHouse/mcp-clickhouse). @@ -55,7 +55,7 @@ If you don't have an Anthropic API key, and want to use another LLM provider, you can find the instructions for setting up your credentials in the [Langchain Providers docs](https://python.langchain.com/docs/integrations/providers/) ::: -## Initialize MCP Server {#initialize-mcp-and-agent} +## Initialize mcp server {#initialize-mcp-and-agent} Now configure the ClickHouse MCP Server to point at the ClickHouse SQL playground: diff --git a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/llamaindex.md b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/llamaindex.md index 9e10ec18d1d..9127f4aafe9 100644 --- a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/llamaindex.md +++ b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/llamaindex.md @@ -10,7 +10,7 @@ show_related_blogs: true doc_type: 'guide' --- -# How to build a LlamaIndex AI agent using ClickHouse MCP Server +# How to build a llamaindex ai agent using ClickHouse mcp server In this guide, you'll learn how to build a [LlamaIndex](https://docs.llamaindex.ai) AI agent that can interact with [ClickHouse's SQL playground](https://sql.clickhouse.com/) using [ClickHouse's MCP Server](https://github.com/ClickHouse/mcp-clickhouse). @@ -55,7 +55,7 @@ If you don't have an Anthropic API key, and want to use another LLM provider, you can find the instructions for setting up your credentials in the [LlamaIndex "LLMs" docs](https://docs.llamaindex.ai/en/stable/examples/) ::: -## Initialize MCP Server {#initialize-mcp-and-agent} +## Initialize mcp server {#initialize-mcp-and-agent} Now configure the ClickHouse MCP Server to point at the ClickHouse SQL playground. You'll need to convert those from Python functions into Llama Index tools: @@ -101,7 +101,7 @@ agent_worker = FunctionCallingAgentWorker.from_tools( agent = AgentRunner(agent_worker) ``` -## Initialize the LLM {#initialize-llm} +## Initialize the llm {#initialize-llm} Initialize the Claude Sonnet 4.0 model with the following code: diff --git a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/mcp-agent.md b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/mcp-agent.md index 71535f51069..a52668338ef 100644 --- a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/mcp-agent.md +++ b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/mcp-agent.md @@ -10,7 +10,7 @@ show_related_blogs: true doc_type: 'guide' --- -# How to build an AI Agent with CrewAI and the ClickHouse MCP Server +# How to build an ai agent with crewai and the ClickHouse mcp server In this guide you'll learn how to build a [mcp-agent](https://github.com/lastmile-ai/mcp-agent) AI agent that can interact with [ClickHouse's SQL playground](https://sql.clickhouse.com/) using [ClickHouse's MCP Server](https://github.com/ClickHouse/mcp-clickhouse). @@ -63,7 +63,7 @@ env = { } ``` -## Initialize MCP Server and mcp-agent agent {#initialize-mcp-and-agent} +## Initialize mcp server and mcp-agent agent {#initialize-mcp-and-agent} Now configure the ClickHouse MCP Server to point at the ClickHouse SQL playground and also initialize our agent and ask it a question: diff --git a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/microsoft-agent-framework.md b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/microsoft-agent-framework.md index 74154121447..8b4d21d28a5 100644 --- a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/microsoft-agent-framework.md +++ b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/microsoft-agent-framework.md @@ -10,7 +10,7 @@ show_related_blogs: true doc_type: 'guide' --- -# How to build an AI Agent with Microsoft Agent Framework and the ClickHouse MCP Server +# How to build an ai agent with Microsoft agent framework and the ClickHouse mcp server In this guide you'll learn how to build a [Microsoft Agent Framework](https://github.com/microsoft/agent-framework) AI agent that can interact with [ClickHouse's SQL playground](https://sql.clickhouse.com/) using [ClickHouse's MCP Server](https://github.com/ClickHouse/mcp-clickhouse). @@ -63,7 +63,7 @@ env = { } ``` -## Initialize MCP Server and Microsoft Agent Framework agent {#initialize-mcp-and-agent} +## Initialize mcp server and Microsoft agent framework agent {#initialize-mcp-and-agent} Now configure the ClickHouse MCP Server to point at the ClickHouse SQL playground and also initialize our agent and ask it a question: diff --git a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/openai-agents.md b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/openai-agents.md index c109916042a..b425b378e61 100644 --- a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/openai-agents.md +++ b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/openai-agents.md @@ -10,7 +10,7 @@ show_related_blogs: true doc_type: 'guide' --- -# How to build an OpenAI agent using ClickHouse MCP Server +# How to build an openai agent using ClickHouse mcp server In this guide, you'll learn how to build an [OpenAI](https://github.com/openai/openai-agents-python) agent that can interact with [ClickHouse's SQL playground](https://sql.clickhouse.com/) using [ClickHouse's MCP Server](https://github.com/ClickHouse/mcp-clickhouse). @@ -50,7 +50,7 @@ os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter OpenAI API Key:") Enter OpenAI API Key: ········ ``` -## Initialize MCP Server and OpenAI agent {#initialize-mcp-and-agent} +## Initialize mcp server and openai agent {#initialize-mcp-and-agent} Now configure the ClickHouse MCP Server to point at the ClickHouse SQL playground, initialize your OpenAI agent and ask it a question: diff --git a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/pydantic-ai.md b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/pydantic-ai.md index aa3365f6726..5fe9f606d6c 100644 --- a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/pydantic-ai.md +++ b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/pydantic-ai.md @@ -10,7 +10,7 @@ show_related_blogs: true doc_type: 'guide' --- -# How to build a PydanticAI agent using ClickHouse MCP Server +# How to build a pydanticai agent using ClickHouse mcp server In this guide, you'll learn how to build a [PydanticAI](https://ai.pydantic.dev/mcp/client/#__tabbed_1_1) agent that can interact with [ClickHouse's SQL playground](https://sql.clickhouse.com/) using [ClickHouse's MCP Server](https://github.com/ClickHouse/mcp-clickhouse). @@ -68,7 +68,7 @@ env = { } ``` -## Initialize MCP Server and PydanticAI agent {#initialize-mcp} +## Initialize mcp server and pydanticai agent {#initialize-mcp} Now configure the ClickHouse MCP Server to point at the ClickHouse SQL playground: diff --git a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/slackbot.md b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/slackbot.md index 5144129aba5..627a3831dfe 100644 --- a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/slackbot.md +++ b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/slackbot.md @@ -10,7 +10,7 @@ show_related_blogs: true doc_type: 'guide' --- -# How to build a SlackBot agent using ClickHouse MCP Server +# How to build a slackbot agent using ClickHouse mcp server In this guide, you'll learn how to build a [SlackBot](https://slack.com/intl/en-gb/help/articles/202026038-An-introduction-to-Slackbot) agent. This bot allows you to ask questions about your ClickHouse data directly from Slack, using natural language. It uses the @@ -27,7 +27,7 @@ The code for this example can be found in the [examples repository](https://gith -## Create a Slack App {#create-a-slack-app} +## Create a slack app {#create-a-slack-app} 1. Go to [slack.com/apps](https://slack.com/apps) and click `Create New App`. 2. Choose option `From scratch` and give your app a name. @@ -39,7 +39,7 @@ Next, you'll need to add the app created in the previous step to your workspace. You can follow the instructions for ["Add apps to your Slack workspace"](https://slack.com/intl/en-gb/help/articles/202035138-Add-apps-to-your-Slack-workspace) in the Slack documentation. -## Configure Slack app settings {#configure-slack-app-settings} +## Configure slack app settings {#configure-slack-app-settings} - Go to `App Home` - Under `Show Tabs` → `Messages Tab`: Enable `Allow users to send Slash commands and messages from the messages tab` diff --git a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/streamlit.md b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/streamlit.md index a2441bf3598..a114ec249be 100644 --- a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/streamlit.md +++ b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/streamlit.md @@ -10,7 +10,7 @@ show_related_blogs: true doc_type: 'guide' --- -# How to build a ClickHouse-backed AI Agent with Streamlit +# How to build a clickhouse-backed ai agent with streamlit In this guide you'll learn how to build a web-based AI agent using [Streamlit](https://streamlit.io/) that can interact with [ClickHouse's SQL playground](https://sql.clickhouse.com/) using [ClickHouse's MCP Server](https://github.com/ClickHouse/mcp-clickhouse) and [Agno](https://github.com/agno-agi/agno). @@ -173,7 +173,7 @@ async def _agent_stream_to_queue(message, queue): queue.put(chunk) ``` -## Create the Streamlit interface {#create-interface} +## Create the streamlit interface {#create-interface} Add the Streamlit UI components and chat functionality: diff --git a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/upsonic.md b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/upsonic.md index 44132502b0f..630cb761cf7 100644 --- a/docs/use-cases/AI_ML/MCP/ai_agent_libraries/upsonic.md +++ b/docs/use-cases/AI_ML/MCP/ai_agent_libraries/upsonic.md @@ -10,7 +10,7 @@ show_related_blogs: true doc_type: 'guide' --- -# How to build an AI Agent with Upsonic and the ClickHouse MCP Server +# How to build an ai agent with upsonic and the ClickHouse mcp server In this guide you'll learn how to build a [Upsonic](https://github.com/Upsonic/Upsonic/tree/master) AI agent that can interact with [ClickHouse's SQL playground](https://sql.clickhouse.com/) using [ClickHouse's MCP Server](https://github.com/ClickHouse/mcp-clickhouse). @@ -63,7 +63,7 @@ env = { } ``` -## Initialize MCP Server and Upsonic agent {#initialize-mcp-and-agent} +## Initialize mcp server and upsonic agent {#initialize-mcp-and-agent} Now configure the ClickHouse MCP Server to point at the ClickHouse SQL playground and also initialize our agent and ask it a question: diff --git a/docs/use-cases/AI_ML/MCP/index.md b/docs/use-cases/AI_ML/MCP/index.md index 6241f8b9fd2..bf73ec2e153 100644 --- a/docs/use-cases/AI_ML/MCP/index.md +++ b/docs/use-cases/AI_ML/MCP/index.md @@ -22,7 +22,7 @@ You can think of it as a universal API standard designed specifically for AI sys The key advantage of MCP is that AI libraries only need to implement support for the protocol once. After that, all MCP-compatible services become immediately accessible, saving AI library maintainers a lot of time. -## What is MCP's architecture? {#mcp-architecture} +## What is mcp's architecture? {#mcp-architecture} MCP follows a client-server architecture: @@ -34,7 +34,7 @@ A diagram showing the architecture is below: MCP overview -## Does ClickHouse have an MCP Server? {#clickhouse-mcp-server} +## Does ClickHouse have an mcp server? {#clickhouse-mcp-server} It does! The [ClickHouse MCP Server](https://github.com/ClickHouse/mcp-clickhouse) offers the following tools: @@ -43,7 +43,7 @@ The [ClickHouse MCP Server](https://github.com/ClickHouse/mcp-clickhouse) offers * `list_databases` - List all databases on your ClickHouse cluster. * `list_tables` - List all tables in a database. -## Guides for using the ClickHouse MCP Server {#clickhouse-mcp-server-guides} +## Guides for using the ClickHouse mcp server {#clickhouse-mcp-server-guides} Below are some guides showing how to use the ClickHouse MCP Server. diff --git a/docs/use-cases/AI_ML/ai-powered-sql-generation.md b/docs/use-cases/AI_ML/ai-powered-sql-generation.md index 3862d124931..0e5a618a77f 100644 --- a/docs/use-cases/AI_ML/ai-powered-sql-generation.md +++ b/docs/use-cases/AI_ML/ai-powered-sql-generation.md @@ -128,7 +128,7 @@ If we run that query, we'll see the following output: If we want to ask follow up questions, we need to ask our question from scratch. -### Finding expensive properties in Greater London {#finding-expensive-properties-in-greater-london} +### Finding expensive properties in greater London {#finding-expensive-properties-in-greater-london} Since the feature doesn't maintain conversation history, each query must be self-contained. When asking follow-up questions, you need to provide the full context rather than referring to previous queries. For example, after seeing the previous results, we might want to focus specifically on Greater London properties. Rather than asking "What about Greater London?", we need to include the complete context: diff --git a/docs/use-cases/AI_ML/data-exploration/jupyter-notebook.md b/docs/use-cases/AI_ML/data-exploration/jupyter-notebook.md index 4aa3dbb8540..ac5a436b3e3 100644 --- a/docs/use-cases/AI_ML/data-exploration/jupyter-notebook.md +++ b/docs/use-cases/AI_ML/data-exploration/jupyter-notebook.md @@ -18,7 +18,7 @@ import image_7 from '@site/static/images/use-cases/AI_ML/jupyter/7.png'; import image_8 from '@site/static/images/use-cases/AI_ML/jupyter/8.png'; import image_9 from '@site/static/images/use-cases/AI_ML/jupyter/9.png'; -# Exploring data with Jupyter notebooks and chDB +# Exploring data with jupyter notebooks and chDB In this guide, you will learn how you can explore a dataset on ClickHouse Cloud data in Jupyter notebook with the help of [chDB](/chdb) - a fast in-process SQL OLAP Engine powered by ClickHouse. diff --git a/docs/use-cases/AI_ML/data-exploration/marimo-notebook.md b/docs/use-cases/AI_ML/data-exploration/marimo-notebook.md index 2b490a817b1..392361c5342 100644 --- a/docs/use-cases/AI_ML/data-exploration/marimo-notebook.md +++ b/docs/use-cases/AI_ML/data-exploration/marimo-notebook.md @@ -74,7 +74,7 @@ The environment variables above persist only as long as your terminal session. To set them permanently, add them to your shell configuration file. ::: -### Installing Marimo {#installing-marimo} +### Installing marimo {#installing-marimo} Now activate your virtual environment. From within your virtual environment, install the following packages that we will be using in this guide: diff --git a/docs/use-cases/AI_ML/index.md b/docs/use-cases/AI_ML/index.md index 0649664c482..3f104570fd5 100644 --- a/docs/use-cases/AI_ML/index.md +++ b/docs/use-cases/AI_ML/index.md @@ -8,7 +8,7 @@ keywords: ['machine learning', 'genAI', 'AI'] doc_type: 'landing-page' --- -# Machine Learning and GenAI +# Machine learning and genai ClickHouse is ideally suited as a real-time database to power Machine Learning workloads. With ClickHouse, it's easier than ever to unleash GenAI on your analytics data. diff --git a/docs/use-cases/data_lake/glue_catalog.md b/docs/use-cases/data_lake/glue_catalog.md index 0324fc8548b..20cc7cdfedb 100644 --- a/docs/use-cases/data_lake/glue_catalog.md +++ b/docs/use-cases/data_lake/glue_catalog.md @@ -24,7 +24,7 @@ Glue supports many different table formats, but this integration only supports Iceberg tables. ::: -## Configuring Glue in AWS {#configuring} +## Configuring glue in AWS {#configuring} To connect to the glue catalog, you will need to identify the region of your catalog and provide an access and secret key. @@ -34,7 +34,7 @@ Currently, the Glue catalog only supports access and secret keys, but we will support additional authentication approaches in the future. ::: -## Creating a connection between Glue data catalog and ClickHouse {#connecting} +## Creating a connection between glue data catalog and ClickHouse {#connecting} With your Unity Catalog configured and authentication in place, establish a connection between ClickHouse and Unity Catalog. @@ -49,7 +49,7 @@ SETTINGS aws_secret_access_key = '' ``` -## Query the Glue data catalog using ClickHouse {#query-glue-catalog} +## Query the glue data catalog using ClickHouse {#query-glue-catalog} Now that the connection is in place, you can start querying Glue: @@ -201,7 +201,7 @@ SHOW CREATE TABLE `iceberg-benchmark.hitsiceberg`; └─────────────────────────────────────────────────────────┘ ``` -## Loading data from your Data Lake into ClickHouse {#loading-data-into-clickhouse} +## Loading data from your data lake into ClickHouse {#loading-data-into-clickhouse} If you need to load data from Databricks into ClickHouse, start by creating a local ClickHouse table: diff --git a/docs/use-cases/data_lake/lakekeeper_catalog.md b/docs/use-cases/data_lake/lakekeeper_catalog.md index 18f76e96fe3..9dba87df6b3 100644 --- a/docs/use-cases/data_lake/lakekeeper_catalog.md +++ b/docs/use-cases/data_lake/lakekeeper_catalog.md @@ -32,7 +32,7 @@ As this feature is experimental, you will need to enable it using: `SET allow_experimental_database_iceberg = 1;` ::: -## Local Development Setup {#local-development-setup} +## Local development setup {#local-development-setup} For local development and testing, you can use a containerized Lakekeeper setup. This approach is ideal for learning, prototyping, and development environments. @@ -41,7 +41,7 @@ For local development and testing, you can use a containerized Lakekeeper setup. 1. **Docker and Docker Compose**: Ensure Docker is installed and running 2. **Sample Setup**: You can use the Lakekeeper docker-compose setup -### Setting up Local Lakekeeper Catalog {#setting-up-local-lakekeeper-catalog} +### Setting up local lakekeeper catalog {#setting-up-local-lakekeeper-catalog} You can use the official [Lakekeeper docker-compose setup](https://github.com/lakekeeper/lakekeeper/tree/main/examples/minimal) which provides a complete environment with Lakekeeper, PostgreSQL metadata backend, and MinIO for object storage. @@ -229,7 +229,7 @@ docker-compose logs -f The Lakekeeper setup requires that sample data be loaded into the Iceberg tables first. Make sure the environment has created and populated the tables before attempting to query them through ClickHouse. The availability of tables depends on the specific docker-compose setup and sample data loading scripts. ::: -### Connecting to Local Lakekeeper Catalog {#connecting-to-local-lakekeeper-catalog} +### Connecting to local lakekeeper catalog {#connecting-to-local-lakekeeper-catalog} Connect to your ClickHouse container: @@ -247,7 +247,7 @@ ENGINE = DataLakeCatalog('http://lakekeeper:8181/catalog', 'minio', 'ClickHouse_ SETTINGS catalog_type = 'rest', storage_endpoint = 'http://minio:9002/warehouse-rest', warehouse = 'demo' ``` -## Querying Lakekeeper catalog tables using ClickHouse {#querying-lakekeeper-catalog-tables-using-clickhouse} +## Querying lakekeeper catalog tables using ClickHouse {#querying-lakekeeper-catalog-tables-using-clickhouse} Now that the connection is in place, you can start querying via the Lakekeeper catalog. For example: @@ -327,7 +327,7 @@ SHOW CREATE TABLE `default.taxis`; └───────────────────────────────────────────────────────────────────────────────────────────────┘ ``` -## Loading data from your Data Lake into ClickHouse {#loading-data-from-your-data-lake-into-clickhouse} +## Loading data from your data lake into ClickHouse {#loading-data-from-your-data-lake-into-clickhouse} If you need to load data from the Lakekeeper catalog into ClickHouse, start by creating a local ClickHouse table: diff --git a/docs/use-cases/data_lake/nessie_catalog.md b/docs/use-cases/data_lake/nessie_catalog.md index 01609c3debf..a671ffeca2b 100644 --- a/docs/use-cases/data_lake/nessie_catalog.md +++ b/docs/use-cases/data_lake/nessie_catalog.md @@ -34,7 +34,7 @@ As this feature is experimental, you will need to enable it using: `SET allow_experimental_database_iceberg = 1;` ::: -## Local Development Setup {#local-development-setup} +## Local development setup {#local-development-setup} For local development and testing, you can use a containerized Nessie setup. This approach is ideal for learning, prototyping, and development environments. @@ -43,7 +43,7 @@ For local development and testing, you can use a containerized Nessie setup. Thi 1. **Docker and Docker Compose**: Ensure Docker is installed and running 2. **Sample Setup**: You can use the official Nessie docker-compose setup -### Setting up Local Nessie Catalog {#setting-up-local-nessie-catalog} +### Setting up local nessie catalog {#setting-up-local-nessie-catalog} You can use the official [Nessie docker-compose setup](https://projectnessie.org/guides/setting-up/) which provides a complete environment with Nessie, in-memory version store, and MinIO for object storage. @@ -147,7 +147,7 @@ docker-compose logs -f The Nessie setup uses an in-memory version store and requires that sample data be loaded into the Iceberg tables first. Make sure the environment has created and populated the tables before attempting to query them through ClickHouse. ::: -### Connecting to Local Nessie Catalog {#connecting-to-local-nessie-catalog} +### Connecting to local nessie catalog {#connecting-to-local-nessie-catalog} Connect to your ClickHouse container: @@ -165,7 +165,7 @@ ENGINE = DataLakeCatalog('http://nessie:19120/iceberg', 'admin', 'password') SETTINGS catalog_type = 'rest', storage_endpoint = 'http://minio:9002/my-bucket', warehouse = 'warehouse' ``` -## Querying Nessie catalog tables using ClickHouse {#querying-nessie-catalog-tables-using-clickhouse} +## Querying nessie catalog tables using ClickHouse {#querying-nessie-catalog-tables-using-clickhouse} Now that the connection is in place, you can start querying via the Nessie catalog. For example: @@ -245,7 +245,7 @@ SHOW CREATE TABLE `default.taxis`; └───────────────────────────────────────────────────────────────────────────────────────────────┘ ``` -## Loading data from your Data Lake into ClickHouse {#loading-data-from-your-data-lake-into-clickhouse} +## Loading data from your data lake into ClickHouse {#loading-data-from-your-data-lake-into-clickhouse} If you need to load data from the Nessie catalog into ClickHouse, start by creating a local ClickHouse table: diff --git a/docs/use-cases/data_lake/onelake_catalog.md b/docs/use-cases/data_lake/onelake_catalog.md index 158d6b56086..90e944e5009 100644 --- a/docs/use-cases/data_lake/onelake_catalog.md +++ b/docs/use-cases/data_lake/onelake_catalog.md @@ -23,7 +23,7 @@ As this feature is beta, you will need to enable it using: `SET allow_database_iceberg = 1;` ::: -## Gathering Requirements OneLake {#gathering-requirements} +## Gathering requirements onelake {#gathering-requirements} Before querying your table in Microsoft Fabric, you'll need to collect the following information: @@ -34,7 +34,7 @@ Before querying your table in Microsoft Fabric, you'll need to collect the follo See [Microsoft OneLake's documentation](http://learn.microsoft.com/en-us/fabric/onelake/table-apis/table-apis-overview#prerequisites) for help finding these values. -## Creating a connection between OneLake and ClickHouse {#creating-a-connection-between-unity-catalog-and-clickhouse} +## Creating a connection between onelake and ClickHouse {#creating-a-connection-between-unity-catalog-and-clickhouse} With the required info above you can now create a connection between Microsoft OneLake and ClickHouse, but before that you need to enable catalogs: @@ -42,7 +42,7 @@ With the required info above you can now create a connection between Microsoft O SET allow_database_iceberg=1 ``` -### Connect to OneLake {#connect-onelake} +### Connect to onelake {#connect-onelake} ```sql CREATE DATABASE onelake_catalog @@ -57,7 +57,7 @@ onelake_client_id = '', onelake_client_secret = '' ``` -## Querying OneLake using ClickHouse {#querying-onelake-using-clickhouse} +## Querying onelake using ClickHouse {#querying-onelake-using-clickhouse} Now that the connection is in place, you can start querying OneLake: @@ -151,7 +151,7 @@ Query id: 8bd5bd8e-83be-453e-9a88-32de12ba7f24 └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ``` -## Loading data from your Data Lake into ClickHouse {#loading-data-from-onelake-into-clickhouse} +## Loading data from your data lake into ClickHouse {#loading-data-from-onelake-into-clickhouse} If you need to load data from OneLake into ClickHouse: diff --git a/docs/use-cases/data_lake/rest_catalog.md b/docs/use-cases/data_lake/rest_catalog.md index 7d91e001d7b..28e2afb68af 100644 --- a/docs/use-cases/data_lake/rest_catalog.md +++ b/docs/use-cases/data_lake/rest_catalog.md @@ -32,7 +32,7 @@ As this feature is experimental, you will need to enable it using: `SET allow_experimental_database_iceberg = 1;` ::: -## Local Development Setup {#local-development-setup} +## Local development setup {#local-development-setup} For local development and testing, you can use a containerized REST catalog setup. This approach is ideal for learning, prototyping, and development environments. @@ -41,7 +41,7 @@ For local development and testing, you can use a containerized REST catalog setu 1. **Docker and Docker Compose**: Ensure Docker is installed and running 2. **Sample Setup**: You can use various docker-compose setups (see Alternative Docker Images below) -### Setting up Local REST Catalog {#setting-up-local-rest-catalog} +### Setting up local REST catalog {#setting-up-local-rest-catalog} You can use various containerized REST catalog implementations such as **[Databricks docker-spark-iceberg](https://github.com/databricks/docker-spark-iceberg/blob/main/docker-compose.yml?ref=blog.min.io)** which provides a complete Spark + Iceberg + REST catalog environment with docker-compose, making it ideal for testing Iceberg integrations. @@ -88,7 +88,7 @@ docker-compose logs -f The REST catalog setup requires that sample data be loaded into the Iceberg tables first. Make sure the Spark environment has created and populated the tables before attempting to query them through ClickHouse. The availability of tables depends on the specific docker-compose setup and sample data loading scripts. ::: -### Connecting to Local REST Catalog {#connecting-to-local-rest-catalog} +### Connecting to local REST catalog {#connecting-to-local-rest-catalog} Connect to your ClickHouse container: @@ -189,7 +189,7 @@ SHOW CREATE TABLE `default.taxis`; └───────────────────────────────────────────────────────────────────────────────────────────────┘ ``` -## Loading data from your Data Lake into ClickHouse {#loading-data-from-your-data-lake-into-clickhouse} +## Loading data from your data lake into ClickHouse {#loading-data-from-your-data-lake-into-clickhouse} If you need to load data from the REST catalog into ClickHouse, start by creating a local ClickHouse table: diff --git a/docs/use-cases/data_lake/unity_catalog.md b/docs/use-cases/data_lake/unity_catalog.md index 36b6ac85487..7feac1c66e6 100644 --- a/docs/use-cases/data_lake/unity_catalog.md +++ b/docs/use-cases/data_lake/unity_catalog.md @@ -29,7 +29,7 @@ As this feature is experimental, you will need to enable it using: `SET allow_experimental_database_unity_catalog = 1;` ::: -## Configuring Unity in Databricks {#configuring-unity-in-databricks} +## Configuring unity in databricks {#configuring-unity-in-databricks} To allow ClickHouse to interact with the Unity catalog, you need to make sure the Unity Catalog is configured to allow interaction with an external reader. This can be achieved by following the[ "Enable external data access to Unity Catalog"](https://docs.databricks.com/aws/en/external-access/admin) guide. @@ -41,11 +41,11 @@ Once your catalog is configured, you must generate credentials for ClickHouse. T * For Delta clients, use a Personal Access Token ([PAT](https://docs.databricks.com/aws/en/dev-tools/auth/pat)). -## Creating a connection between Unity Catalog and ClickHouse {#creating-a-connection-between-unity-catalog-and-clickhouse} +## Creating a connection between unity catalog and ClickHouse {#creating-a-connection-between-unity-catalog-and-clickhouse} With your Unity Catalog configured and authentication in place, establish a connection between ClickHouse and Unity Catalog. -### Read Delta {#read-delta} +### Read delta {#read-delta} ```sql CREATE DATABASE unity @@ -53,7 +53,7 @@ ENGINE = DataLakeCatalog('https://.cloud.databricks.com/api/2.1/un SETTINGS warehouse = 'CATALOG_NAME', catalog_credential = '', catalog_type = 'unity' ``` -### Read Iceberg {#read-iceberg} +### Read iceberg {#read-iceberg} ```sql CREATE DATABASE unity @@ -62,7 +62,7 @@ SETTINGS catalog_type = 'rest', catalog_credential = ':); ``` -## Loading data from your Data Lake into ClickHouse {#loading-data-from-your-data-lake-into-clickhouse} +## Loading data from your data lake into ClickHouse {#loading-data-from-your-data-lake-into-clickhouse} If you need to load data from Databricks into ClickHouse, start by creating a local ClickHouse table: diff --git a/docs/use-cases/observability/build-your-own/grafana.md b/docs/use-cases/observability/build-your-own/grafana.md index 70b0488b2bb..8ff4cde4d66 100644 --- a/docs/use-cases/observability/build-your-own/grafana.md +++ b/docs/use-cases/observability/build-your-own/grafana.md @@ -19,14 +19,14 @@ import observability_23 from '@site/static/images/use-cases/observability/observ import observability_24 from '@site/static/images/use-cases/observability/observability-24.png'; import Image from '@theme/IdealImage'; -# Using Grafana and ClickHouse for Observability +# Using Grafana and ClickHouse for observability Grafana represents the preferred visualization tool for Observability data in ClickHouse. This is achieved using the official ClickHouse plugin for Grafana. Users can follow the installation instructions found [here](/integrations/grafana). V4 of the plugin makes logs and traces a first-class citizen in a new query builder experience. This minimizes the need for SREs to write SQL queries and simplifies SQL-based Observability, moving the needle forward for this emerging paradigm. Part of this has been placing OpenTelemetry (OTel) at the core of the plugin, as we believe this will be the foundation of SQL-based Observability over the coming years and how data will be collected. -## OpenTelemetry Integration {#open-telemetry-integration} +## Opentelemetry integration {#open-telemetry-integration} On configuring a ClickHouse datasource in Grafana, the plugin allows the user to specify a default database and table for logs and traces and whether these tables conform to the OTel schema. This allows the plugin to return the columns required for correct log and trace rendering in Grafana. If you've made changes to the default OTel schema and prefer to use your own column names, these can be specified. Usage of the default OTel column names for columns such as time (`Timestamp`), log level (`SeverityText`), or message body (`Body`) means no changes need to be made. diff --git a/docs/use-cases/observability/build-your-own/integrating-opentelemetry.md b/docs/use-cases/observability/build-your-own/integrating-opentelemetry.md index 6f87bccd860..e042dcd3bbc 100644 --- a/docs/use-cases/observability/build-your-own/integrating-opentelemetry.md +++ b/docs/use-cases/observability/build-your-own/integrating-opentelemetry.md @@ -16,7 +16,7 @@ import observability_8 from '@site/static/images/use-cases/observability/observa import observability_9 from '@site/static/images/use-cases/observability/observability-9.png'; import Image from '@theme/IdealImage'; -# Integrating OpenTelemetry for data collection +# Integrating opentelemetry for data collection Any Observability solution requires a means of collecting and exporting logs and traces. For this purpose, ClickHouse recommends [the OpenTelemetry (OTel) project](https://opentelemetry.io/). @@ -44,7 +44,7 @@ This distribution contains many components and allows users to experiment with v Building a [custom collector](https://opentelemetry.io/docs/collector/custom-collector/) can be achieved using the [OpenTelemetry Collector Builder](https://github.com/open-telemetry/opentelemetry-collector/tree/main/cmd/builder). -## Ingesting data with OTel {#ingesting-data-with-otel} +## Ingesting data with otel {#ingesting-data-with-otel} ### Collector deployment roles {#collector-deployment-roles} diff --git a/docs/use-cases/observability/build-your-own/introduction.md b/docs/use-cases/observability/build-your-own/introduction.md index 073298371ff..a8b12316115 100644 --- a/docs/use-cases/observability/build-your-own/introduction.md +++ b/docs/use-cases/observability/build-your-own/introduction.md @@ -45,7 +45,7 @@ More specifically, the following means ClickHouse is ideally suited for the stor - **Secondary indices** - ClickHouse supports secondary indexes, such as bloom filters, to accelerate specific query profiles. These can be optionally enabled at a column level, giving the user granular control and allowing them to assess the cost-performance benefit. - **Open-source & Open standards** - As an open-source database, ClickHouse embraces open standards such as OpenTelemetry. The ability to contribute and actively participate in projects is appealing while avoiding the challenges of vendor lock-in. -## When should you use ClickHouse for Observability {#when-should-you-use-clickhouse-for-observability} +## When should you use ClickHouse for observability {#when-should-you-use-clickhouse-for-observability} Using ClickHouse for observability data requires users to embrace SQL-based observability. We recommend [this blog post](https://clickhouse.com/blog/the-state-of-sql-based-observability) for a history of SQL-based observability, but in summary: diff --git a/docs/use-cases/observability/build-your-own/managing-data.md b/docs/use-cases/observability/build-your-own/managing-data.md index 537ac240c61..ff479543fd6 100644 --- a/docs/use-cases/observability/build-your-own/managing-data.md +++ b/docs/use-cases/observability/build-your-own/managing-data.md @@ -156,7 +156,7 @@ We explore both of these in detail below. While partitions can assist with query performance, this depends heavily on the access patterns. If queries target only a few partitions (ideally one), performance can potentially improve. This is only typically useful if the partitioning key is not in the primary key and you are filtering by it. However, queries which need to cover many partitions may perform worse than if no partitioning is used (as there may possibly be more parts). The benefit of targeting a single partition will be even less pronounced to non-existent if the partitioning key is already an early entry in the primary key. Partitioning can also be used to [optimize GROUP BY queries](/engines/table-engines/mergetree-family/custom-partitioning-key#group-by-optimisation-using-partition-key) if values in each partition are unique. However, in general, users should ensure the primary key is optimized and only consider partitioning as a query optimization technique in exceptional cases where access patterns access a specific predictable subset of the data, e.g., partitioning by day, with most queries in the last day. See [here](https://medium.com/datadenys/using-partitions-in-clickhouse-3ea0decb89c4) for an example of this behavior. -## Data management with TTL (Time-to-live) {#data-management-with-ttl-time-to-live} +## Data management with TTL (time-to-live) {#data-management-with-ttl-time-to-live} Time-to-Live (TTL) is a crucial feature in observability solutions powered by ClickHouse for efficient data retention and management, especially given vast amounts of data are continuously generated. Implementing TTL in ClickHouse allows for automatic expiration and deletion of older data, ensuring that the storage is optimally used and performance is maintained without manual intervention. This capability is essential for keeping the database lean, reducing storage costs, and ensuring that queries remain fast and efficient by focusing on the most relevant and recent data. Moreover, it helps in compliance with data retention policies by systematically managing data life cycles, thus enhancing the overall sustainability and scalability of the observability solution. diff --git a/docs/use-cases/observability/build-your-own/schema-design.md b/docs/use-cases/observability/build-your-own/schema-design.md index 68f99d78f05..6c05c097b37 100644 --- a/docs/use-cases/observability/build-your-own/schema-design.md +++ b/docs/use-cases/observability/build-your-own/schema-design.md @@ -600,7 +600,7 @@ We recommend that users familiarize themselves with the basics of dictionaries. For simple enrichment examples see the guide on Dictionaries [here](/dictionary). Below, we focus on common observability enrichment tasks. -### Using IP dictionaries {#using-ip-dictionaries} +### Using ip dictionaries {#using-ip-dictionaries} Geo-enriching logs and traces with latitude and longitude values using IP addresses is a common Observability requirement. We can achieve this using `ip_trie` structured dictionary. @@ -1021,7 +1021,7 @@ For more examples and details on dictionaries, we recommend the following articl ClickHouse supports a number of techniques for accelerating query performance. The following should be considered only after choosing an appropriate primary/ordering key to optimize for the most popular access patterns and to maximize compression. This will usually have the largest impact on performance for the least effort. -### Using Materialized views (incremental) for aggregations {#using-materialized-views-incremental-for-aggregations} +### Using materialized views (incremental) for aggregations {#using-materialized-views-incremental-for-aggregations} In earlier sections, we explored the use of Materialized views for data transformation and filtering. Materialized views can, however, also be used to precompute aggregations at insert time and store the result. This result can be updated with the results from subsequent inserts, thus effectively allowing an aggregation to be precomputed at insert time. @@ -1226,7 +1226,7 @@ ORDER BY Hour DESC Note we use a `GROUP BY` here instead of using `FINAL`. -### Using Materialized views (incremental) for fast lookups {#using-materialized-views-incremental--for-fast-lookups} +### Using materialized views (incremental) for fast lookups {#using-materialized-views-incremental--for-fast-lookups} Users should consider their access patterns when choosing the ClickHouse ordering key with the columns that are frequently used in filter and aggregation clauses. This can be restrictive in Observability use cases, where users have more diverse access patterns that cannot be encapsulated in a single set of columns. This is best illustrated in an example built into the default OTel schemas. Consider the default schema for the traces: diff --git a/docs/use-cases/observability/clickstack/alerts.md b/docs/use-cases/observability/clickstack/alerts.md index c215137f0d4..b4b04b131ad 100644 --- a/docs/use-cases/observability/clickstack/alerts.md +++ b/docs/use-cases/observability/clickstack/alerts.md @@ -34,7 +34,7 @@ Alerting works seamlessly across your ClickStack data, helping you track system ClickStack supports two complementary ways to create alerts: **Search alerts** and **Dashboard chart alerts**. Once the alert is created, it is attached to either the search or the chart. -### 1. Search alerts {#search-alerts} +### 1. search alerts {#search-alerts} Search alerts allow you to trigger notifications based on the results of a saved search. They help you detect when specific events or patterns occur more (or less) frequently than expected. @@ -71,7 +71,7 @@ Note that multiple alerts can be added to a search. If the above process is repe Multiple alerts -### 2. Dashboard chart alerts {#dashboard-alerts} +### 2. dashboard chart alerts {#dashboard-alerts} Dashboard alerts extend alerting to charts. diff --git a/docs/use-cases/observability/clickstack/architecture.md b/docs/use-cases/observability/clickstack/architecture.md index 67290f6dd29..ddaa45ba173 100644 --- a/docs/use-cases/observability/clickstack/architecture.md +++ b/docs/use-cases/observability/clickstack/architecture.md @@ -29,7 +29,7 @@ At the heart of ClickStack is ClickHouse, a column-oriented database designed fo ClickHouse handles observability data as wide events, allowing for deep correlation across logs, metrics, and traces in a single unified structure. -## OpenTelemetry collector: data ingestion {#open-telemetry-collector} +## Opentelemetry collector: data ingestion {#open-telemetry-collector} ClickStack includes a pre-configured OpenTelemetry (OTel) collector to ingest telemetry in an open, standardized way. Users can send data using the OTLP protocol via: @@ -38,7 +38,7 @@ ClickStack includes a pre-configured OpenTelemetry (OTel) collector to ingest te The collector exports telemetry to ClickHouse in efficient batches. It supports optimized table schemas per data source, ensuring scalable performance across all signal types. -## HyperDX: the interface {#hyperdx} +## Hyperdx: the interface {#hyperdx} HyperDX is the user interface for ClickStack. It offers: diff --git a/docs/use-cases/observability/clickstack/config.md b/docs/use-cases/observability/clickstack/config.md index 3f974ab0d28..8c6af364b01 100644 --- a/docs/use-cases/observability/clickstack/config.md +++ b/docs/use-cases/observability/clickstack/config.md @@ -24,7 +24,7 @@ If using the [All in One](/use-cases/observability/clickstack/deployment/all-in- docker run -e HYPERDX_LOG_LEVEL='debug' -p 8080:8080 -p 4317:4317 -p 4318:4318 clickhouse/clickstack-all-in-one:latest ``` -### Docker Compose {#docker-compose} +### Docker compose {#docker-compose} If using the [Docker Compose](/use-cases/observability/clickstack/deployment/docker-compose) deployment guide, the [`.env`](https://github.com/hyperdxio/hyperdx/blob/main/.env) file can be used to modify settings. @@ -93,7 +93,7 @@ ingress: value: abc ``` -## HyperDX {#hyperdx} +## Hyperdx {#hyperdx} ### Data source settings {#datasource-settings} @@ -346,7 +346,7 @@ These settings can't be modified when HyperDX is managed in ClickHouse Cloud. - **Guidance:** - Set to `true` to enable JSON support in ClickStack. -## OpenTelemetry collector {#otel-collector} +## Opentelemetry collector {#otel-collector} See ["ClickStack OpenTelemetry Collector"](/use-cases/observability/clickstack/ingesting-data/otel-collector) for more details. diff --git a/docs/use-cases/observability/clickstack/dashboards.md b/docs/use-cases/observability/clickstack/dashboards.md index fe258396f52..525c81b1601 100644 --- a/docs/use-cases/observability/clickstack/dashboards.md +++ b/docs/use-cases/observability/clickstack/dashboards.md @@ -43,7 +43,7 @@ For the examples below, we use the remote dataset available at [sql.clickhouse.c -### Navigate to Chart Explorer {#navigate-chart-explorer} +### Navigate to chart explorer {#navigate-chart-explorer} Select `Chart Explorer` from the left menu. @@ -87,7 +87,7 @@ We create a dashboard with two visualizations below using the log and trace data -### Navigate to Dashboards {#navigate-dashboards} +### Navigate to dashboards {#navigate-dashboards} Select `Dashboards` from the left menu. @@ -152,18 +152,18 @@ The dashboard will be auto-saved. To set the dashboard name, select the title an -## Dashboards - Editing visualizations {#dashboards-editing-visualizations} +## Dashboards - editing visualizations {#dashboards-editing-visualizations} To remove, edit, or duplicate a visualization, hover over it and use the corresponding action buttons. Dashboard edit -## Dashboard - Listing and search {#dashboard-listing-search} +## Dashboard - listing and search {#dashboard-listing-search} Dashboards are accessible from the left-hand menu, with built-in search to quickly locate specific dashboards. Dashboard search -## Dashboards - Tagging {#tagging} +## Dashboards - tagging {#tagging} ## Presets {#presets} diff --git a/docs/use-cases/observability/clickstack/deployment/all-in-one.md b/docs/use-cases/observability/clickstack/deployment/all-in-one.md index 1b28a62e208..aa08a4f3ccc 100644 --- a/docs/use-cases/observability/clickstack/deployment/all-in-one.md +++ b/docs/use-cases/observability/clickstack/deployment/all-in-one.md @@ -45,7 +45,7 @@ docker run -p 8080:8080 -p 4317:4317 -p 4318:4318 clickhouse/clickstack-all-in-o ClickStack images are now published as `clickhouse/clickstack-*` (previously `docker.hyperdx.io/hyperdx/*`). ::: -### Navigate to the HyperDX UI {#navigate-to-hyperdx-ui} +### Navigate to the hyperdx UI {#navigate-to-hyperdx-ui} Visit [http://localhost:8080](http://localhost:8080) to access the HyperDX UI. @@ -116,7 +116,7 @@ The `CLICKHOUSE_ENDPOINT` should be the ClickHouse Cloud HTTPS endpoint, includi On connecting to the HyperDX UI, navigate to [`Team Settings`](http://localhost:8080/team) and create a connection to your ClickHouse Cloud service - followed by the required sources. For an example flow, see [here](/use-cases/observability/clickstack/getting-started#create-a-cloud-connection). -## Configuring the OpenTelemetry collector {#configuring-collector} +## Configuring the opentelemetry collector {#configuring-collector} The OTel collector configuration can be modified if required - see ["Modifying configuration"](/use-cases/observability/clickstack/ingesting-data/otel-collector#modifying-otel-collector-configuration). diff --git a/docs/use-cases/observability/clickstack/deployment/docker-compose.md b/docs/use-cases/observability/clickstack/deployment/docker-compose.md index 9f93e40b2a3..40bbf4398d9 100644 --- a/docs/use-cases/observability/clickstack/deployment/docker-compose.md +++ b/docs/use-cases/observability/clickstack/deployment/docker-compose.md @@ -54,7 +54,7 @@ git clone https://github.com/ClickHouse/ClickStack.git docker compose up ``` -### Navigate to the HyperDX UI {#navigate-to-hyperdx-ui} +### Navigate to the hyperdx UI {#navigate-to-hyperdx-ui} Visit [http://localhost:8080](http://localhost:8080) to access the HyperDX UI. @@ -111,7 +111,7 @@ HYPERDX_OPAMP_PORT=4320 HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE=default ``` -### Configuring the OpenTelemetry collector {#configuring-collector} +### Configuring the opentelemetry collector {#configuring-collector} The OTel collector configuration can be modified if required - see ["Modifying configuration"](/use-cases/observability/clickstack/ingesting-data/otel-collector#modifying-otel-collector-configuration). diff --git a/docs/use-cases/observability/clickstack/deployment/helm/helm-cloud.md b/docs/use-cases/observability/clickstack/deployment/helm/helm-cloud.md index b323ed9dbda..536e676ae87 100644 --- a/docs/use-cases/observability/clickstack/deployment/helm/helm-cloud.md +++ b/docs/use-cases/observability/clickstack/deployment/helm/helm-cloud.md @@ -11,11 +11,11 @@ keywords: ['ClickStack GKE', 'ClickStack EKS', 'ClickStack AKS', 'Kubernetes clo This guide covers cloud-specific configurations for deploying ClickStack on managed Kubernetes services. For basic installation, see the [main Helm deployment guide](/docs/use-cases/observability/clickstack/deployment/helm). -## Google Kubernetes Engine (GKE) {#google-kubernetes-engine-gke} +## Google Kubernetes engine (gke) {#google-kubernetes-engine-gke} When deploying to GKE, you may need to override certain values due to cloud-specific networking behavior. -### LoadBalancer DNS resolution issue {#loadbalancer-dns-resolution-issue} +### Loadbalancer DNS resolution issue {#loadbalancer-dns-resolution-issue} GKE's LoadBalancer service can cause internal DNS resolution issues where pod-to-pod communication resolves to external IPs instead of staying within the cluster network. This specifically affects the OTEL collector's connection to the OpAMP server. @@ -32,7 +32,7 @@ helm install my-clickstack clickstack/clickstack \ --set otel.opampServerUrl="http://my-clickstack-clickstack-app.default.svc.cluster.local:4320" ``` -### Other GKE considerations {#other-gke-considerations} +### Other gke considerations {#other-gke-considerations} ```yaml # values-gke.yaml @@ -50,7 +50,7 @@ clickhouse: - "10.0.0.0/8" # Fallback for other configurations ``` -## Amazon EKS {#amazon-eks} +## Amazon eks {#amazon-eks} For EKS deployments, consider these common configurations: ```yaml @@ -74,7 +74,7 @@ hyperdx: enabled: true ``` -## Azure AKS {#azure-aks} +## Azure aks {#azure-aks} For AKS deployments: ```yaml @@ -90,7 +90,7 @@ clickhouse: - "10.0.0.0/8" ``` -## Production Cloud deployment checklist {#production-cloud-deployment-checklist} +## Production cloud deployment checklist {#production-cloud-deployment-checklist} Before deploying ClickStack to production on any cloud provider: diff --git a/docs/use-cases/observability/clickstack/deployment/helm/helm-configuration.md b/docs/use-cases/observability/clickstack/deployment/helm/helm-configuration.md index 65bcae02926..318ede47c7a 100644 --- a/docs/use-cases/observability/clickstack/deployment/helm/helm-configuration.md +++ b/docs/use-cases/observability/clickstack/deployment/helm/helm-configuration.md @@ -19,7 +19,7 @@ After successfully deploying ClickStack, configure the API key to enable telemet 2. **Log into the HyperDX dashboard** and navigate to Team settings to generate or retrieve your API key 3. **Update your deployment** with the API key using one of the following methods: -### Method 1: Update via Helm upgrade with values file {#api-key-values-file} +### Method 1: update via Helm upgrade with values file {#api-key-values-file} Add the API key to your `values.yaml`: ```yaml @@ -32,7 +32,7 @@ Then upgrade your deployment: helm upgrade my-clickstack clickstack/clickstack -f values.yaml ``` -### Method 2: Update via Helm upgrade with --set flag {#api-key-set-flag} +### Method 2: update via Helm upgrade with --set flag {#api-key-set-flag} ```shell helm upgrade my-clickstack clickstack/clickstack --set hyperdx.apiKey="your-api-key-here" ``` @@ -181,7 +181,7 @@ spec: kubectl -n ingress-nginx get pods -l app.kubernetes.io/name=ingress-nginx -o jsonpath="{.items[0].spec.containers[0].image}" ``` -## OTEL collector ingress {#otel-collector-ingress} +## Otel collector ingress {#otel-collector-ingress} If you need to expose your OTEL collector endpoints (for traces, metrics, logs) through ingress, use the `additionalIngresses` configuration. This is useful for sending telemetry data from outside the cluster or using a custom domain for the collector. diff --git a/docs/use-cases/observability/clickstack/deployment/helm/helm-deployment-options.md b/docs/use-cases/observability/clickstack/deployment/helm/helm-deployment-options.md index 5d016242899..617e7cfd777 100644 --- a/docs/use-cases/observability/clickstack/deployment/helm/helm-deployment-options.md +++ b/docs/use-cases/observability/clickstack/deployment/helm/helm-deployment-options.md @@ -23,7 +23,7 @@ ClickStack's Helm chart supports multiple deployment configurations: If you have an existing ClickHouse cluster (including ClickHouse Cloud), you can disable the built-in ClickHouse and connect to your external instance. -### Option 1: Inline configuration (development/testing) {#external-clickhouse-inline} +### Option 1: inline configuration (development/testing) {#external-clickhouse-inline} Use this approach for quick testing or non-production environments: ```yaml @@ -53,7 +53,7 @@ Install with this configuration: helm install my-clickstack clickstack/clickstack -f values-external-clickhouse.yaml ``` -### Option 2: External secret (production recommended) {#external-clickhouse-secret} +### Option 2: external secret (production recommended) {#external-clickhouse-secret} For production deployments where you want to keep credentials separate from your Helm configuration: @@ -169,7 +169,7 @@ hyperdx: For a complete example of connecting to ClickHouse Cloud, see ["Create a ClickHouse Cloud connection"](/docs/use-cases/observability/clickstack/getting-started#create-a-cloud-connection). -## External OTEL Collector {#external-otel-collector} +## External otel collector {#external-otel-collector} If you have an existing OTEL collector infrastructure: ```yaml @@ -186,7 +186,7 @@ helm install my-clickstack clickstack/clickstack -f values-external-otel.yaml For instructions on exposing OTEL collector endpoints via ingress, see [Ingress Configuration](/docs/use-cases/observability/clickstack/deployment/helm-configuration#otel-collector-ingress). -## Minimal Deployment {#minimal-deployment} +## Minimal deployment {#minimal-deployment} For organizations with existing infrastructure, deploy only HyperDX: ```yaml @@ -222,7 +222,7 @@ hyperdx: helm install my-clickstack clickstack/clickstack -f values-minimal.yaml ``` -## Next Steps {#next-steps} +## Next steps {#next-steps} - [Configuration Guide](/docs/use-cases/observability/clickstack/deployment/helm-configuration) - API keys, secrets, and ingress setup - [Cloud Deployments](/docs/use-cases/observability/clickstack/deployment/helm-cloud) - GKE, EKS, and AKS specific configurations diff --git a/docs/use-cases/observability/clickstack/deployment/helm/helm.md b/docs/use-cases/observability/clickstack/deployment/helm/helm.md index f4d1dfe2c7b..dcb16b98448 100644 --- a/docs/use-cases/observability/clickstack/deployment/helm/helm.md +++ b/docs/use-cases/observability/clickstack/deployment/helm/helm.md @@ -347,7 +347,7 @@ helm install my-clickstack clickstack/clickstack \ --set "otel.env[0].value=--feature-gates=clickhouse.json" ``` -## Related Documentation {#related-documentation} +## Related documentation {#related-documentation} ### Deployment guides {#deployment-guides} - [Deployment options](/docs/use-cases/observability/clickstack/deployment/helm-deployment-options) - External ClickHouse, OTEL collector, and minimal deployments diff --git a/docs/use-cases/observability/clickstack/deployment/hyperdx-clickhouse-cloud.md b/docs/use-cases/observability/clickstack/deployment/hyperdx-clickhouse-cloud.md index 44f078d3f17..d5a58252161 100644 --- a/docs/use-cases/observability/clickstack/deployment/hyperdx-clickhouse-cloud.md +++ b/docs/use-cases/observability/clickstack/deployment/hyperdx-clickhouse-cloud.md @@ -62,7 +62,7 @@ Press the `Connect` button from the navigation menu. A modal will open offering ClickHouse Cloud connect -### Deploy Open Telemetry Collector (optional) {#deploy-otel-collector} +### Deploy open telemetry collector (optional) {#deploy-otel-collector} **If you have existing observability events you wish to visualize in your service, this step can be skipped.** @@ -276,7 +276,7 @@ docker run --rm -it \ In production, we recommend creating a dedicated user for ingestion, restricting access permissions to the database and tables needed. See ["Database and ingestion user"](/use-cases/observability/clickstack/production#database-ingestion-user) for further details. ::: -### Connect to HyperDX {#connect-to-hyperdx} +### Connect to hyperdx {#connect-to-hyperdx} Select your service, then select `HyperDX` from the left menu. @@ -311,7 +311,7 @@ To enable alerts, at least one user with **Service Admin** permissions (mapped t HyperDX is Open Telemetry native but not Open Telemetry exclusive - users can use their own table schemas if desired. -#### Using Open Telemetry schemas {#using-otel-schemas} +#### Using open telemetry schemas {#using-otel-schemas} If you're using the above OTel collector to create the database and tables within ClickHouse, retain all default values within the create source model, completing the `Table` field with the value `otel_logs` - to create a logs source. All other settings should be auto-detected, allowing you to click `Save New Source`. diff --git a/docs/use-cases/observability/clickstack/deployment/hyperdx-only.md b/docs/use-cases/observability/clickstack/deployment/hyperdx-only.md index 20437404b6f..fea91b17944 100644 --- a/docs/use-cases/observability/clickstack/deployment/hyperdx-only.md +++ b/docs/use-cases/observability/clickstack/deployment/hyperdx-only.md @@ -41,7 +41,7 @@ Run the following command, modifying `YOUR_MONGODB_URI` as required. docker run -e MONGO_URI=mongodb://YOUR_MONGODB_URI -p 8080:8080 docker.hyperdx.io/hyperdx/hyperdx ``` -### Navigate to the HyperDX UI {#navigate-to-hyperdx-ui} +### Navigate to the hyperdx UI {#navigate-to-hyperdx-ui} Visit [http://localhost:8080](http://localhost:8080) to access the HyperDX UI. @@ -65,11 +65,11 @@ Creating a source requires tables to exist in ClickHouse. If you don't have data -## Using Docker Compose {#using-docker-compose} +## Using Docker compose {#using-docker-compose} Users can modify the [Docker Compose configuration](/use-cases/observability/clickstack/deployment/docker-compose) to achieve the same effect as this guide, removing the OTel collector and ClickHouse instance from the manifest. -## ClickStack OpenTelemetry collector {#otel-collector} +## ClickStack opentelemetry collector {#otel-collector} Even if you are managing your own OpenTelemetry collector, independent of the other components in the stack, we still recommend using the ClickStack distribution of the collector. This ensures the default schema is used and best practices for ingestion are applied. diff --git a/docs/use-cases/observability/clickstack/deployment/local-mode-only.md b/docs/use-cases/observability/clickstack/deployment/local-mode-only.md index f5f99dec91a..5526fbe1468 100644 --- a/docs/use-cases/observability/clickstack/deployment/local-mode-only.md +++ b/docs/use-cases/observability/clickstack/deployment/local-mode-only.md @@ -42,7 +42,7 @@ Local mode deploys the HyperDX UI on port 8080. docker run -p 8080:8080 clickhouse/clickstack-local:latest ``` -### Navigate to the HyperDX UI {#navigate-to-hyperdx-ui} +### Navigate to the hyperdx UI {#navigate-to-hyperdx-ui} Visit [http://localhost:8080](http://localhost:8080) to access the HyperDX UI. diff --git a/docs/use-cases/observability/clickstack/event_deltas.md b/docs/use-cases/observability/clickstack/event_deltas.md index d57ba321c05..ddeae541237 100644 --- a/docs/use-cases/observability/clickstack/event_deltas.md +++ b/docs/use-cases/observability/clickstack/event_deltas.md @@ -26,7 +26,7 @@ Instead of manually sifting through trace data, event deltas surface the key pro Event Deltas -## Using Event Deltas {#using-event-deltas} +## Using event deltas {#using-event-deltas} Event Deltas are available directly through the **Search** panel in ClickStack when selecting a source of type `Trace`. @@ -52,7 +52,7 @@ Consider the example above where the `SpanAttributes.app.payment.card_type` colu Conversely, values exclusively associated with inliers can also be interesting. In the example above, the error `Visa Cash Full` appears exclusively in the inliers and is completely absent from the outlier spans. Where this occurs, latency is always less than approximately 50 milliseconds, suggesting this error is associated with low latencies. -## How Event Deltas work {#how-event-deltas-work} +## How event deltas work {#how-event-deltas-work} Event Deltas work by issuing two queries: one for the selected outlier area and one for the inlier area. Each query is limited to the appropriate duration and time window. A sample of events from both result sets is then inspected, and columns for which a high concentration of values appears predominantly in the outliers are identified. Columns for which 100% of a value occurs only in the outlier subset are shown first, highlighting the attributes most responsible for the observed differences. diff --git a/docs/use-cases/observability/clickstack/example-datasets/_snippets/_demo.md b/docs/use-cases/observability/clickstack/example-datasets/_snippets/_demo.md index fc26e6ef3bd..ebca8fabec3 100644 --- a/docs/use-cases/observability/clickstack/example-datasets/_snippets/_demo.md +++ b/docs/use-cases/observability/clickstack/example-datasets/_snippets/_demo.md @@ -1,7 +1,7 @@ import Image from '@theme/IdealImage'; import architecture from '@site/static/images/use-cases/observability/hyperdx-demo/architecture.png'; -### Demo Architecture {#demo-architecture} +### Demo architecture {#demo-architecture} The demo is composed of microservices written in different programming languages that talk to each other over gRPC and HTTP and a load generator that uses Locust to fake user traffic. The original source code for this demo has been modified to use [ClickStack instrumentation](/use-cases/observability/clickstack/sdks). diff --git a/docs/use-cases/observability/clickstack/example-datasets/kubernetes.md b/docs/use-cases/observability/clickstack/example-datasets/kubernetes.md index a25fa4875d4..5ed27084f3f 100644 --- a/docs/use-cases/observability/clickstack/example-datasets/kubernetes.md +++ b/docs/use-cases/observability/clickstack/example-datasets/kubernetes.md @@ -46,7 +46,7 @@ To simulate application traffic, you can optionally deploy the ClickStack fork o -### Install cert-manager (Optional) {#install-cert-manager} +### Install cert-manager (optional) {#install-cert-manager} If your setup needs TLS certificates, install [cert-manager](https://cert-manager.io/) using Helm: @@ -58,7 +58,7 @@ helm repo add jetstack https://charts.jetstack.io helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --set startupapicheck.timeout=5m --set installCRDs=true --set global.leaderElection.namespace=cert-manager ``` -### Deploy the OpenTelemetry Demo (Optional) {#deploy-otel-demo} +### Deploy the opentelemetry demo (optional) {#deploy-otel-demo} This **step is optional and intended for users with no existing pods to monitor**. Although users with existing services deployed in their Kubernetes environment can skip, this demo does include instrumented microservices which generate trace and session replay data - allowing users to explore all features of ClickStack. @@ -190,7 +190,7 @@ my-hyperdx-hdx-oss-v2-mongodb-984845f96-czb6m 1/1 Running 0 my-hyperdx-hdx-oss-v2-otel-collector-64cf698f5c-8s7qj 1/1 Running 0 14m ``` -### Access the HyperDX UI {#access-the-hyperdx-ui} +### Access the hyperdx UI {#access-the-hyperdx-ui} :::note Even when using ClickHouse Cloud, the local HyperDX instance deployed in the Kubernetes cluster is still required. It provides an ingestion key managed by the OpAMP server bundled with HyperDX, with secures ingestion through the deployed OTel collector - a capability not currently available in the ClickHouse Cloud-hosted version. @@ -221,7 +221,7 @@ Navigate to [`Team Settings`](http://localhost:8080/team) and copy the `Ingestio Copy API key -### Create API Key Kubernetes Secret {#create-api-key-kubernetes-secret} +### Create API key Kubernetes secret {#create-api-key-kubernetes-secret} Create a new Kubernetes secret with the Ingestion API Key and a config map containing the location of the OTel collector deployed with the ClickStack helm chart. Later components will use this to allow ingest into the collector deployed with the ClickStack Helm chart: @@ -245,7 +245,7 @@ Trace and log data from demo services should now begin to flow into HyperDX. HyperDX Kubernetes Data -### Add the OpenTelemetry Helm repo {#add-otel-helm-repo} +### Add the opentelemetry Helm repo {#add-otel-helm-repo} To collect Kubernetes metrics, we will deploy a standard OTel collector, configuring this to send data securely to our ClickStack collector using the above ingestion API key. @@ -461,7 +461,7 @@ config:
-### Explore Kubernetes data in HyperDX {#explore-kubernetes-data-hyperdx} +### Explore Kubernetes data in hyperdx {#explore-kubernetes-data-hyperdx} Navigate to your HyperDX UI - either using your Kubernetes-deployed instance or via ClickHouse Cloud. diff --git a/docs/use-cases/observability/clickstack/example-datasets/local-data.md b/docs/use-cases/observability/clickstack/example-datasets/local-data.md index 6a3427a3694..8bbc4c416a6 100644 --- a/docs/use-cases/observability/clickstack/example-datasets/local-data.md +++ b/docs/use-cases/observability/clickstack/example-datasets/local-data.md @@ -25,7 +25,7 @@ This sample dataset can also be used with HyperDX in ClickHouse Cloud, with only -## Create a custom OpenTelemetry configuration {#create-otel-configuration} +## Create a custom opentelemetry configuration {#create-otel-configuration} Create a `custom-local-config.yaml` file with the following content: @@ -145,7 +145,7 @@ docker run -d \ The collector will immediately begin collecting local system logs and metrics. -## Navigate to the HyperDX UI {#navigate-to-the-hyperdx-ui} +## Navigate to the hyperdx UI {#navigate-to-the-hyperdx-ui} Visit [http://localhost:8080](http://localhost:8080) to access the HyperDX UI if deploying locally. If using HyperDX in ClickHouse Cloud, select your service and `HyperDX` from the left menu. diff --git a/docs/use-cases/observability/clickstack/example-datasets/remote-demo-data.md b/docs/use-cases/observability/clickstack/example-datasets/remote-demo-data.md index e7c55a6c8d7..43990f5b717 100644 --- a/docs/use-cases/observability/clickstack/example-datasets/remote-demo-data.md +++ b/docs/use-cases/observability/clickstack/example-datasets/remote-demo-data.md @@ -62,7 +62,7 @@ The customer support team has reported that users are experiencing issues comple Using HyperDX, the SRE team will analyze logs, traces, and metrics to diagnose and resolve the issue—then review session data to confirm whether their conclusions align with actual user behavior. -## Open Telemetry Demo {#otel-demo} +## Open telemetry demo {#otel-demo} This demo uses a [ClickStack maintained fork](https://github.com/ClickHouse/opentelemetry-demo) of the official OpenTelemetry demo. diff --git a/docs/use-cases/observability/clickstack/example-datasets/sample-data.md b/docs/use-cases/observability/clickstack/example-datasets/sample-data.md index 1c1cf271402..9adadb6b18e 100644 --- a/docs/use-cases/observability/clickstack/example-datasets/sample-data.md +++ b/docs/use-cases/observability/clickstack/example-datasets/sample-data.md @@ -31,7 +31,7 @@ import hyperdx_18 from '@site/static/images/use-cases/observability/hyperdx-18.p import hyperdx_19 from '@site/static/images/use-cases/observability/hyperdx-19.png'; import copy_api_key from '@site/static/images/use-cases/observability/copy_api_key.png'; -# ClickStack - Sample logs, traces and metrics {#clickstack-sample-dataset} +# ClickStack - sample logs, traces and metrics {#clickstack-sample-dataset} The following example assumes you have started ClickStack using the [instructions for the all-in-one image](/use-cases/observability/clickstack/getting-started) and connected to the [local ClickHouse instance](/use-cases/observability/clickstack/getting-started#complete-connection-credentials) or a [ClickHouse Cloud instance](/use-cases/observability/clickstack/getting-started#create-a-cloud-connection). @@ -41,7 +41,7 @@ This sample dataset can also be used with HyperDX in ClickHouse Cloud, with only -## Navigate to the HyperDX UI {#navigate-to-the-hyperdx-ui} +## Navigate to the hyperdx UI {#navigate-to-the-hyperdx-ui} Visit [http://localhost:8080](http://localhost:8080) to access the HyperDX UI if deploying locally. If using HyperDX in ClickHouse Cloud, select your service and `HyperDX` from the left menu. diff --git a/docs/use-cases/observability/clickstack/getting-started.md b/docs/use-cases/observability/clickstack/getting-started.md index 4f1b8aa6f47..39ca7b402b1 100644 --- a/docs/use-cases/observability/clickstack/getting-started.md +++ b/docs/use-cases/observability/clickstack/getting-started.md @@ -37,7 +37,7 @@ This all-in-one image allows you to launch the full stack with a single command, -### Deploy stack with docker {#deploy-stack-with-docker} +### Deploy stack with Docker {#deploy-stack-with-docker} The following will run an OpenTelemetry collector (on port 4317 and 4318) and the HyperDX UI (on port 8080). @@ -67,7 +67,7 @@ docker run \ ``` ::: -### Navigate to the HyperDX UI {#navigate-to-hyperdx-ui} +### Navigate to the hyperdx UI {#navigate-to-hyperdx-ui} Visit [http://localhost:8080](http://localhost:8080) to access the HyperDX UI. @@ -115,7 +115,7 @@ Copy the HTTP connection details, specifically the HTTPS endpoint (`endpoint`) a While we will use the `default` user to connect HyperDX, we recommend creating a dedicated user when [going to production](/use-cases/observability/clickstack/production#create-a-user). ::: -### Deploy with docker {#deploy-with-docker} +### Deploy with Docker {#deploy-with-docker} Open a terminal and export the credentials copied above: @@ -133,7 +133,7 @@ docker run -e CLICKHOUSE_ENDPOINT=${CLICKHOUSE_ENDPOINT} -e CLICKHOUSE_USER=defa This will expose an OpenTelemetry collector (on port 4317 and 4318), and the HyperDX UI (on port 8080). -### Navigate to the HyperDX UI {#navigate-to-hyperdx-ui-cloud} +### Navigate to the hyperdx UI {#navigate-to-hyperdx-ui-cloud} Visit [http://localhost:8080](http://localhost:8080) to access the HyperDX UI. @@ -176,7 +176,7 @@ You can use a hosted version of HyperDX in local mode available at [play.hyperdx -### Run with docker {#run-local-with-docker} +### Run with Docker {#run-local-with-docker} The self-hosted local mode image comes with an OpenTelemetry collector and a ClickHouse server pre-configured as well. This makes it easy to consume telemetry data from your applications and visualize it in HyperDX with minimal external setup. To get started with the self-hosted version, simply run the Docker container with the appropriate ports forwarded: diff --git a/docs/use-cases/observability/clickstack/host-logs/ec2-host-logs.md b/docs/use-cases/observability/clickstack/host-logs/ec2-host-logs.md index 943c59f2888..4dc07e3296f 100644 --- a/docs/use-cases/observability/clickstack/host-logs/ec2-host-logs.md +++ b/docs/use-cases/observability/clickstack/host-logs/ec2-host-logs.md @@ -22,7 +22,7 @@ import { TrackedLink } from '@site/src/components/GalaxyTrackedLink/GalaxyTracke import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -# Monitoring EC2 Host Logs with ClickStack {#ec2-host-logs-clickstack} +# Monitoring ec2 host logs with ClickStack {#ec2-host-logs-clickstack} :::note[TL;DR] Monitor EC2 system logs with ClickStack by installing OpenTelemetry Collector on your instances. The collector automatically enriches logs with EC2 metadata (instance ID, region, availability zone, instance type). You'll learn how to: @@ -37,7 +37,7 @@ A demo dataset with sample logs and simulated EC2 metadata is available for test Time Required: 10-15 minutes ::: -## Integration with existing EC2 instance {#existing-ec2} +## Integration with existing ec2 instance {#existing-ec2} This section covers installing OpenTelemetry Collector on your EC2 instances to collect system logs and send them to ClickStack with automatic EC2 metadata enrichment. This distributed architecture is production-ready and scales to multiple instances. @@ -55,7 +55,7 @@ If you would like to test the EC2 host logs integration before configuring your -#### Verify EC2 metadata is accessible {#verify-metadata} +#### Verify ec2 metadata is accessible {#verify-metadata} From your EC2 instance, verify the metadata service is accessible: ```bash @@ -93,7 +93,7 @@ tail -20 /var/log/syslog tail -20 /var/log/messages ``` -#### Install OpenTelemetry Collector {#install-collector} +#### Install opentelemetry collector {#install-collector} Install the OpenTelemetry Collector Contrib distribution on your EC2 instance: ```bash @@ -279,7 +279,7 @@ CLICKSTACK_API_KEY="your-api-key-here" /usr/local/bin/otelcol-contrib --config / Configure the collector to run as a systemd service so it starts automatically on boot and restarts on failure. See the [OpenTelemetry Collector documentation](https://opentelemetry.io/docs/collector/deployment/) for details. ::: -#### Verifying Logs in HyperDX {#verifying-logs} +#### Verifying logs in hyperdx {#verifying-logs} Once the collector is running, log into HyperDX and verify logs are flowing with EC2 metadata: @@ -405,7 +405,7 @@ docker run --name clickstack-demo \ docker.hyperdx.io/hyperdx/hyperdx-all-in-one:latest ``` -#### Verify logs in HyperDX {#verify-demo-logs} +#### Verify logs in hyperdx {#verify-demo-logs} Once the collector is running: @@ -436,7 +436,7 @@ To help you get started monitoring EC2 host logs with ClickStack, we provide ess -#### Download the dashboard configuration {#download} +#### Download the dashboard configuration {#download} #### Import the pre-built dashboard {#import-dashboard} @@ -468,7 +468,7 @@ For the demo dataset, set the time range to **2025-11-11 00:00:00 - 2025-11-12 0 ## Troubleshooting {#troubleshooting} -### EC2 metadata not appearing in logs {#no-metadata} +### Ec2 metadata not appearing in logs {#no-metadata} **Verify the EC2 metadata service is accessible:** ```bash @@ -492,7 +492,7 @@ sudo journalctl -u otelcol-contrib -f | grep -i "ec2\|metadata\|resourcedetectio # If running in foreground, check stdout ``` -### No logs appearing in HyperDX {#no-logs} +### No logs appearing in hyperdx {#no-logs} **Verify syslog files exist and are being written:** ```bash diff --git a/docs/use-cases/observability/clickstack/host-logs/generic-host-logs.md b/docs/use-cases/observability/clickstack/host-logs/generic-host-logs.md index 9e45b9ceebd..53a3b486ae5 100644 --- a/docs/use-cases/observability/clickstack/host-logs/generic-host-logs.md +++ b/docs/use-cases/observability/clickstack/host-logs/generic-host-logs.md @@ -20,7 +20,7 @@ import { TrackedLink } from '@site/src/components/GalaxyTrackedLink/GalaxyTracke import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -# Monitoring Host Logs with ClickStack {#host-logs-clickstack} +# Monitoring host logs with ClickStack {#host-logs-clickstack} :::note[TL;DR] This guide shows you how to monitor host system logs with ClickStack by configuring the OpenTelemetry collector to collect logs from systemd, kernel, SSH, cron, and other system services. You'll learn how to: @@ -67,7 +67,7 @@ Common syslog locations: - **RHEL/CentOS/Fedora**: `/var/log/messages` - **macOS**: `/var/log/system.log` -#### Create custom OTel collector configuration {#custom-otel} +#### Create custom otel collector configuration {#custom-otel} ClickStack allows you to extend the base OpenTelemetry Collector configuration by mounting a custom configuration file and setting an environment variable. @@ -220,7 +220,7 @@ To enable custom collector configuration in your existing ClickStack deployment, 2. Set the environment variable `CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml` 3. Mount your syslog directory so the collector can read them -##### Option 1: Docker Compose {#docker-compose} +##### Option 1: Docker compose {#docker-compose} Update your ClickStack deployment configuration: ```yaml @@ -236,7 +236,7 @@ services: # ... other volumes ... ``` -##### Option 2: Docker Run (All-in-One Image) {#all-in-one} +##### Option 2: Docker run (all-in-one image) {#all-in-one} If you're using the all-in-one image with docker run: ```bash @@ -252,7 +252,7 @@ docker run --name clickstack \ Ensure the ClickStack collector has appropriate permissions to read the syslog files. In production, use read-only mounts (`:ro`) and follow the principle of least privilege. ::: -#### Verifying Logs in HyperDX {#verifying-logs} +#### Verifying logs in hyperdx {#verifying-logs} Once configured, log into HyperDX and verify logs are flowing: @@ -348,7 +348,7 @@ docker run --name clickstack-demo \ **This mounts the log file directly into the container. This is done for testing purposes with static demo data.** ::: -#### Verify logs in HyperDX {#verify-demo-logs} +#### Verify logs in hyperdx {#verify-demo-logs} Once ClickStack is running: @@ -371,7 +371,7 @@ To help you get started monitoring host logs with ClickStack, we provide essenti -#### Download the dashboard configuration {#download} +#### Download the dashboard configuration {#download} #### Import the pre-built dashboard {#import-dashboard} @@ -418,7 +418,7 @@ Check the custom config file is mounted and readable: docker exec cat /etc/otelcol-contrib/custom.config.yaml | head -10 ``` -### No logs appearing in HyperDX {#no-logs} +### No logs appearing in hyperdx {#no-logs} **Verify syslog files exist and are being written:** ```bash diff --git a/docs/use-cases/observability/clickstack/ingesting-data/collector.md b/docs/use-cases/observability/clickstack/ingesting-data/collector.md index 713180f9ddd..83258559a64 100644 --- a/docs/use-cases/observability/clickstack/ingesting-data/collector.md +++ b/docs/use-cases/observability/clickstack/ingesting-data/collector.md @@ -57,7 +57,7 @@ Users should use a user with the [appropriate credentials](/use-cases/observabil ### Modifying configuration {#modifying-otel-collector-configuration} -#### Using docker {#using-docker} +#### Using Docker {#using-docker} All docker images, which include the OpenTelemetry collector, can be configured to use a clickhouse instance via the environment variables `OPAMP_SERVER_URL`,`CLICKHOUSE_ENDPOINT`, `CLICKHOUSE_USERNAME` and `CLICKHOUSE_PASSWORD`: @@ -78,7 +78,7 @@ docker run -e OPAMP_SERVER_URL=${OPAMP_SERVER_URL} -e CLICKHOUSE_ENDPOINT=${CLIC ClickStack images are now published as `clickhouse/clickstack-*` (previously `docker.hyperdx.io/hyperdx/*`). ::: -#### Docker Compose {#docker-compose-otel} +#### Docker compose {#docker-compose-otel} With Docker Compose, modify the collector configuration using the same environment variables as above: @@ -429,7 +429,7 @@ To migrate from the Map-based schemas, follow these steps: -#### Stop the OTel collector {#stop-the-collector} +#### Stop the otel collector {#stop-the-collector} #### Rename existing tables and update sources {#rename-existing-tables-sources} @@ -446,7 +446,7 @@ RENAME TABLE otel_metrics TO otel_metrics_map; Deploy the collector with `OTEL_AGENT_FEATURE_GATE_ARG` set. -#### Restart the HyperDX container with JSON schema support {#restart-the-hyperdx-container} +#### Restart the hyperdx container with JSON schema support {#restart-the-hyperdx-container} ```shell export BETA_CH_OTEL_JSON_SCHEMA_ENABLED=true diff --git a/docs/use-cases/observability/clickstack/ingesting-data/opentelemetry.md b/docs/use-cases/observability/clickstack/ingesting-data/opentelemetry.md index 64b9ec9079d..e790697c20e 100644 --- a/docs/use-cases/observability/clickstack/ingesting-data/opentelemetry.md +++ b/docs/use-cases/observability/clickstack/ingesting-data/opentelemetry.md @@ -15,7 +15,7 @@ All data is ingested into ClickStack via an **OpenTelemetry (OTel) collector** i Users send data to this collector from [language SDKs](/use-cases/observability/clickstack/sdks) or through data collection agents collecting infrastructure metrics and logs (such OTel collectors in an [agent](/use-cases/observability/clickstack/ingesting-data/otel-collector#collector-roles) role or other technologies e.g. [Fluentd](https://www.fluentd.org/) or [Vector](https://vector.dev/)). -## Installing ClickStack OpenTelemetry collector {#installing-otel-collector} +## Installing ClickStack opentelemetry collector {#installing-otel-collector} The ClickStack OpenTelemetry collector is included in most ClickStack distributions, including: @@ -38,7 +38,7 @@ This allows users to benefit from standardized ingestion, enforced schemas, and For further details see ["Deploying the collector"](/use-cases/observability/clickstack/ingesting-data/otel-collector). -## Sending OpenTelemetry data {#sending-otel-data} +## Sending opentelemetry data {#sending-otel-data} To send data to ClickStack, point your OpenTelemetry instrumentation to the following endpoints made available by the OpenTelemetry collector: diff --git a/docs/use-cases/observability/clickstack/ingesting-data/sdks/aws-lambda.md b/docs/use-cases/observability/clickstack/ingesting-data/sdks/aws-lambda.md index 4f83809a022..01a11e9a98c 100644 --- a/docs/use-cases/observability/clickstack/ingesting-data/sdks/aws-lambda.md +++ b/docs/use-cases/observability/clickstack/ingesting-data/sdks/aws-lambda.md @@ -24,7 +24,7 @@ import TabItem from '@theme/TabItem'; -## Installing the OpenTelemetry Lambda layers {#installing-the-otel-lambda-layers} +## Installing the opentelemetry lambda layers {#installing-the-otel-lambda-layers} The OpenTelemetry project provides separate lambda layers to: @@ -128,7 +128,7 @@ OTEL_TRACES_SAMPLER=always_on -### Installing the OpenTelemetry collector Lambda layer {#installing-the-otel-collector-layer} +### Installing the opentelemetry collector lambda layer {#installing-the-otel-collector-layer} The collector Lambda layer allows you to forward logs, metrics, and traces from your Lambda function to ClickStack without impacting response times due to exporter latency. diff --git a/docs/use-cases/observability/clickstack/ingesting-data/sdks/browser.md b/docs/use-cases/observability/clickstack/ingesting-data/sdks/browser.md index 7fbb332d91c..1dc3c0b496f 100644 --- a/docs/use-cases/observability/clickstack/ingesting-data/sdks/browser.md +++ b/docs/use-cases/observability/clickstack/ingesting-data/sdks/browser.md @@ -126,7 +126,7 @@ HyperDX.setGlobalAttributes({ }); ``` -### Auto capture React error boundary errors {#auto-capture-react-error-boundary-errors} +### Auto capture react error boundary errors {#auto-capture-react-error-boundary-errors} If you're using React, you can automatically capture errors that occur within React error boundaries by passing your error boundary component @@ -165,7 +165,7 @@ To enable or disable network capture dynamically, simply invoke the `enableAdvan HyperDX.enableAdvancedNetworkCapture(); ``` -### Enable resource timing for CORS requests {#enable-resource-timing-for-cors-requests} +### Enable resource timing for cors requests {#enable-resource-timing-for-cors-requests} If your frontend application makes API requests to a different domain, you can optionally enable the `Timing-Allow-Origin`[header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Timing-Allow-Origin) to be sent with the request. This will allow ClickStack to capture fine-grained diff --git a/docs/use-cases/observability/clickstack/ingesting-data/sdks/golang.md b/docs/use-cases/observability/clickstack/ingesting-data/sdks/golang.md index 80ea82d5eba..70056ae0f4e 100644 --- a/docs/use-cases/observability/clickstack/ingesting-data/sdks/golang.md +++ b/docs/use-cases/observability/clickstack/ingesting-data/sdks/golang.md @@ -27,7 +27,7 @@ instrumentation isn't required to get value out of tracing. ## Getting started {#getting-started} -### Install OpenTelemetry instrumentation packages {#install-opentelemetry} +### Install opentelemetry instrumentation packages {#install-opentelemetry} To install the OpenTelemetry and HyperDX Go packages, use the command below. It is recommended to check out the [current instrumentation packages](https://github.com/open-telemetry/opentelemetry-go-contrib/tree/v1.4.0/instrumentation#instrumentation-packages) and install the necessary packages to ensure that the trace information is attached correctly. diff --git a/docs/use-cases/observability/clickstack/ingesting-data/sdks/java.md b/docs/use-cases/observability/clickstack/ingesting-data/sdks/java.md index acc8db26d54..c43b855298e 100644 --- a/docs/use-cases/observability/clickstack/ingesting-data/sdks/java.md +++ b/docs/use-cases/observability/clickstack/ingesting-data/sdks/java.md @@ -31,7 +31,7 @@ instrumentation isn't required to get value out of tracing. At present, the integration is compatible exclusively with **Java 8+** ::: -### Download OpenTelemetry Java agent {#download-opentelemtry-java-agent} +### Download opentelemetry Java agent {#download-opentelemtry-java-agent} Download [`opentelemetry-javaagent.jar`](https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar) and place the JAR in your preferred directory. The JAR file contains the agent @@ -59,7 +59,7 @@ _The `OTEL_SERVICE_NAME` environment variable is used to identify your service i The `OTEL_EXPORTER_OTLP_HEADERS` environment variable contains the API Key available via HyperDX app in `Team Settings → API Keys`. -### Run the application with OpenTelemetry Java agent {#run-the-application-with-otel-java-agent} +### Run the application with opentelemetry Java agent {#run-the-application-with-otel-java-agent} ```shell java -jar target/ diff --git a/docs/use-cases/observability/clickstack/ingesting-data/sdks/nestjs.md b/docs/use-cases/observability/clickstack/ingesting-data/sdks/nestjs.md index 52421aab3e7..69204cd864c 100644 --- a/docs/use-cases/observability/clickstack/ingesting-data/sdks/nestjs.md +++ b/docs/use-cases/observability/clickstack/ingesting-data/sdks/nestjs.md @@ -68,7 +68,7 @@ export class CatsController { } ``` -### Replacing the Nest logger (also for bootstrapping) {#replacing-the-nest-logger} +### Replacing the nest logger (also for bootstrapping) {#replacing-the-nest-logger} :::note Important By doing this, you give up the dependency injection, meaning that `forRoot` and `forRootAsync` are not needed and shouldn't be used. Remove them from your main module. diff --git a/docs/use-cases/observability/clickstack/ingesting-data/sdks/nextjs.md b/docs/use-cases/observability/clickstack/ingesting-data/sdks/nextjs.md index b4be3eb5499..37194241817 100644 --- a/docs/use-cases/observability/clickstack/ingesting-data/sdks/nextjs.md +++ b/docs/use-cases/observability/clickstack/ingesting-data/sdks/nextjs.md @@ -54,7 +54,7 @@ const nextConfig = { module.exports = nextConfig; ``` -### Install ClickHouse OpenTelemetry SDK {#install-sdk} +### Install ClickHouse opentelemetry SDK {#install-sdk} diff --git a/docs/use-cases/observability/clickstack/ingesting-data/sdks/nodejs.md b/docs/use-cases/observability/clickstack/ingesting-data/sdks/nodejs.md index d8b6467c86f..fd3c460f7c3 100644 --- a/docs/use-cases/observability/clickstack/ingesting-data/sdks/nodejs.md +++ b/docs/use-cases/observability/clickstack/ingesting-data/sdks/nodejs.md @@ -25,7 +25,7 @@ This guide integrates: ## Getting started {#getting-started} -### Install HyperDX OpenTelemetry instrumentation package {#install-hyperdx-opentelemetry-instrumentation-package} +### Install hyperdx opentelemetry instrumentation package {#install-hyperdx-opentelemetry-instrumentation-package} Use the following command to install the [ClickStack OpenTelemetry package](https://www.npmjs.com/package/@hyperdx/node-opentelemetry). @@ -266,7 +266,7 @@ enable trace attributes. export HDX_NODE_BETA_MODE=1 ``` -### Google Cloud Run {#google-cloud-run} +### Google cloud run {#google-cloud-run} If you're running your application on Google Cloud Run, Cloud Trace automatically injects sampling headers into incoming requests, currently @@ -307,7 +307,7 @@ The following libraries will be automatically instrumented (traced) by the SDK: ## Alternative installation {#alternative-installation} -### Run the Application with ClickStack OpenTelemetry CLI {#run-the-application-with-cli} +### Run the application with ClickStack opentelemetry CLI {#run-the-application-with-cli} Alternatively, you can auto-instrument your application without any code changes by using the `opentelemetry-instrument` CLI or using the Node.js `--require` flag. The CLI installation exposes a wider range of auto-instrumented libraries and frameworks. diff --git a/docs/use-cases/observability/clickstack/ingesting-data/sdks/python.md b/docs/use-cases/observability/clickstack/ingesting-data/sdks/python.md index 4b6cef8d0d5..90724370d95 100644 --- a/docs/use-cases/observability/clickstack/ingesting-data/sdks/python.md +++ b/docs/use-cases/observability/clickstack/ingesting-data/sdks/python.md @@ -24,7 +24,7 @@ This guide integrates: ## Getting started {#getting-started} -### Install ClickStack OpenTelemetry instrumentation package {#install-clickstack-otel-instrumentation-package} +### Install ClickStack opentelemetry instrumentation package {#install-clickstack-otel-instrumentation-package} Use the following command to install the [ClickStack OpenTelemetry package](https://pypi.org/project/hyperdx-opentelemetry/). @@ -51,7 +51,7 @@ OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 _The `OTEL_SERVICE_NAME` environment variable is used to identify your service in the HyperDX app, it can be any name you want._ -### Run the application with OpenTelemetry Python agent {#run-the-application-with-otel-python-agent} +### Run the application with opentelemetry Python agent {#run-the-application-with-otel-python-agent} Now you can run the application with the OpenTelemetry Python agent (`opentelemetry-instrument`). diff --git a/docs/use-cases/observability/clickstack/ingesting-data/sdks/react-native.md b/docs/use-cases/observability/clickstack/ingesting-data/sdks/react-native.md index 8d22bb974d6..7facc534198 100644 --- a/docs/use-cases/observability/clickstack/ingesting-data/sdks/react-native.md +++ b/docs/use-cases/observability/clickstack/ingesting-data/sdks/react-native.md @@ -41,7 +41,7 @@ HyperDXRum.init({ }); ``` -### Attach user information or metadata (Optional) {#attach-user-information-metadata} +### Attach user information or metadata (optional) {#attach-user-information-metadata} Attaching user information will allow you to search/filter sessions and events in HyperDX. This can be called at any point during the client session. The diff --git a/docs/use-cases/observability/clickstack/ingesting-data/sdks/ruby.md b/docs/use-cases/observability/clickstack/ingesting-data/sdks/ruby.md index abaf5f7d559..42125416ce9 100644 --- a/docs/use-cases/observability/clickstack/ingesting-data/sdks/ruby.md +++ b/docs/use-cases/observability/clickstack/ingesting-data/sdks/ruby.md @@ -25,7 +25,7 @@ _To send logs to ClickStack, please send logs via the [OpenTelemetry collector]( ## Getting started {#getting-started} -### Install OpenTelemetry packages {#install-otel-packages} +### Install opentelemetry packages {#install-otel-packages} Use the following command to install the OpenTelemetry package. @@ -33,7 +33,7 @@ Use the following command to install the OpenTelemetry package. bundle add opentelemetry-sdk opentelemetry-instrumentation-all opentelemetry-exporter-otlp ``` -### Configure OpenTelemetry + logger formatter {#configure-otel-logger-formatter} +### Configure opentelemetry + logger formatter {#configure-otel-logger-formatter} Next, you'll need to initialize the OpenTelemetry tracing instrumentation and configure the log message formatter for Rails logger so that logs can be diff --git a/docs/use-cases/observability/clickstack/integration-examples/kafka-metrics.md b/docs/use-cases/observability/clickstack/integration-examples/kafka-metrics.md index 1ac02036ad5..35fad632fec 100644 --- a/docs/use-cases/observability/clickstack/integration-examples/kafka-metrics.md +++ b/docs/use-cases/observability/clickstack/integration-examples/kafka-metrics.md @@ -17,7 +17,7 @@ import finish_import from '@site/static/images/clickstack/kafka/import-kafka-das import example_dashboard from '@site/static/images/clickstack/kafka/kafka-metrics-dashboard.png'; import { TrackedLink } from '@site/src/components/GalaxyTrackedLink/GalaxyTrackedLink'; -# Monitoring Kafka Metrics with ClickStack {#kafka-metrics-clickstack} +# Monitoring Kafka metrics with ClickStack {#kafka-metrics-clickstack} :::note[TL;DR] This guide shows you how to monitor Apache Kafka performance metrics with ClickStack by using the OpenTelemetry JMX Metric Gatherer. You'll learn how to: @@ -61,7 +61,7 @@ The JMX Metric Gatherer sends data to ClickStack's OTLP endpoint, which requires export CLICKSTACK_API_KEY=your-api-key-here ``` -#### Download the OpenTelemetry JMX Metric Gatherer {#download-jmx} +#### Download the opentelemetry jmx metric gatherer {#download-jmx} Download the JMX Metric Gatherer JAR: ```bash @@ -69,7 +69,7 @@ curl -L -o opentelemetry-jmx-metrics.jar \ https://github.com/open-telemetry/opentelemetry-java-contrib/releases/download/v1.32.0/opentelemetry-jmx-metrics.jar ``` -#### Verify Kafka JMX is enabled {#verify-jmx} +#### Verify Kafka jmx is enabled {#verify-jmx} Ensure JMX is enabled on your Kafka brokers. For Docker deployments: ```yaml @@ -95,7 +95,7 @@ Verify JMX is accessible: netstat -an | grep 9999 ``` -#### Deploy JMX Metric Gatherer with Docker Compose {#deploy-jmx} +#### Deploy jmx metric gatherer with Docker compose {#deploy-jmx} This example shows a complete setup with Kafka, the JMX Metric Gatherer, and ClickStack. Adjust service names and endpoints to match your existing deployment: ```yaml @@ -172,7 +172,7 @@ networks: - `service.name=kafka,kafka.broker.id=broker-0` - Resource attributes for filtering - `10000` - Collection interval in milliseconds (10 seconds) -#### Verify metrics in HyperDX {#verify-metrics} +#### Verify metrics in hyperdx {#verify-metrics} Log into HyperDX and confirm metrics are flowing: @@ -250,7 +250,7 @@ cat kafka-metrics-sum.csv | docker exec -i clickstack-demo \ clickhouse-client --query "INSERT INTO otel_metrics_sum FORMAT CSVWithNames" ``` -#### Verify metrics in HyperDX {#verify-demo-metrics} +#### Verify metrics in hyperdx {#verify-demo-metrics} Once loaded, the quickest way to see your metrics is through the pre-built dashboard. @@ -268,7 +268,7 @@ To help you get started monitoring Kafka with ClickStack, we provide essential v -#### Download the dashboard configuration {#download} +#### Download the dashboard configuration {#download} #### Import the pre-built dashboard {#import-dashboard} @@ -295,7 +295,7 @@ For the demo dataset, set the time range to **2025-11-05 16:00:00 - 2025-11-06 1 ## Troubleshooting {#troubleshooting} -#### No metrics appearing in HyperDX {#no-metrics} +#### No metrics appearing in hyperdx {#no-metrics} **Verify API key is set and passed to the container:** diff --git a/docs/use-cases/observability/clickstack/integration-examples/kubernetes.md b/docs/use-cases/observability/clickstack/integration-examples/kubernetes.md index 350f2d931c8..25110bb55b4 100644 --- a/docs/use-cases/observability/clickstack/integration-examples/kubernetes.md +++ b/docs/use-cases/observability/clickstack/integration-examples/kubernetes.md @@ -21,7 +21,7 @@ To send over application-level metrics or APM/traces, you'll need to add the cor The following guide assumes you have deployed a [ClickStack OTel collector as a gateway](/use-cases/observability/clickstack/ingesting-data/otel-collector), secured with an ingestion API key. -## Creating the OTel Helm chart configuration files {#creating-the-otel-helm-chart-config-files} +## Creating the otel Helm chart configuration files {#creating-the-otel-helm-chart-config-files} To collect logs and metrics from both each node and the cluster itself, we'll need to deploy two separate OpenTelemetry collectors. One will be deployed as a DaemonSet to collect logs and metrics from each node, and the other will be deployed as a deployment to collect logs and metrics from the cluster itself. @@ -41,7 +41,7 @@ kubectl create configmap -n=otel-demo otel-config-vars --from-literal=YOUR_OTEL_ # e.g. kubectl create configmap -n=otel-demo otel-config-vars --from-literal=YOUR_OTEL_COLLECTOR_ENDPOINT=http://my-hyperdx-hdx-oss-v2-otel-collector:4318 ``` -### Creating the DaemonSet configuration {#creating-the-daemonset-configuration} +### Creating the daemonset configuration {#creating-the-daemonset-configuration} The DaemonSet will collect logs and metrics from each node in the cluster but will not collect Kubernetes events or cluster-wide metrics. @@ -231,7 +231,7 @@ config: -## Deploying the OpenTelemetry collector {#deploying-the-otel-collector} +## Deploying the opentelemetry collector {#deploying-the-otel-collector} The OpenTelemetry collector can now be deployed in your Kubernetes cluster using the [OpenTelemetry Helm Chart](https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-collector). @@ -252,7 +252,7 @@ helm install my-opentelemetry-collector-daemonset open-telemetry/opentelemetry-c Now the metrics, logs and Kubernetes events from your Kubernetes cluster should now appear inside HyperDX. -## Forwarding resource tags to pods (Recommended) {#forwarding-resouce-tags-to-pods} +## Forwarding resource tags to pods (recommended) {#forwarding-resouce-tags-to-pods} To correlate application-level logs, metrics, and traces with Kubernetes metadata (ex. pod name, namespace, etc.), you'll want to forward the Kubernetes metadata diff --git a/docs/use-cases/observability/clickstack/integration-examples/nginx-logs.md b/docs/use-cases/observability/clickstack/integration-examples/nginx-logs.md index 29d69a44d51..67dc23fc707 100644 --- a/docs/use-cases/observability/clickstack/integration-examples/nginx-logs.md +++ b/docs/use-cases/observability/clickstack/integration-examples/nginx-logs.md @@ -17,7 +17,7 @@ import log_view from '@site/static/images/clickstack/log-view.png'; import search_view from '@site/static/images/clickstack/nginx-logs-search-view.png'; import { TrackedLink } from '@site/src/components/GalaxyTrackedLink/GalaxyTrackedLink'; -# Monitoring Nginx Logs with ClickStack {#nginx-clickstack} +# Monitoring nginx logs with ClickStack {#nginx-clickstack} :::note[TL;DR] This guide shows you how to monitor Nginx with ClickStack by configuring the OpenTelemetry collector to ingest Nginx access logs. You'll learn how to: @@ -32,7 +32,7 @@ A demo dataset with sample logs is available if you want to test the integration Time Required: 5-10 minutes ::: -## Integration with existing Nginx {#existing-nginx} +## Integration with existing nginx {#existing-nginx} This section covers configuring your existing Nginx installation to send logs to ClickStack by modifying the ClickStack OTel collector configuration. If you would like to test the integration before configuring your own existing setup, you can test with our preconfigured setup and sample data in the [following section](/use-cases/observability/clickstack/integrations/nginx#demo-dataset). @@ -44,7 +44,7 @@ If you would like to test the integration before configuring your own existing s -#### Configure Nginx log format {#configure-nginx} +#### Configure nginx log format {#configure-nginx} First, configure Nginx to output logs in JSON format for easier parsing. Add this log format definition to your nginx.conf: The `nginx.conf` file is typically located at: @@ -77,7 +77,7 @@ http { After making this change, reload Nginx. -#### Create custom OTel collector configuration {#custom-otel} +#### Create custom otel collector configuration {#custom-otel} ClickStack allows you to extend the base OpenTelemetry Collector configuration by mounting a custom configuration file and setting an environment variable. The custom configuration is merged with the base configuration managed by HyperDX via OpAMP. @@ -135,7 +135,7 @@ To enable custom collector configuration in your existing ClickStack deployment, 2. Set the environment variable CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml 3. Mount your Nginx log directories so the collector can read them -##### Option 1: Docker Compose {#docker-compose} +##### Option 1: Docker compose {#docker-compose} Update your ClickStack deployment configuration: ```yaml @@ -151,7 +151,7 @@ services: # ... other volumes ... ``` -##### Option 2: Docker Run (All-in-One Image) {#all-in-one} +##### Option 2: Docker run (all-in-one image) {#all-in-one} If using the all-in-one image with docker run: ```bash @@ -167,7 +167,7 @@ docker run --name clickstack \ Ensure the ClickStack collector has appropriate permissions to read the nginx log files. In production, use read-only mounts (:ro) and follow the principle of least privilege. ::: -#### Verifying Logs in HyperDX {#verifying-logs} +#### Verifying logs in hyperdx {#verifying-logs} Once configured, log into HyperDX and verify logs are flowing: 1. Navigate to the search view @@ -248,7 +248,7 @@ docker run --name clickstack-demo \ clickhouse/clickstack-all-in-one:latest ``` -#### Verify logs in HyperDX {#verify-demo-logs} +#### Verify logs in hyperdx {#verify-demo-logs} Once ClickStack is running: @@ -274,7 +274,7 @@ To help you get started monitoring nginx with ClickStack, we provide essential v -#### Download the dashboard configuration {#download} +#### Download the dashboard configuration {#download} #### Import the pre-built dashboard {#import-dashboard} 1. Open HyperDX and navigate to the Dashboards section. @@ -318,7 +318,7 @@ docker exec ls -lh /etc/otelcol-contrib/custom.config.yaml docker exec cat /etc/otelcol-contrib/custom.config.yaml ``` -### No logs appearing in HyperDX {#no-logs} +### No logs appearing in hyperdx {#no-logs} - Ensure nginx is writing JSON logs ```bash diff --git a/docs/use-cases/observability/clickstack/integration-examples/nginx-traces.md b/docs/use-cases/observability/clickstack/integration-examples/nginx-traces.md index b991ca41436..53559174148 100644 --- a/docs/use-cases/observability/clickstack/integration-examples/nginx-traces.md +++ b/docs/use-cases/observability/clickstack/integration-examples/nginx-traces.md @@ -17,7 +17,7 @@ import example_dashboard from '@site/static/images/clickstack/nginx-traces-dashb import view_traces from '@site/static/images/clickstack/nginx-traces-search-view.png'; import { TrackedLink } from '@site/src/components/GalaxyTrackedLink/GalaxyTrackedLink'; -# Monitoring Nginx Traces with ClickStack {#nginx-traces-clickstack} +# Monitoring nginx traces with ClickStack {#nginx-traces-clickstack} :::note[TL;DR] This guide shows you how to capture distributed traces from your existing Nginx installation and visualize them in ClickStack. You'll learn how to: @@ -32,7 +32,7 @@ A demo dataset with sample traces is available if you want to test the integrati Time Required: 5-10 minutes :::: -## Integration with existing Nginx {#existing-nginx} +## Integration with existing nginx {#existing-nginx} This section covers adding distributed tracing to your existing Nginx installation by installing the OpenTelemetry module and configuring it to send traces to ClickStack. If you would like to test the integration before configuring your own existing setup, you can test with our preconfigured setup and sample data in the [following section](/use-cases/observability/clickstack/integrations/nginx-traces#demo-dataset). @@ -45,7 +45,7 @@ If you would like to test the integration before configuring your own existing s -#### Install OpenTelemetry Nginx module {#install-module} +#### Install opentelemetry nginx module {#install-module} The easiest way to add tracing to Nginx is using the official Nginx image with OpenTelemetry support built-in. @@ -64,7 +64,7 @@ This image includes the `ngx_otel_module.so` pre-installed and ready to use. If you're running Nginx outside of Docker, refer to the [OpenTelemetry Nginx documentation](https://github.com/open-telemetry/opentelemetry-cpp-contrib/tree/main/instrumentation/nginx) for manual installation instructions. ::: -#### Configure Nginx to send traces to ClickStack {#configure-nginx} +#### Configure nginx to send traces to ClickStack {#configure-nginx} Add OpenTelemetry configuration to your `nginx.conf` file. The configuration loads the module and directs traces to ClickStack's OTLP endpoint. @@ -162,7 +162,7 @@ docker-compose restart nginx sudo systemctl reload nginx ``` -#### Verifying traces in HyperDX {#verifying-traces} +#### Verifying traces in hyperdx {#verifying-traces} Once configured, log into HyperDX and verify traces are flowing, you should see something like this, if you don't see traces, try adjusting your time range: @@ -236,7 +236,7 @@ This demo assumes ClickStack is running locally on `localhost:4318`. For remote You should see a response like `{"partialSuccess":{}}` indicating the traces were successfully sent. All 1,000 traces will be ingested into ClickStack. -#### Verify traces in HyperDX {#verify-demo-traces} +#### Verify traces in hyperdx {#verify-demo-traces} 1. Open [HyperDX](http://localhost:8080/) and log in to your account (you may need to create an account first) 2. Navigate to the Search view and set the source to `Traces` @@ -258,7 +258,7 @@ To help you get started monitoring traces with ClickStack, we provide essential -#### Download the dashboard configuration {#download} +#### Download the dashboard configuration {#download} #### Import the pre-built dashboard {#import-dashboard} 1. Open HyperDX and navigate to the Dashboards section. @@ -282,7 +282,7 @@ For the demo dataset, set the time range to **2025-10-26 13:00:00 - 2025-10-27 1 ## Troubleshooting {#troubleshooting} -### No traces appearing in HyperDX {#no-traces} +### No traces appearing in hyperdx {#no-traces} **Verify nginx module is loaded:** ```bash diff --git a/docs/use-cases/observability/clickstack/integration-examples/postgres-logs.md b/docs/use-cases/observability/clickstack/integration-examples/postgres-logs.md index 2cfa2364fb4..ae924e28bff 100644 --- a/docs/use-cases/observability/clickstack/integration-examples/postgres-logs.md +++ b/docs/use-cases/observability/clickstack/integration-examples/postgres-logs.md @@ -18,7 +18,7 @@ import logs_dashboard from '@site/static/images/clickstack/postgres/postgres-log import finish_import from '@site/static/images/clickstack/postgres/import-logs-dashboard.png'; import { TrackedLink } from '@site/src/components/GalaxyTrackedLink/GalaxyTrackedLink'; -# Monitoring PostgreSQL Logs with ClickStack {#postgres-logs-clickstack} +# Monitoring PostgreSQL logs with ClickStack {#postgres-logs-clickstack} :::note[TL;DR] This guide shows you how to monitor PostgreSQL with ClickStack by configuring the OpenTelemetry collector to ingest PostgreSQL server logs. You'll learn how to: @@ -98,7 +98,7 @@ tail -f /var/lib/postgresql/{version}/main/log/postgresql-*.log tail -f /usr/local/var/postgres/log/postgresql-*.log ``` -#### Create custom OTel collector configuration {#custom-otel} +#### Create custom otel collector configuration {#custom-otel} ClickStack allows you to extend the base OpenTelemetry Collector configuration by mounting a custom configuration file and setting an environment variable. The custom configuration is merged with the base configuration managed by HyperDX via OpAMP. @@ -167,7 +167,7 @@ To enable custom collector configuration in your existing ClickStack deployment, 2. Set the environment variable `CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml` 3. Mount your PostgreSQL log directory so the collector can read them -##### Option 1: Docker Compose {#docker-compose} +##### Option 1: Docker compose {#docker-compose} Update your ClickStack deployment configuration: ```yaml @@ -183,7 +183,7 @@ services: # ... other volumes ... ``` -##### Option 2: Docker Run (All-in-One Image) {#all-in-one} +##### Option 2: Docker run (all-in-one image) {#all-in-one} If you're using the all-in-one image with docker run: ```bash @@ -199,7 +199,7 @@ docker run --name clickstack \ Ensure the ClickStack collector has appropriate permissions to read the PostgreSQL log files. In production, use read-only mounts (`:ro`) and follow the principle of least privilege. ::: -#### Verifying Logs in HyperDX {#verifying-logs} +#### Verifying logs in hyperdx {#verifying-logs} Once configured, log into HyperDX and verify logs are flowing: @@ -286,7 +286,7 @@ docker run --name clickstack-demo \ clickhouse/clickstack-all-in-one:latest ``` -#### Verify logs in HyperDX {#verify-demo-logs} +#### Verify logs in hyperdx {#verify-demo-logs} Once ClickStack is running: @@ -310,7 +310,7 @@ To help you get started monitoring PostgreSQL with ClickStack, we provide essent -#### Download the dashboard configuration {#download} +#### Download the dashboard configuration {#download} #### Import the pre-built dashboard {#import-dashboard} @@ -349,7 +349,7 @@ Check the custom config file is mounted and readable: docker exec cat /etc/otelcol-contrib/custom.config.yaml | head -10 ``` -### No logs appearing in HyperDX {#no-logs} +### No logs appearing in hyperdx {#no-logs} Check the effective config includes your filelog receiver: ```bash diff --git a/docs/use-cases/observability/clickstack/integration-examples/postgres-metrics.md b/docs/use-cases/observability/clickstack/integration-examples/postgres-metrics.md index bc43edd703f..40c5e7dd654 100644 --- a/docs/use-cases/observability/clickstack/integration-examples/postgres-metrics.md +++ b/docs/use-cases/observability/clickstack/integration-examples/postgres-metrics.md @@ -16,7 +16,7 @@ import finish_import from '@site/static/images/clickstack/postgres/import-dashbo import example_dashboard from '@site/static/images/clickstack/postgres/postgres-metrics-dashboard.png'; import { TrackedLink } from '@site/src/components/GalaxyTrackedLink/GalaxyTrackedLink'; -# Monitoring PostgreSQL Metrics with ClickStack {#postgres-metrics-clickstack} +# Monitoring PostgreSQL metrics with ClickStack {#postgres-metrics-clickstack} :::note[TL;DR] This guide shows you how to monitor PostgreSQL performance metrics with ClickStack by configuring the OpenTelemetry collector's PostgreSQL receiver. You'll learn how to: @@ -52,7 +52,7 @@ The PostgreSQL receiver requires a user with read access to statistics views. Gr GRANT pg_monitor TO your_monitoring_user; ``` -#### Create custom OTel collector configuration {#create-custom-config} +#### Create custom otel collector configuration {#create-custom-config} ClickStack allows you to extend the base OpenTelemetry collector configuration by mounting a custom configuration file and setting an environment variable. @@ -180,7 +180,7 @@ cat postgres-metrics-sum.csv | docker exec -i clickstack-postgres-demo \ clickhouse-client --query "INSERT INTO otel_metrics_sum FORMAT CSVWithNames" ``` -#### Verify metrics in HyperDX {#verify-metrics-demo} +#### Verify metrics in hyperdx {#verify-metrics-demo} Once loaded, the quickest way to see your metrics is through the pre-built dashboard. @@ -198,7 +198,7 @@ To help you get started monitoring PostgreSQL with ClickStack, we provide essent -#### Download the dashboard configuration {#download} +#### Download the dashboard configuration {#download} #### Import the pre-built dashboard {#import-dashboard} @@ -239,7 +239,7 @@ Check the custom config file is mounted: docker exec cat /etc/otelcol-contrib/custom.config.yaml ``` -### No metrics appearing in HyperDX {#no-metrics} +### No metrics appearing in hyperdx {#no-metrics} Verify PostgreSQL is accessible: diff --git a/docs/use-cases/observability/clickstack/integration-examples/redis-logs.md b/docs/use-cases/observability/clickstack/integration-examples/redis-logs.md index 5cb5823c728..0668c9fa681 100644 --- a/docs/use-cases/observability/clickstack/integration-examples/redis-logs.md +++ b/docs/use-cases/observability/clickstack/integration-examples/redis-logs.md @@ -17,7 +17,7 @@ import log_view from '@site/static/images/clickstack/redis/redis-log-view.png'; import log from '@site/static/images/clickstack/redis/redis-log.png'; import { TrackedLink } from '@site/src/components/GalaxyTrackedLink/GalaxyTrackedLink'; -# Monitoring Redis Logs with ClickStack {#redis-clickstack} +# Monitoring Redis logs with ClickStack {#redis-clickstack} :::note[TL;DR] This guide shows you how to monitor Redis with ClickStack by configuring the OpenTelemetry collector to ingest Redis server logs. You'll learn how to: @@ -76,7 +76,7 @@ sudo systemctl restart redis docker restart ``` -#### Create custom OTel collector configuration {#custom-otel} +#### Create custom otel collector configuration {#custom-otel} ClickStack allows you to extend the base OpenTelemetry Collector configuration by mounting a custom configuration file and setting an environment variable. The custom configuration is merged with the base configuration managed by HyperDX via OpAMP. @@ -138,7 +138,7 @@ To enable custom collector configuration in your existing ClickStack deployment, 2. Set the environment variable `CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml` 3. Mount your Redis log directory so the collector can read them -##### Option 1: Docker Compose {#docker-compose} +##### Option 1: Docker compose {#docker-compose} Update your ClickStack deployment configuration: ```yaml @@ -154,7 +154,7 @@ services: # ... other volumes ... ``` -##### Option 2: Docker Run (All-in-One Image) {#all-in-one} +##### Option 2: Docker run (all-in-one image) {#all-in-one} If you're using the all-in-one image with docker, run: ```bash @@ -170,7 +170,7 @@ docker run --name clickstack \ Ensure the ClickStack collector has appropriate permissions to read the Redis log files. In production, use read-only mounts (`:ro`) and follow the principle of least privilege. ::: -#### Verifying Logs in HyperDX {#verifying-logs} +#### Verifying logs in hyperdx {#verifying-logs} Once configured, log into HyperDX and verify that logs are flowing: @@ -253,7 +253,7 @@ docker run --name clickstack-demo \ **This mounts the log file directly into the container. This is done for testing purposes with static demo data.** ::: -## Verify logs in HyperDX {#verify-demo-logs} +## Verify logs in hyperdx {#verify-demo-logs} Once ClickStack is running: @@ -277,9 +277,9 @@ To help you get started monitoring Redis with ClickStack, we provide essential v -#### Download the dashboard configuration {#download} +#### Download the dashboard configuration {#download} -#### Import Pre-built Dashboard {#import-dashboard} +#### Import pre-built dashboard {#import-dashboard} 1. Open HyperDX and navigate to the Dashboards section. 2. Click "Import Dashboard" in the upper right corner under the ellipses. @@ -328,7 +328,7 @@ docker exec cat /etc/otel/supervisor-data/effective.yaml | grep -A 1 # Should show your filelog/redis receiver configuration ``` -### No logs appearing in HyperDX {#no-logs} +### No logs appearing in hyperdx {#no-logs} **Ensure Redis is writing logs to a file:** ```bash @@ -375,7 +375,7 @@ If your Redis Logs have a different format, you may need to adjust the regex pat - `pid:role timestamp level message` - Example: `12345:M 28 Oct 2024 14:23:45.123 * Server started` -## Next Steps {#next-steps} +## Next steps {#next-steps} If you want to explore further, here are some next steps to experiment with your dashboard diff --git a/docs/use-cases/observability/clickstack/integration-examples/redis-metrics.md b/docs/use-cases/observability/clickstack/integration-examples/redis-metrics.md index e96050a4da9..edce246c08d 100644 --- a/docs/use-cases/observability/clickstack/integration-examples/redis-metrics.md +++ b/docs/use-cases/observability/clickstack/integration-examples/redis-metrics.md @@ -16,7 +16,7 @@ import finish_import from '@site/static/images/clickstack/import-redis-metrics-d import example_dashboard from '@site/static/images/clickstack/redis-metrics-dashboard.png'; import { TrackedLink } from '@site/src/components/GalaxyTrackedLink/GalaxyTrackedLink'; -# Monitoring Redis Metrics with ClickStack {#redis-metrics-clickstack} +# Monitoring Redis metrics with ClickStack {#redis-metrics-clickstack} :::note[TL;DR] This guide shows you how to monitor Redis performance metrics with ClickStack by configuring the OpenTelemetry collector's Redis receiver. You'll learn how to: @@ -67,7 +67,7 @@ redis-cli -a ping - **Docker**: Use container name or service name (e.g., `redis:6379`) - **Remote**: `:6379` -#### Create custom OTel collector configuration {#custom-otel} +#### Create custom otel collector configuration {#custom-otel} ClickStack allows you to extend the base OpenTelemetry collector configuration by mounting a custom configuration file and setting an environment variable. The custom configuration is merged with the base configuration managed by HyperDX via OpAMP. @@ -153,7 +153,7 @@ To enable custom collector configuration in your existing ClickStack deployment, 2. Set the environment variable `CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml` 3. Ensure network connectivity between ClickStack and Redis -##### Option 1: Docker Compose {#docker-compose} +##### Option 1: Docker compose {#docker-compose} Update your ClickStack deployment configuration: ```yaml @@ -208,7 +208,7 @@ docker run --name clickstack \ clickhouse/clickstack-all-in-one:latest ``` -#### Verify metrics in HyperDX {#verifying-metrics} +#### Verify metrics in hyperdx {#verifying-metrics} Once configured, log into HyperDX and verify metrics are flowing: @@ -267,7 +267,7 @@ cat redis-metrics-sum.csv | docker exec -i clickstack-demo \ clickhouse-client --query "INSERT INTO otel_metrics_sum FORMAT CSVWithNames" ``` -#### Verify metrics in HyperDX {#verify-metrics} +#### Verify metrics in hyperdx {#verify-metrics} Once loaded, the quickest way to see your metrics is through the pre-built dashboard. @@ -290,7 +290,7 @@ To help you get started monitoring Redis with ClickStack, we provide essential v -#### Download the dashboard configuration {#download} +#### Download the dashboard configuration {#download} #### Import the pre-built dashboard {#import-dashboard} @@ -334,7 +334,7 @@ View the custom config content to verify it's readable: docker exec cat /etc/otelcol-contrib/custom.config.yaml ``` -### No metrics appearing in HyperDX {#no-metrics} +### No metrics appearing in hyperdx {#no-metrics} Verify Redis is accessible from the collector: ```bash diff --git a/docs/use-cases/observability/clickstack/migration/elastic/concepts.md b/docs/use-cases/observability/clickstack/migration/elastic/concepts.md index db96a4fba48..e27259f24dc 100644 --- a/docs/use-cases/observability/clickstack/migration/elastic/concepts.md +++ b/docs/use-cases/observability/clickstack/migration/elastic/concepts.md @@ -19,7 +19,7 @@ import elasticsearch_execution from '@site/static/images/use-cases/observability import elasticsearch_transforms from '@site/static/images/use-cases/observability/es-transforms.png'; import clickhouse_mvs from '@site/static/images/use-cases/observability/ch-mvs.png'; -## Elastic Stack vs ClickStack {#elastic-vs-clickstack} +## Elastic stack vs ClickStack {#elastic-vs-clickstack} Both Elastic Stack and ClickStack cover the core roles of an observability platform, but they approach these roles with different design philosophies. These roles include: diff --git a/docs/use-cases/observability/clickstack/migration/elastic/intro.md b/docs/use-cases/observability/clickstack/migration/elastic/intro.md index ea874d54f85..55c757df230 100644 --- a/docs/use-cases/observability/clickstack/migration/elastic/intro.md +++ b/docs/use-cases/observability/clickstack/migration/elastic/intro.md @@ -11,7 +11,7 @@ keywords: ['Elasticsearch'] doc_type: 'guide' --- -## Migrating to ClickStack from Elastic {#migrating-to-clickstack-from-elastic} +## Migrating to ClickStack from elastic {#migrating-to-clickstack-from-elastic} This guide is intended for users migrating from the Elastic Stack — specifically those using Kibana to monitor logs, traces, and metrics collected via Elastic Agent and stored in Elasticsearch. It outlines equivalent concepts and data types in ClickStack, explains how to translate Kibana Lucene-based queries to HyperDX's syntax, and provides guidance on migrating both data and agents for a smooth transition. diff --git a/docs/use-cases/observability/clickstack/migration/elastic/migrating-agents.md b/docs/use-cases/observability/clickstack/migration/elastic/migrating-agents.md index 0effc78a1f4..809f09f4f5a 100644 --- a/docs/use-cases/observability/clickstack/migration/elastic/migrating-agents.md +++ b/docs/use-cases/observability/clickstack/migration/elastic/migrating-agents.md @@ -17,7 +17,7 @@ import add_logstash_output from '@site/static/images/use-cases/observability/add import agent_output_settings from '@site/static/images/use-cases/observability/agent-output-settings.png'; import migrating_agents from '@site/static/images/use-cases/observability/clickstack-migrating-agents.png'; -## Migrating agents from Elastic {#migrating-agents-from-elastic} +## Migrating agents from elastic {#migrating-agents-from-elastic} The Elastic Stack provides a number of Observability data collection agents. Specifically: @@ -35,7 +35,7 @@ Where possible we recommend migrating to the [OpenTelemetry (OTel) Collector](ht The OpenTelemetry Collector provides a sustainable and vendor-neutral solution for observability data ingestion. We recognize that some organizations operate fleets of thousands—or even tens of thousands—of Elastic agents. For these users, maintaining compatibility with existing agent infrastructure may be critical. This documentation is designed to support this, while also helping teams gradually transition to OpenTelemetry-based collection. ::: -## ClickHouse OpenTelemetry endpoint {#clickhouse-otel-endpoint} +## ClickHouse opentelemetry endpoint {#clickhouse-otel-endpoint} All data is ingested into ClickStack via an **OpenTelemetry (OTel) collector** instance, which acts as the primary entry point for logs, metrics, traces, and session data. We recommend using the official [ClickStack distribution](/use-cases/observability/clickstack/ingesting-data/opentelemetry#installing-otel-collector) of the collector for this instance, if not [already bundled in your ClickStack deployment model](/use-cases/observability/clickstack/deployment). @@ -289,7 +289,7 @@ sinks: content-type: application/json ``` -### Configure Filebeat {#configure-filebeat} +### Configure filebeat {#configure-filebeat} Existing Filebeat installations simply need to be modified to send their events to Vector. This requires the configuration of a Logstash output - again, TLS can be optionally configured: @@ -312,7 +312,7 @@ output.logstash: -## Migrating from Elastic Agent {#migrating-from-elastic-agent} +## Migrating from elastic agent {#migrating-from-elastic-agent} The Elastic Agent consolidates the different Elastic Beats into a single package. This agent integrates with [Elastic Fleet](https://www.elastic.co/docs/reference/fleet/fleet-server), allowing it to be centrally orchestrated and configured. @@ -323,15 +323,15 @@ Users with Elastic Agents deployed have several migration paths: We demonstrate both of these options below. -### Sending data via Vector {#sending-data-via-vector} +### Sending data via vector {#sending-data-via-vector} -#### Install and configure Vector {#install-configure-vector} +#### Install and configure vector {#install-configure-vector} Install and configure Vector using the [same steps](#install-vector) as those documented for migrating from Filebeat. -#### Configure Elastic Agent {#configure-elastic-agent} +#### Configure elastic agent {#configure-elastic-agent} Elastic Agent needs to be configured to send data via the Logstash protocol Lumberjack. This is a [supported deployment pattern](https://www.elastic.co/docs/manage-data/ingest/ingest-reference-architectures/ls-networkbridge) and can either be configured centrally or [via the agent configuration file `elastic-agent.yaml`](https://www.elastic.co/docs/reference/fleet/logstash-output) if deploying without Fleet. @@ -363,7 +363,7 @@ sources: -### Run Elastic Agent as OpenTelemetry collector {#run-agent-as-otel} +### Run elastic agent as opentelemetry collector {#run-agent-as-otel} The Elastic Agent includes an embedded EDOT Collector that allows you to instrument your applications and infrastructure once and send data to multiple vendors and backends. @@ -404,6 +404,6 @@ exporters: key_file: /path/to/client.key ``` -## Migrating from the Elastic OpenTelemetry collector {#migrating-from-elastic-otel-collector} +## Migrating from the elastic opentelemetry collector {#migrating-from-elastic-otel-collector} Users already running the [Elastic OpenTelemetry Collector (EDOT)](https://www.elastic.co/docs/reference/opentelemetry) can simply reconfigure their agents to send to ClickStack OpenTelemetry collector via OTLP. The steps involved are identical to those outlined above for running the [Elastic Agent as an OpenTelemetry collector](#run-agent-as-otel). This approach can be used for all data types. diff --git a/docs/use-cases/observability/clickstack/migration/elastic/migrating-sdks.md b/docs/use-cases/observability/clickstack/migration/elastic/migrating-sdks.md index f83c6bfc770..8b738f64afc 100644 --- a/docs/use-cases/observability/clickstack/migration/elastic/migrating-sdks.md +++ b/docs/use-cases/observability/clickstack/migration/elastic/migrating-sdks.md @@ -24,7 +24,7 @@ The Elastic Stack provides two types of language SDKs for instrumenting applicat While standard OpenTelemetry SDKs are supported, we strongly recommend using the [**ClickStack-distributed SDKs**](/use-cases/observability/clickstack/sdks) for each language. These distributions include additional instrumentation, enhanced defaults, and custom extensions designed to work seamlessly with the ClickStack pipeline and HyperDX UI. By using the ClickStack SDKs, you can unlock advanced features such as exception stack traces that are not available with vanilla OpenTelemetry or EDOT SDKs. ::: -## Migrating EDOT SDKs {#migrating-edot-sdks} +## Migrating edot sdks {#migrating-edot-sdks} Similar to the ClickStack OpenTelemetry-based SDKs, the Elastic Distributions of the OpenTelemetry SDKs (EDOT SDKs) are customized versions of the official OpenTelemetry SDKs. For example, the [EDOT Python SDK](https://www.elastic.co/docs/reference/opentelemetry/edot-sdks/python/) is a vendor-customized distribution of the [OpenTelemetry Python SDK](https://opentelemetry.io/docs/languages/python/) designed to work seamlessly with Elastic Observability. diff --git a/docs/use-cases/observability/clickstack/migration/elastic/search.md b/docs/use-cases/observability/clickstack/migration/elastic/search.md index 1e656688d61..caa2567f81c 100644 --- a/docs/use-cases/observability/clickstack/migration/elastic/search.md +++ b/docs/use-cases/observability/clickstack/migration/elastic/search.md @@ -14,7 +14,7 @@ import Image from '@theme/IdealImage'; import hyperdx_search from '@site/static/images/use-cases/observability/hyperdx-search.png'; import hyperdx_sql from '@site/static/images/use-cases/observability/hyperdx-sql.png'; -## Search in ClickStack and Elastic {#search-in-clickstack-and-elastic} +## Search in ClickStack and elastic {#search-in-clickstack-and-elastic} ClickHouse is a SQL-native engine, designed from the ground up for high-performance analytical workloads. In contrast, Elasticsearch provides a SQL-like interface, transpiling SQL into the underlying Elasticsearch query DSL — meaning it is not a first-class citizen, and [feature parity](https://www.elastic.co/docs/explore-analyze/query-filter/languages/sql-limitations) is limited. diff --git a/docs/use-cases/observability/clickstack/overview.md b/docs/use-cases/observability/clickstack/overview.md index 25aa20f67ee..f2b47b8f737 100644 --- a/docs/use-cases/observability/clickstack/overview.md +++ b/docs/use-cases/observability/clickstack/overview.md @@ -71,7 +71,7 @@ ClickStack provides full-stack visibility, from front-end user sessions to backe Every layer of the stack is designed to make full use of ClickHouse's capabilities. Queries are optimized to leverage ClickHouse's analytical functions and columnar engine, ensuring fast search and aggregation over massive volumes of data. -### OpenTelemetry-native {#open-telemetry-native} +### Opentelemetry-native {#open-telemetry-native} ClickStack is natively integrated with OpenTelemetry, ingesting all data through an OpenTelemetry collector endpoint. For advanced users, it also supports direct ingestion into ClickHouse using native file formats, custom pipelines, or third-party tools like Vector. diff --git a/docs/use-cases/observability/clickstack/production.md b/docs/use-cases/observability/clickstack/production.md index d6095172c5a..bfc47b3a797 100644 --- a/docs/use-cases/observability/clickstack/production.md +++ b/docs/use-cases/observability/clickstack/production.md @@ -91,7 +91,7 @@ For production deployments, we recommend using [ClickHouse Cloud](https://clickh ### User permissions {#user-permissions} -#### HyperDX user {#hyperdx-user} +#### Hyperdx user {#hyperdx-user} The ClickHouse user for HyperDX only needs to be a `readonly` user with access to change the following settings: @@ -125,7 +125,7 @@ ClickHouse OSS provides robust security features out of the box. However, these See also [external authenticators](/operations/external-authenticators) and [query complexity settings](/operations/settings/query-complexity) for managing users and ensuring query/resource limits. -### Configure Time To Live (TTL) {#configure-ttl} +### Configure time to live (TTL) {#configure-ttl} Ensure the [Time To Live (TTL)](/use-cases/observability/clickstack/ttl) has been [appropriately configured](/use-cases/observability/clickstack/ttl#modifying-ttl) for your ClickStack deployment. This controls how long data is retained for - the default of 3 days often needs to be modified. @@ -151,7 +151,7 @@ To find the connection details for HyperDX, navigate to the ClickHouse Cloud con Connect Cloud -### Create a HyperDX user {#create-a-user} +### Create a hyperdx user {#create-a-user} We recommend you create a dedicated user for HyperDX. Run the following SQL commands in the [Cloud SQL console](/cloud/get-started/sql-console), providing a secure password which meets complexity requirements: @@ -180,7 +180,7 @@ Advanced users can deploy the [OTel collector](/use-cases/observability/clicksta Instructions for using ClickHouse Cloud with the Helm chart can be found [here](/use-cases/observability/clickstack/deployment/helm#using-clickhouse-cloud). Equivalent instructions for Docker Compose can be found [here](/use-cases/observability/clickstack/deployment/docker-compose). -### Navigate to the HyperDX UI {#navigate-to-hyperdx-ui} +### Navigate to the hyperdx UI {#navigate-to-hyperdx-ui} Visit [http://localhost:8080](http://localhost:8080) to access the HyperDX UI. diff --git a/docs/use-cases/observability/clickstack/search.md b/docs/use-cases/observability/clickstack/search.md index ae4752a47d2..d56fdca9500 100644 --- a/docs/use-cases/observability/clickstack/search.md +++ b/docs/use-cases/observability/clickstack/search.md @@ -19,7 +19,7 @@ ClickStack allows you to do a full-text search on your events (logs and traces). This same search syntax is used for filtering events with Dashboards and Charts as well. -## Search Features {#search-features} +## Search features {#search-features} ### Natural language search syntax {#natural-language-syntax} diff --git a/docs/use-cases/time-series/02_basic-operations.md b/docs/use-cases/time-series/02_basic-operations.md index 334dba570cf..daceb3eb0cd 100644 --- a/docs/use-cases/time-series/02_basic-operations.md +++ b/docs/use-cases/time-series/02_basic-operations.md @@ -214,7 +214,7 @@ ORDER BY hour ASC WITH FILL STEP toIntervalHour(1); └─────────────────────┴───────────┘ ``` -## Rolling time windows {#time-series-rolling-time-windows} +## Rolling time Windows {#time-series-rolling-time-windows} Sometimes, we don't want to deal with the start of intervals (like the start of a day or an hour) but window intervals. Let's say we want to understand the total hits for a window, not based on days but on a 24-hour period offset from 6 pm. diff --git a/docs/whats-new/changelog/2017.md b/docs/whats-new/changelog/2017.md index 48c6b07e81e..ac8063b49fd 100644 --- a/docs/whats-new/changelog/2017.md +++ b/docs/whats-new/changelog/2017.md @@ -8,13 +8,13 @@ doc_type: 'changelog' keywords: ['ClickHouse 2017', 'changelog 2017', 'release notes', 'version history', 'early releases'] --- -### ClickHouse Release 1.1.54327, 2017-12-21 {#clickhouse-release-1-1-54327-2017-12-21} +### ClickHouse release 1.1.54327, 2017-12-21 {#clickhouse-release-1-1-54327-2017-12-21} This release contains bug fixes for the previous release 1.1.54318: - Fixed bug with possible race condition in replication that could lead to data loss. This issue affects versions 1.1.54310 and 1.1.54318. If you use one of these versions with Replicated tables, the update is strongly recommended. This issue shows in logs in Warning messages like `Part ... from own log does not exist.` The issue is relevant even if you do not see these messages in logs. -### ClickHouse Release 1.1.54318, 2017-11-30 {#clickhouse-release-1-1-54318-2017-11-30} +### ClickHouse release 1.1.54318, 2017-11-30 {#clickhouse-release-1-1-54318-2017-11-30} This release contains bug fixes for the previous release 1.1.54310: @@ -24,9 +24,9 @@ This release contains bug fixes for the previous release 1.1.54310: - Fixed an issue that was causing the replication queue to stop running - Fixed rotation and archiving of server logs -### ClickHouse Release 1.1.54310, 2017-11-01 {#clickhouse-release-1-1-54310-2017-11-01} +### ClickHouse release 1.1.54310, 2017-11-01 {#clickhouse-release-1-1-54310-2017-11-01} -#### New Features: {#new-features} +#### New features: {#new-features} - Custom partitioning key for the MergeTree family of table engines. - [Kafka](/engines/table-engines/integrations/kafka) table engine. @@ -43,13 +43,13 @@ This release contains bug fixes for the previous release 1.1.54310: - Added support for the Cap'n Proto input format. - You can now customize compression level when using the zstd algorithm. -#### Backward Incompatible Changes: {#backward-incompatible-changes} +#### Backward incompatible changes: {#backward-incompatible-changes} - Creation of temporary tables with an engine other than Memory is not allowed. - Explicit creation of tables with the View or MaterializedView engine is not allowed. - During table creation, a new check verifies that the sampling key expression is included in the primary key. -#### Bug Fixes: {#bug-fixes} +#### Bug fixes: {#bug-fixes} - Fixed hangups when synchronously inserting into a Distributed table. - Fixed nonatomic adding and removing of parts in Replicated tables. @@ -60,17 +60,17 @@ This release contains bug fixes for the previous release 1.1.54310: - Fixed hangups when the disk volume containing server logs is full. - Fixed an overflow in the toRelativeWeekNum function for the first week of the Unix epoch. -#### Build Improvements: {#build-improvements} +#### Build improvements: {#build-improvements} - Several third-party libraries (notably Poco) were updated and converted to git submodules. -### ClickHouse Release 1.1.54304, 2017-10-19 {#clickhouse-release-1-1-54304-2017-10-19} +### ClickHouse release 1.1.54304, 2017-10-19 {#clickhouse-release-1-1-54304-2017-10-19} -#### New Features: {#new-features-1} +#### New features: {#new-features-1} - TLS support in the native protocol (to enable, set `tcp_ssl_port` in `config.xml` ). -#### Bug Fixes: {#bug-fixes-1} +#### Bug fixes: {#bug-fixes-1} - `ALTER` for replicated tables now tries to start running as soon as possible. - Fixed crashing when reading data with the setting `preferred_block_size_bytes=0.` @@ -84,9 +84,9 @@ This release contains bug fixes for the previous release 1.1.54310: - Users are updated correctly with invalid `users.xml` - Correct handling when an executable dictionary returns a non-zero response code. -### ClickHouse Release 1.1.54292, 2017-09-20 {#clickhouse-release-1-1-54292-2017-09-20} +### ClickHouse release 1.1.54292, 2017-09-20 {#clickhouse-release-1-1-54292-2017-09-20} -#### New Features: {#new-features-2} +#### New features: {#new-features-2} - Added the `pointInPolygon` function for working with coordinates on a coordinate plane. - Added the `sumMap` aggregate function for calculating the sum of arrays, similar to `SummingMergeTree`. @@ -94,7 +94,7 @@ This release contains bug fixes for the previous release 1.1.54310: - The ClickHouse executable file is now less dependent on the libc version. The same ClickHouse executable file can run on a wide variety of Linux systems. There is still a dependency when using compiled queries (with the setting `compile = 1` , which is not used by default). - Reduced the time needed for dynamic compilation of queries. -#### Bug Fixes: {#bug-fixes-2} +#### Bug fixes: {#bug-fixes-2} - Fixed an error that sometimes produced `part ... intersects previous part` messages and weakened replica consistency. - Fixed an error that caused the server to lock up if ZooKeeper was unavailable during shutdown. @@ -103,9 +103,9 @@ This release contains bug fixes for the previous release 1.1.54310: - Fixed an error in the concat function that occurred if the first column in a block has the Array type. - Progress is now displayed correctly in the system.merges table. -### ClickHouse Release 1.1.54289, 2017-09-13 {#clickhouse-release-1-1-54289-2017-09-13} +### ClickHouse release 1.1.54289, 2017-09-13 {#clickhouse-release-1-1-54289-2017-09-13} -#### New Features: {#new-features-3} +#### New features: {#new-features-3} - `SYSTEM` queries for server administration: `SYSTEM RELOAD DICTIONARY`, `SYSTEM RELOAD DICTIONARIES`, `SYSTEM DROP DNS CACHE`, `SYSTEM SHUTDOWN`, `SYSTEM KILL`. - Added functions for working with arrays: `concat`, `arraySlice`, `arrayPushBack`, `arrayPushFront`, `arrayPopBack`, `arrayPopFront`. @@ -121,7 +121,7 @@ This release contains bug fixes for the previous release 1.1.54310: - Option to set `umask` in the config file. - Improved performance for queries with `DISTINCT` . -#### Bug Fixes: {#bug-fixes-3} +#### Bug fixes: {#bug-fixes-3} - Improved the process for deleting old nodes in ZooKeeper. Previously, old nodes sometimes didn't get deleted if there were very frequent inserts, which caused the server to be slow to shut down, among other things. - Fixed randomization when choosing hosts for the connection to ZooKeeper. @@ -135,21 +135,21 @@ This release contains bug fixes for the previous release 1.1.54310: - Resolved the appearance of zombie processes when using a dictionary with an `executable` source. - Fixed segfault for the HEAD query. -#### Improved Workflow for Developing and Assembling ClickHouse: {#improved-workflow-for-developing-and-assembling-clickhouse} +#### Improved workflow for developing and assembling ClickHouse: {#improved-workflow-for-developing-and-assembling-clickhouse} - You can use `pbuilder` to build ClickHouse. - You can use `libc++` instead of `libstdc++` for builds on Linux. - Added instructions for using static code analysis tools: `Coverage`, `clang-tidy`, `cppcheck`. -#### Please Note When Upgrading: {#please-note-when-upgrading} +#### Please note when upgrading: {#please-note-when-upgrading} - There is now a higher default value for the MergeTree setting `max_bytes_to_merge_at_max_space_in_pool` (the maximum total size of data parts to merge, in bytes): it has increased from 100 GiB to 150 GiB. This might result in large merges running after the server upgrade, which could cause an increased load on the disk subsystem. If the free space available on the server is less than twice the total amount of the merges that are running, this will cause all other merges to stop running, including merges of small data parts. As a result, INSERT queries will fail with the message "Merges are processing significantly slower than inserts." Use the `SELECT * FROM system.merges` query to monitor the situation. You can also check the `DiskSpaceReservedForMerge` metric in the `system.metrics` table, or in Graphite. You do not need to do anything to fix this, since the issue will resolve itself once the large merges finish. If you find this unacceptable, you can restore the previous value for the `max_bytes_to_merge_at_max_space_in_pool` setting. To do this, go to the `` section in config.xml, set ``` ``107374182400 ``` and restart the server. -### ClickHouse Release 1.1.54284, 2017-08-29 {#clickhouse-release-1-1-54284-2017-08-29} +### ClickHouse release 1.1.54284, 2017-08-29 {#clickhouse-release-1-1-54284-2017-08-29} - This is a bugfix release for the previous 1.1.54282 release. It fixes leaks in the parts directory in ZooKeeper. -### ClickHouse Release 1.1.54282, 2017-08-23 {#clickhouse-release-1-1-54282-2017-08-23} +### ClickHouse release 1.1.54282, 2017-08-23 {#clickhouse-release-1-1-54282-2017-08-23} This release contains bug fixes for the previous release 1.1.54276: @@ -157,9 +157,9 @@ This release contains bug fixes for the previous release 1.1.54276: - Fixed parsing when inserting in RowBinary format if input data starts with';'. - Errors during runtime compilation of certain aggregate functions (e.g. `groupArray()`). -### ClickHouse Release 1.1.54276, 2017-08-16 {#clickhouse-release-1-1-54276-2017-08-16} +### ClickHouse release 1.1.54276, 2017-08-16 {#clickhouse-release-1-1-54276-2017-08-16} -#### New Features: {#new-features-4} +#### New features: {#new-features-4} - Added an optional WITH section for a SELECT query. Example query: `WITH 1+1 AS a SELECT a, a*a` - INSERT can be performed synchronously in a Distributed table: OK is returned only after all the data is saved on all the shards. This is activated by the setting insert_distributed_sync=1. @@ -170,7 +170,7 @@ This release contains bug fixes for the previous release 1.1.54276: - Added support for non-constant arguments and negative offsets in the function `substring(str, pos, len).` - Added the max_size parameter for the `groupArray(max_size)(column)` aggregate function, and optimized its performance. -#### Main Changes: {#main-changes} +#### Main changes: {#main-changes} - Security improvements: all server files are created with 0640 permissions (can be changed via `` config parameter). - Improved error messages for queries with invalid syntax. @@ -178,11 +178,11 @@ This release contains bug fixes for the previous release 1.1.54276: - Significantly increased the performance of data merges for the ReplacingMergeTree engine. - Improved performance for asynchronous inserts from a Distributed table by combining multiple source inserts. To enable this functionality, use the setting distributed_directory_monitor_batch_inserts=1. -#### Backward Incompatible Changes: {#backward-incompatible-changes-1} +#### Backward incompatible changes: {#backward-incompatible-changes-1} - Changed the binary format of aggregate states of `groupArray(array_column)` functions for arrays. -#### Complete List of Changes: {#complete-list-of-changes} +#### Complete list of changes: {#complete-list-of-changes} - Added the `output_format_json_quote_denormals` setting, which enables outputting nan and inf values in JSON format. - Optimized stream allocation when reading from a Distributed table. @@ -201,7 +201,7 @@ This release contains bug fixes for the previous release 1.1.54276: - It is possible to connect to MySQL through a socket in the file system. - The system.parts table has a new column with information about the size of marks, in bytes. -#### Bug Fixes: {#bug-fixes-4} +#### Bug fixes: {#bug-fixes-4} - Distributed tables using a Merge table now work correctly for a SELECT query with a condition on the `_table` field. - Fixed a rare race condition in ReplicatedMergeTree when checking data parts. @@ -225,15 +225,15 @@ This release contains bug fixes for the previous release 1.1.54276: - Fixed the "Cannot mremap" error when using arrays in IN and JOIN clauses with more than 2 billion elements. - Fixed the failover for dictionaries with MySQL as the source. -#### Improved Workflow for Developing and Assembling ClickHouse: {#improved-workflow-for-developing-and-assembling-clickhouse-1} +#### Improved workflow for developing and assembling ClickHouse: {#improved-workflow-for-developing-and-assembling-clickhouse-1} - Builds can be assembled in Arcadia. - You can use gcc 7 to compile ClickHouse. - Parallel builds using ccache+distcc are faster now. -### ClickHouse Release 1.1.54245, 2017-07-04 {#clickhouse-release-1-1-54245-2017-07-04} +### ClickHouse release 1.1.54245, 2017-07-04 {#clickhouse-release-1-1-54245-2017-07-04} -#### New Features: {#new-features-5} +#### New features: {#new-features-5} - Distributed DDL (for example, `CREATE TABLE ON CLUSTER`) - The replicated query `ALTER TABLE CLEAR COLUMN IN PARTITION.` @@ -245,16 +245,16 @@ This release contains bug fixes for the previous release 1.1.54276: - Sessions in the HTTP interface. - The OPTIMIZE query for a Replicated table can can run not only on the leader. -#### Backward Incompatible Changes: {#backward-incompatible-changes-2} +#### Backward incompatible changes: {#backward-incompatible-changes-2} - Removed SET GLOBAL. -#### Minor Changes: {#minor-changes} +#### Minor changes: {#minor-changes} - Now after an alert is triggered, the log prints the full stack trace. - Relaxed the verification of the number of damaged/extra data parts at startup (there were too many false positives). -#### Bug Fixes: {#bug-fixes-5} +#### Bug fixes: {#bug-fixes-5} - Fixed a bad connection "sticking" when inserting into a Distributed table. - GLOBAL IN now works for a query from a Merge table that looks at a Distributed table. diff --git a/docs/whats-new/changelog/2018.md b/docs/whats-new/changelog/2018.md index 59df7280626..0e5becc9ae5 100644 --- a/docs/whats-new/changelog/2018.md +++ b/docs/whats-new/changelog/2018.md @@ -8,11 +8,11 @@ doc_type: 'changelog' keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version history', 'legacy releases'] --- -## ClickHouse Release 18.16 {#clickhouse-release-18-16} +## ClickHouse release 18.16 {#clickhouse-release-18-16} -### ClickHouse Release 18.16.1, 2018-12-21 {#clickhouse-release-18-16-1-2018-12-21} +### ClickHouse release 18.16.1, 2018-12-21 {#clickhouse-release-18-16-1-2018-12-21} -#### Bug Fixes: {#bug-fixes} +#### Bug fixes: {#bug-fixes} - Fixed an error that led to problems with updating dictionaries with the ODBC source. [#3825](https://github.com/ClickHouse/ClickHouse/issues/3825), [#3829](https://github.com/ClickHouse/ClickHouse/issues/3829) - JIT compilation of aggregate functions now works with LowCardinality columns. [#3838](https://github.com/ClickHouse/ClickHouse/issues/3838) @@ -21,13 +21,13 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Added the `low_cardinality_allow_in_native_format` setting (enabled by default). When disabled, LowCardinality columns will be converted to ordinary columns for SELECT queries and ordinary columns will be expected for INSERT queries. [#3879](https://github.com/ClickHouse/ClickHouse/pull/3879) -#### Build Improvements: {#build-improvements} +#### Build improvements: {#build-improvements} - Fixes for builds on macOS and ARM. -### ClickHouse Release 18.16.0, 2018-12-14 {#clickhouse-release-18-16-0-2018-12-14} +### ClickHouse release 18.16.0, 2018-12-14 {#clickhouse-release-18-16-0-2018-12-14} -#### New Features: {#new-features} +#### New features: {#new-features} - `DEFAULT` expressions are evaluated for missing fields when loading data in semi-structured input formats (`JSONEachRow`, `TSKV`). The feature is enabled with the `insert_sample_with_metadata` setting. [#3555](https://github.com/ClickHouse/ClickHouse/pull/3555) - The `ALTER TABLE` query now has the `MODIFY ORDER BY` action for changing the sorting key when adding or removing a table column. This is useful for tables in the `MergeTree` family that perform additional tasks when merging based on this sorting key, such as `SummingMergeTree`, `AggregatingMergeTree`, and so on. [#3581](https://github.com/ClickHouse/ClickHouse/pull/3581) [#3755](https://github.com/ClickHouse/ClickHouse/pull/3755) @@ -46,7 +46,7 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Added the `is_in_partition_key`, `is_in_sorting_key`, `is_in_primary_key`, and `is_in_sampling_key` columns to the `system.columns` table. [#3609](https://github.com/ClickHouse/ClickHouse/pull/3609) - Added the `min_time` and `max_time` columns to the `system.parts` table. These columns are populated when the partitioning key is an expression consisting of `DateTime` columns. [Emmanuel Donin de Rosière](https://github.com/ClickHouse/ClickHouse/pull/3800) -#### Bug Fixes: {#bug-fixes-1} +#### Bug fixes: {#bug-fixes-1} - Fixes and performance improvements for the `LowCardinality` data type. `GROUP BY` using `LowCardinality(Nullable(...))`. Getting the values of `extremes`. Processing high-order functions. `LEFT ARRAY JOIN`. Distributed `GROUP BY`. Functions that return `Array`. Execution of `ORDER BY`. Writing to `Distributed` tables (nicelulu). Backward compatibility for `INSERT` queries from old clients that implement the `Native` protocol. Support for `LowCardinality` for `JOIN`. Improved performance when working in a single stream. [#3823](https://github.com/ClickHouse/ClickHouse/pull/3823) [#3803](https://github.com/ClickHouse/ClickHouse/pull/3803) [#3799](https://github.com/ClickHouse/ClickHouse/pull/3799) [#3769](https://github.com/ClickHouse/ClickHouse/pull/3769) [#3744](https://github.com/ClickHouse/ClickHouse/pull/3744) [#3681](https://github.com/ClickHouse/ClickHouse/pull/3681) [#3651](https://github.com/ClickHouse/ClickHouse/pull/3651) [#3649](https://github.com/ClickHouse/ClickHouse/pull/3649) [#3641](https://github.com/ClickHouse/ClickHouse/pull/3641) [#3632](https://github.com/ClickHouse/ClickHouse/pull/3632) [#3568](https://github.com/ClickHouse/ClickHouse/pull/3568) [#3523](https://github.com/ClickHouse/ClickHouse/pull/3523) [#3518](https://github.com/ClickHouse/ClickHouse/pull/3518) - Fixed how the `select_sequential_consistency` option works. Previously, when this setting was enabled, an incomplete result was sometimes returned after beginning to write to a new partition. [#2863](https://github.com/ClickHouse/ClickHouse/pull/2863) @@ -105,7 +105,7 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Fixed the behavior of stateful functions like `rowNumberInAllBlocks`. They previously output a result that was one number larger due to starting during query analysis. [Amos Bird](https://github.com/ClickHouse/ClickHouse/pull/3729) - If the `force_restore_data` file can't be deleted, an error message is displayed. [Amos Bird](https://github.com/ClickHouse/ClickHouse/pull/3794) -#### Build Improvements: {#build-improvements-1} +#### Build improvements: {#build-improvements-1} - Updated the `jemalloc` library, which fixes a potential memory leak. [Amos Bird](https://github.com/ClickHouse/ClickHouse/pull/3557) - Profiling with `jemalloc` is enabled by default in order to debug builds. [2cc82f5c](https://github.com/ClickHouse/ClickHouse/commit/2cc82f5cbe266421cd4c1165286c2c47e5ffcb15) @@ -116,40 +116,40 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - For a Docker image, added support for initializing databases using files in the `/docker-entrypoint-initdb.d` directory. [Konstantin Lebedev](https://github.com/ClickHouse/ClickHouse/pull/3695) - Fixes for builds on ARM. [#3709](https://github.com/ClickHouse/ClickHouse/pull/3709) -#### Backward Incompatible Changes: {#backward-incompatible-changes} +#### Backward incompatible changes: {#backward-incompatible-changes} - Removed the ability to compare the `Date` type with a number. Instead of `toDate('2018-12-18') = 17883`, you must use explicit type conversion `= toDate(17883)` [#3687](https://github.com/ClickHouse/ClickHouse/pull/3687) -## ClickHouse Release 18.14 {#clickhouse-release-18-14} +## ClickHouse release 18.14 {#clickhouse-release-18-14} -### ClickHouse Release 18.14.19, 2018-12-19 {#clickhouse-release-18-14-19-2018-12-19} +### ClickHouse release 18.14.19, 2018-12-19 {#clickhouse-release-18-14-19-2018-12-19} -#### Bug Fixes: {#bug-fixes-2} +#### Bug fixes: {#bug-fixes-2} - Fixed an error that led to problems with updating dictionaries with the ODBC source. [#3825](https://github.com/ClickHouse/ClickHouse/issues/3825), [#3829](https://github.com/ClickHouse/ClickHouse/issues/3829) - Databases are correctly specified when executing DDL `ON CLUSTER` queries. [#3460](https://github.com/ClickHouse/ClickHouse/pull/3460) - Fixed a segfault if the `max_temporary_non_const_columns` limit was exceeded. [#3788](https://github.com/ClickHouse/ClickHouse/pull/3788) -#### Build Improvements: {#build-improvements-2} +#### Build improvements: {#build-improvements-2} - Fixes for builds on ARM. -### ClickHouse Release 18.14.18, 2018-12-04 {#clickhouse-release-18-14-18-2018-12-04} +### ClickHouse release 18.14.18, 2018-12-04 {#clickhouse-release-18-14-18-2018-12-04} -#### Bug Fixes: {#bug-fixes-3} +#### Bug fixes: {#bug-fixes-3} - Fixed error in `dictGet...` function for dictionaries of type `range`, if one of the arguments is constant and other is not. [#3751](https://github.com/ClickHouse/ClickHouse/pull/3751) - Fixed error that caused messages `netlink: '...': attribute type 1 has an invalid length` to be printed in Linux kernel log, that was happening only on fresh enough versions of Linux kernel. [#3749](https://github.com/ClickHouse/ClickHouse/pull/3749) - Fixed segfault in function `empty` for argument of `FixedString` type. [Daniel, Dao Quang Minh](https://github.com/ClickHouse/ClickHouse/pull/3703) - Fixed excessive memory allocation when using large value of `max_query_size` setting (a memory chunk of `max_query_size` bytes was preallocated at once). [#3720](https://github.com/ClickHouse/ClickHouse/pull/3720) -#### Build Changes: {#build-changes} +#### Build changes: {#build-changes} - Fixed build with LLVM/Clang libraries of version 7 from the OS packages (these libraries are used for runtime query compilation). [#3582](https://github.com/ClickHouse/ClickHouse/pull/3582) -### ClickHouse Release 18.14.17, 2018-11-30 {#clickhouse-release-18-14-17-2018-11-30} +### ClickHouse release 18.14.17, 2018-11-30 {#clickhouse-release-18-14-17-2018-11-30} -#### Bug Fixes: {#bug-fixes-4} +#### Bug fixes: {#bug-fixes-4} - Fixed cases when the ODBC bridge process did not terminate with the main server process. [#3642](https://github.com/ClickHouse/ClickHouse/pull/3642) - Fixed synchronous insertion into the `Distributed` table with a columns list that differs from the column list of the remote table. [#3673](https://github.com/ClickHouse/ClickHouse/pull/3673) @@ -157,65 +157,65 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Fixed a query deadlock in case when query thread creation fails with the `Resource temporarily unavailable` error. [#3643](https://github.com/ClickHouse/ClickHouse/pull/3643) - Fixed parsing of the `ENGINE` clause when the `CREATE AS table` syntax was used and the `ENGINE` clause was specified before the `AS table` (the error resulted in ignoring the specified engine). [#3692](https://github.com/ClickHouse/ClickHouse/pull/3692) -### ClickHouse Release 18.14.15, 2018-11-21 {#clickhouse-release-18-14-15-2018-11-21} +### ClickHouse release 18.14.15, 2018-11-21 {#clickhouse-release-18-14-15-2018-11-21} -#### Bug Fixes: {#bug-fixes-5} +#### Bug fixes: {#bug-fixes-5} - The size of memory chunk was overestimated while deserializing the column of type `Array(String)` that leads to "Memory limit exceeded" errors. The issue appeared in version 18.12.13. [#3589](https://github.com/ClickHouse/ClickHouse/issues/3589) -### ClickHouse Release 18.14.14, 2018-11-20 {#clickhouse-release-18-14-14-2018-11-20} +### ClickHouse release 18.14.14, 2018-11-20 {#clickhouse-release-18-14-14-2018-11-20} -#### Bug Fixes: {#bug-fixes-6} +#### Bug fixes: {#bug-fixes-6} - Fixed `ON CLUSTER` queries when cluster configured as secure (flag ``). [#3599](https://github.com/ClickHouse/ClickHouse/pull/3599) -#### Build Changes: {#build-changes-1} +#### Build changes: {#build-changes-1} - Fixed problems (llvm-7 from system, macos) [#3582](https://github.com/ClickHouse/ClickHouse/pull/3582) -### ClickHouse Release 18.14.13, 2018-11-08 {#clickhouse-release-18-14-13-2018-11-08} +### ClickHouse release 18.14.13, 2018-11-08 {#clickhouse-release-18-14-13-2018-11-08} -#### Bug Fixes: {#bug-fixes-7} +#### Bug fixes: {#bug-fixes-7} - Fixed the `Block structure mismatch in MergingSorted stream` error. [#3162](https://github.com/ClickHouse/ClickHouse/issues/3162) - Fixed `ON CLUSTER` queries in case when secure connections were turned on in the cluster config (the `` flag). [#3465](https://github.com/ClickHouse/ClickHouse/pull/3465) - Fixed an error in queries that used `SAMPLE`, `PREWHERE` and alias columns. [#3543](https://github.com/ClickHouse/ClickHouse/pull/3543) - Fixed a rare `unknown compression method` error when the `min_bytes_to_use_direct_io` setting was enabled. [3544](https://github.com/ClickHouse/ClickHouse/pull/3544) -#### Performance Improvements: {#performance-improvements} +#### Performance improvements: {#performance-improvements} - Fixed performance regression of queries with `GROUP BY` of columns of UInt16 or Date type when executing on AMD EPYC processors. [Igor Lapko](https://github.com/ClickHouse/ClickHouse/pull/3512) - Fixed performance regression of queries that process long strings. [#3530](https://github.com/ClickHouse/ClickHouse/pull/3530) -#### Build Improvements: {#build-improvements-3} +#### Build improvements: {#build-improvements-3} - Improvements for simplifying the Arcadia build. [#3475](https://github.com/ClickHouse/ClickHouse/pull/3475), [#3535](https://github.com/ClickHouse/ClickHouse/pull/3535) -### ClickHouse Release 18.14.12, 2018-11-02 {#clickhouse-release-18-14-12-2018-11-02} +### ClickHouse release 18.14.12, 2018-11-02 {#clickhouse-release-18-14-12-2018-11-02} -#### Bug Fixes: {#bug-fixes-8} +#### Bug fixes: {#bug-fixes-8} - Fixed a crash on joining two unnamed subqueries. [#3505](https://github.com/ClickHouse/ClickHouse/pull/3505) - Fixed generating incorrect queries (with an empty `WHERE` clause) when querying external databases. [hotid](https://github.com/ClickHouse/ClickHouse/pull/3477) - Fixed using an incorrect timeout value in ODBC dictionaries. [Marek Vavruša](https://github.com/ClickHouse/ClickHouse/pull/3511) -### ClickHouse Release 18.14.11, 2018-10-29 {#clickhouse-release-18-14-11-2018-10-29} +### ClickHouse release 18.14.11, 2018-10-29 {#clickhouse-release-18-14-11-2018-10-29} -#### Bug Fixes: {#bug-fixes-9} +#### Bug fixes: {#bug-fixes-9} - Fixed the error `Block structure mismatch in UNION stream: different number of columns` in LIMIT queries. [#2156](https://github.com/ClickHouse/ClickHouse/issues/2156) - Fixed errors when merging data in tables containing arrays inside Nested structures. [#3397](https://github.com/ClickHouse/ClickHouse/pull/3397) - Fixed incorrect query results if the `merge_tree_uniform_read_distribution` setting is disabled (it is enabled by default). [#3429](https://github.com/ClickHouse/ClickHouse/pull/3429) - Fixed an error on inserts to a Distributed table in Native format. [#3411](https://github.com/ClickHouse/ClickHouse/issues/3411) -### ClickHouse Release 18.14.10, 2018-10-23 {#clickhouse-release-18-14-10-2018-10-23} +### ClickHouse release 18.14.10, 2018-10-23 {#clickhouse-release-18-14-10-2018-10-23} - The `compile_expressions` setting (JIT compilation of expressions) is disabled by default. [#3410](https://github.com/ClickHouse/ClickHouse/pull/3410) - The `enable_optimize_predicate_expression` setting is disabled by default. -### ClickHouse Release 18.14.9, 2018-10-16 {#clickhouse-release-18-14-9-2018-10-16} +### ClickHouse release 18.14.9, 2018-10-16 {#clickhouse-release-18-14-9-2018-10-16} -#### New Features: {#new-features-1} +#### New features: {#new-features-1} - The `WITH CUBE` modifier for `GROUP BY` (the alternative syntax `GROUP BY CUBE(...)` is also available). [#3172](https://github.com/ClickHouse/ClickHouse/pull/3172) - Added the `formatDateTime` function. [Alexandr Krasheninnikov](https://github.com/ClickHouse/ClickHouse/pull/2770) @@ -228,7 +228,7 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Now you can use pre-defined `database` and `table` macros when declaring `Replicated` tables. [#3251](https://github.com/ClickHouse/ClickHouse/pull/3251) - Added the ability to read `Decimal` type values in engineering notation (indicating powers of ten). [#3153](https://github.com/ClickHouse/ClickHouse/pull/3153) -#### Experimental Features: {#experimental-features} +#### Experimental features: {#experimental-features} - Optimization of the GROUP BY clause for `LowCardinality data types.` [#3138](https://github.com/ClickHouse/ClickHouse/pull/3138) - Optimized calculation of expressions for `LowCardinality data types.` [#3200](https://github.com/ClickHouse/ClickHouse/pull/3200) @@ -264,7 +264,7 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Reduced the number of `open` and `close` system calls when reading from a `MergeTree table`. [#3283](https://github.com/ClickHouse/ClickHouse/pull/3283) - A `TRUNCATE TABLE` query can be executed on any replica (the query is passed to the leader replica). [Kirill Shvakov](https://github.com/ClickHouse/ClickHouse/pull/3375) -#### Bug Fixes: {#bug-fixes-10} +#### Bug fixes: {#bug-fixes-10} - Fixed an issue with `Dictionary` tables for `range_hashed` dictionaries. This error occurred in version 18.12.17. [#1702](https://github.com/ClickHouse/ClickHouse/pull/1702) - Fixed an error when loading `range_hashed` dictionaries (the message `Unsupported type Nullable (...)`). This error occurred in version 18.12.17. [#3362](https://github.com/ClickHouse/ClickHouse/pull/3362) @@ -300,15 +300,15 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Fixed segfault that could occur in rare cases after optimization that replaced AND chains from equality evaluations with the corresponding IN expression. [liuyimin-bytedance](https://github.com/ClickHouse/ClickHouse/pull/3339) - Minor corrections to `clickhouse-benchmark`: previously, client information was not sent to the server; now the number of queries executed is calculated more accurately when shutting down and for limiting the number of iterations. [#3351](https://github.com/ClickHouse/ClickHouse/pull/3351) [#3352](https://github.com/ClickHouse/ClickHouse/pull/3352) -#### Backward Incompatible Changes: {#backward-incompatible-changes-1} +#### Backward incompatible changes: {#backward-incompatible-changes-1} - Removed the `allow_experimental_decimal_type` option. The `Decimal` data type is available for default use. [#3329](https://github.com/ClickHouse/ClickHouse/pull/3329) -## ClickHouse Release 18.12 {#clickhouse-release-18-12} +## ClickHouse release 18.12 {#clickhouse-release-18-12} -### ClickHouse Release 18.12.17, 2018-09-16 {#clickhouse-release-18-12-17-2018-09-16} +### ClickHouse release 18.12.17, 2018-09-16 {#clickhouse-release-18-12-17-2018-09-16} -#### New Features: {#new-features-2} +#### New features: {#new-features-2} - `invalidate_query` (the ability to specify a query to check whether an external dictionary needs to be updated) is implemented for the `clickhouse` source. [#3126](https://github.com/ClickHouse/ClickHouse/pull/3126) - Added the ability to use `UInt*`, `Int*`, and `DateTime` data types (along with the `Date` type) as a `range_hashed` external dictionary key that defines the boundaries of ranges. Now `NULL` can be used to designate an open range. [Vasily Nemkov](https://github.com/ClickHouse/ClickHouse/pull/3123) @@ -316,20 +316,20 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - The `Decimal` type now supports mathematical functions (`exp`, `sin` and so on.) [#3129](https://github.com/ClickHouse/ClickHouse/pull/3129) - The `system.part_log` table now has the `partition_id` column. [#3089](https://github.com/ClickHouse/ClickHouse/pull/3089) -#### Bug Fixes: {#bug-fixes-11} +#### Bug fixes: {#bug-fixes-11} - `Merge` now works correctly on `Distributed` tables. [Winter Zhang](https://github.com/ClickHouse/ClickHouse/pull/3159) - Fixed incompatibility (unnecessary dependency on the `glibc` version) that made it impossible to run ClickHouse on `Ubuntu Precise` and older versions. The incompatibility arose in version 18.12.13. [#3130](https://github.com/ClickHouse/ClickHouse/pull/3130) - Fixed errors in the `enable_optimize_predicate_expression` setting. [Winter Zhang](https://github.com/ClickHouse/ClickHouse/pull/3107) - Fixed a minor issue with backwards compatibility that appeared when working with a cluster of replicas on versions earlier than 18.12.13 and simultaneously creating a new replica of a table on a server with a newer version (shown in the message `Can not clone replica, because the ... updated to new ClickHouse version`, which is logical, but shouldn't happen). [#3122](https://github.com/ClickHouse/ClickHouse/pull/3122) -#### Backward Incompatible Changes: {#backward-incompatible-changes-2} +#### Backward incompatible changes: {#backward-incompatible-changes-2} - The `enable_optimize_predicate_expression` option is enabled by default (which is rather optimistic). If query analysis errors occur that are related to searching for the column names, set `enable_optimize_predicate_expression` to 0. [Winter Zhang](https://github.com/ClickHouse/ClickHouse/pull/3107) -### ClickHouse Release 18.12.14, 2018-09-13 {#clickhouse-release-18-12-14-2018-09-13} +### ClickHouse release 18.12.14, 2018-09-13 {#clickhouse-release-18-12-14-2018-09-13} -#### New Features: {#new-features-3} +#### New features: {#new-features-3} - Added support for `ALTER UPDATE` queries. [#3035](https://github.com/ClickHouse/ClickHouse/pull/3035) - Added the `allow_ddl` option, which restricts the user's access to DDL queries. [#3104](https://github.com/ClickHouse/ClickHouse/pull/3104) @@ -341,16 +341,16 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - If a data part remains unchanged during mutation, it isn't downloaded by replicas. [#3103](https://github.com/ClickHouse/ClickHouse/pull/3103) - Autocomplete is available for names of settings when working with `clickhouse-client`. [#3106](https://github.com/ClickHouse/ClickHouse/pull/3106) -#### Bug Fixes: {#bug-fixes-12} +#### Bug fixes: {#bug-fixes-12} - Added a check for the sizes of arrays that are elements of `Nested` type fields when inserting. [#3118](https://github.com/ClickHouse/ClickHouse/pull/3118) - Fixed an error updating external dictionaries with the `ODBC` source and `hashed` storage. This error occurred in version 18.12.13. - Fixed a crash when creating a temporary table from a query with an `IN` condition. [Winter Zhang](https://github.com/ClickHouse/ClickHouse/pull/3098) - Fixed an error in aggregate functions for arrays that can have `NULL` elements. [Winter Zhang](https://github.com/ClickHouse/ClickHouse/pull/3097) -### ClickHouse Release 18.12.13, 2018-09-10 {#clickhouse-release-18-12-13-2018-09-10} +### ClickHouse release 18.12.13, 2018-09-10 {#clickhouse-release-18-12-13-2018-09-10} -#### New Features: {#new-features-4} +#### New features: {#new-features-4} - Added the `DECIMAL(digits, scale)` data type (`Decimal32(scale)`, `Decimal64(scale)`, `Decimal128(scale)`). To enable it, use the setting `allow_experimental_decimal_type`. [#2846](https://github.com/ClickHouse/ClickHouse/pull/2846) [#2970](https://github.com/ClickHouse/ClickHouse/pull/2970) [#3008](https://github.com/ClickHouse/ClickHouse/pull/3008) [#3047](https://github.com/ClickHouse/ClickHouse/pull/3047) - New `WITH ROLLUP` modifier for `GROUP BY` (alternative syntax: `GROUP BY ROLLUP(...)`). [#2948](https://github.com/ClickHouse/ClickHouse/pull/2948) @@ -374,7 +374,7 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Now you can add (merge) states of aggregate functions by using the plus operator, and multiply the states of aggregate functions by a nonnegative constant. [#3062](https://github.com/ClickHouse/ClickHouse/pull/3062) [#3034](https://github.com/ClickHouse/ClickHouse/pull/3034) - Tables in the MergeTree family now have the virtual column `_partition_id`. [#3089](https://github.com/ClickHouse/ClickHouse/pull/3089) -#### Experimental Features: {#experimental-features-1} +#### Experimental features: {#experimental-features-1} - Added the `LowCardinality(T)` data type. This data type automatically creates a local dictionary of values and allows data processing without unpacking the dictionary. [#2830](https://github.com/ClickHouse/ClickHouse/pull/2830) - Added a cache of JIT-compiled functions and a counter for the number of uses before compiling. To JIT compile expressions, enable the `compile_expressions` setting. [#2990](https://github.com/ClickHouse/ClickHouse/pull/2990) [#3077](https://github.com/ClickHouse/ClickHouse/pull/3077) @@ -408,7 +408,7 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Added randomization when running the cleanup thread periodically for `ReplicatedMergeTree` tables in order to avoid periodic load spikes when there are a very large number of `ReplicatedMergeTree` tables. - Support for `ATTACH TABLE ... ON CLUSTER` queries. [#3025](https://github.com/ClickHouse/ClickHouse/pull/3025) -#### Bug Fixes: {#bug-fixes-13} +#### Bug fixes: {#bug-fixes-13} - Fixed an issue with `Dictionary` tables (throws the `Size of offsets does not match size of column` or `Unknown compression method` exception). This bug appeared in version 18.10.3. [#2913](https://github.com/ClickHouse/ClickHouse/issues/2913) - Fixed a bug when merging `CollapsingMergeTree` tables if one of the data parts is empty (these parts are formed during merge or `ALTER DELETE` if all data was deleted), and the `vertical` algorithm was used for the merge. [#3049](https://github.com/ClickHouse/ClickHouse/pull/3049) @@ -432,17 +432,17 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Fixed incorrect code for adding nested data structures in a `SummingMergeTree`. - When allocating memory for states of aggregate functions, alignment is correctly taken into account, which makes it possible to use operations that require alignment when implementing states of aggregate functions. [chenxing-xc](https://github.com/ClickHouse/ClickHouse/pull/2808) -#### Security Fix: {#security-fix} +#### Security fix: {#security-fix} - Safe use of ODBC data sources. Interaction with ODBC drivers uses a separate `clickhouse-odbc-bridge` process. Errors in third-party ODBC drivers no longer cause problems with server stability or vulnerabilities. [#2828](https://github.com/ClickHouse/ClickHouse/pull/2828) [#2879](https://github.com/ClickHouse/ClickHouse/pull/2879) [#2886](https://github.com/ClickHouse/ClickHouse/pull/2886) [#2893](https://github.com/ClickHouse/ClickHouse/pull/2893) [#2921](https://github.com/ClickHouse/ClickHouse/pull/2921) - Fixed incorrect validation of the file path in the `catBoostPool` table function. [#2894](https://github.com/ClickHouse/ClickHouse/pull/2894) - The contents of system tables (`tables`, `databases`, `parts`, `columns`, `parts_columns`, `merges`, `mutations`, `replicas`, and `replication_queue`) are filtered according to the user's configured access to databases (`allow_databases`). [Winter Zhang](https://github.com/ClickHouse/ClickHouse/pull/2856) -#### Backward Incompatible Changes: {#backward-incompatible-changes-3} +#### Backward incompatible changes: {#backward-incompatible-changes-3} - In queries with JOIN, the star character expands to a list of columns in all tables, in compliance with the SQL standard. You can restore the old behavior by setting `asterisk_left_columns_only` to 1 on the user configuration level. -#### Build Changes: {#build-changes-2} +#### Build changes: {#build-changes-2} - Most integration tests can now be run by commit. - Code style checks can also be run by commit. @@ -451,11 +451,11 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Debugging the build uses the `jemalloc` debug option. - The interface of the library for interacting with ZooKeeper is declared abstract. [#2950](https://github.com/ClickHouse/ClickHouse/pull/2950) -## ClickHouse Release 18.10 {#clickhouse-release-18-10} +## ClickHouse release 18.10 {#clickhouse-release-18-10} -### ClickHouse Release 18.10.3, 2018-08-13 {#clickhouse-release-18-10-3-2018-08-13} +### ClickHouse release 18.10.3, 2018-08-13 {#clickhouse-release-18-10-3-2018-08-13} -#### New Features: {#new-features-5} +#### New features: {#new-features-5} - HTTPS can be used for replication. [#2760](https://github.com/ClickHouse/ClickHouse/pull/2760) - Added the functions `murmurHash2_64`, `murmurHash3_32`, `murmurHash3_64`, and `murmurHash3_128` in addition to the existing `murmurHash2_32`. [#2791](https://github.com/ClickHouse/ClickHouse/pull/2791) @@ -478,7 +478,7 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Added the `prefer_localhost_replica` setting for disabling the preference for a local replica and going to a local replica without inter-process interaction. [#2832](https://github.com/ClickHouse/ClickHouse/pull/2832) - The `quantileExact` aggregate function returns `nan` in the case of aggregation on an empty `Float32` or `Float64` set. [Sundy Li](https://github.com/ClickHouse/ClickHouse/pull/2855) -#### Bug Fixes: {#bug-fixes-14} +#### Bug fixes: {#bug-fixes-14} - Removed unnecessary escaping of the connection string parameters for ODBC, which made it impossible to establish a connection. This error occurred in version 18.6.0. - Fixed the logic for processing `REPLACE PARTITION` commands in the replication queue. If there are two `REPLACE` commands for the same partition, the incorrect logic could cause one of them to remain in the replication queue and not be executed. [#2814](https://github.com/ClickHouse/ClickHouse/pull/2814) @@ -489,11 +489,11 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Fixed incorrect clickhouse-client response code in case of a query error. - Fixed incorrect behavior of materialized views containing DISTINCT. [#2795](https://github.com/ClickHouse/ClickHouse/issues/2795) -#### Backward Incompatible Changes {#backward-incompatible-changes-4} +#### Backward incompatible changes {#backward-incompatible-changes-4} - Removed support for CHECK TABLE queries for Distributed tables. -#### Build Changes: {#build-changes-3} +#### Build changes: {#build-changes-3} - The allocator has been replaced: `jemalloc` is now used instead of `tcmalloc`. In some scenarios, this increases speed up to 20%. However, there are queries that have slowed by up to 20%. Memory consumption has been reduced by approximately 10% in some scenarios, with improved stability. With highly competitive loads, CPU usage in userspace and in system shows just a slight increase. [#2773](https://github.com/ClickHouse/ClickHouse/pull/2773) - Use of libressl from a submodule. [#1983](https://github.com/ClickHouse/ClickHouse/pull/1983) [#2807](https://github.com/ClickHouse/ClickHouse/pull/2807) @@ -501,11 +501,11 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Use of mariadb-connector-c from a submodule. [#2785](https://github.com/ClickHouse/ClickHouse/pull/2785) - Added functional test files to the repository that depend on the availability of test data (for the time being, without the test data itself). -## ClickHouse Release 18.6 {#clickhouse-release-18-6} +## ClickHouse release 18.6 {#clickhouse-release-18-6} -### ClickHouse Release 18.6.0, 2018-08-02 {#clickhouse-release-18-6-0-2018-08-02} +### ClickHouse release 18.6.0, 2018-08-02 {#clickhouse-release-18-6-0-2018-08-02} -#### New Features: {#new-features-6} +#### New features: {#new-features-6} - Added support for ON expressions for the JOIN ON syntax: `JOIN ON Expr([table.]column ...) = Expr([table.]column, ...) [AND Expr([table.]column, ...) = Expr([table.]column, ...) ...]` @@ -516,11 +516,11 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - The server passes the patch component of its version to the client. Data about the patch version component is in `system.processes` and `query_log`. [#2646](https://github.com/ClickHouse/ClickHouse/pull/2646) -## ClickHouse Release 18.5 {#clickhouse-release-18-5} +## ClickHouse release 18.5 {#clickhouse-release-18-5} -### ClickHouse Release 18.5.1, 2018-07-31 {#clickhouse-release-18-5-1-2018-07-31} +### ClickHouse release 18.5.1, 2018-07-31 {#clickhouse-release-18-5-1-2018-07-31} -#### New Features: {#new-features-7} +#### New features: {#new-features-7} - Added the hash function `murmurHash2_32` [#2756](https://github.com/ClickHouse/ClickHouse/pull/2756). @@ -529,15 +529,15 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Now you can use the `from_env` [#2741](https://github.com/ClickHouse/ClickHouse/pull/2741) attribute to set values in config files from environment variables. - Added case-insensitive versions of the `coalesce`, `ifNull`, and `nullIf functions` [#2752](https://github.com/ClickHouse/ClickHouse/pull/2752). -#### Bug Fixes: {#bug-fixes-15} +#### Bug fixes: {#bug-fixes-15} - Fixed a possible bug when starting a replica [#2759](https://github.com/ClickHouse/ClickHouse/pull/2759). -## ClickHouse Release 18.4 {#clickhouse-release-18-4} +## ClickHouse release 18.4 {#clickhouse-release-18-4} -### ClickHouse Release 18.4.0, 2018-07-28 {#clickhouse-release-18-4-0-2018-07-28} +### ClickHouse release 18.4.0, 2018-07-28 {#clickhouse-release-18-4-0-2018-07-28} -#### New Features: {#new-features-8} +#### New features: {#new-features-8} - Added system tables: `formats`, `data_type_families`, `aggregate_function_combinators`, `table_functions`, `table_engines`, `collations` [#2721](https://github.com/ClickHouse/ClickHouse/pull/2721). - Added the ability to use a table function instead of a table as an argument of a `remote` or `cluster table function` [#2708](https://github.com/ClickHouse/ClickHouse/pull/2708). @@ -551,22 +551,22 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - The `use_minimalistic_checksums_in_zookeeper` option for `ReplicatedMergeTree` tables is enabled by default. This setting was added in version 1.1.54378, 2018-04-16. Versions that are older than 1.1.54378 can no longer be installed. - Support for running `KILL` and `OPTIMIZE` queries that specify `ON CLUSTER` [Winter Zhang](https://github.com/ClickHouse/ClickHouse/pull/2689). -#### Bug Fixes: {#bug-fixes-16} +#### Bug fixes: {#bug-fixes-16} - Fixed the error `Column ... is not under an aggregate function and not in GROUP BY` for aggregation with an IN expression. This bug appeared in version 18.1.0. ([bbdd780b](https://github.com/ClickHouse/ClickHouse/commit/bbdd780be0be06a0f336775941cdd536878dd2c2)) - Fixed a bug in the `windowFunnel aggregate function` [Winter Zhang](https://github.com/ClickHouse/ClickHouse/pull/2735). - Fixed a bug in the `anyHeavy` aggregate function ([a2101df2](https://github.com/ClickHouse/ClickHouse/commit/a2101df25a6a0fba99aa71f8793d762af2b801ee)) - Fixed server crash when using the `countArray()` aggregate function. -#### Backward Incompatible Changes: {#backward-incompatible-changes-5} +#### Backward incompatible changes: {#backward-incompatible-changes-5} - Parameters for `Kafka` engine was changed from `Kafka(kafka_broker_list, kafka_topic_list, kafka_group_name, kafka_format[, kafka_schema, kafka_num_consumers])` to `Kafka(kafka_broker_list, kafka_topic_list, kafka_group_name, kafka_format[, kafka_row_delimiter, kafka_schema, kafka_num_consumers])`. If your tables use `kafka_schema` or `kafka_num_consumers` parameters, you have to manually edit the metadata files `path/metadata/database/table.sql` and add `kafka_row_delimiter` parameter with `''` value. -## ClickHouse Release 18.1 {#clickhouse-release-18-1} +## ClickHouse release 18.1 {#clickhouse-release-18-1} -### ClickHouse Release 18.1.0, 2018-07-23 {#clickhouse-release-18-1-0-2018-07-23} +### ClickHouse release 18.1.0, 2018-07-23 {#clickhouse-release-18-1-0-2018-07-23} -#### New Features: {#new-features-9} +#### New features: {#new-features-9} - Support for the `ALTER TABLE t DELETE WHERE` query for non-replicated MergeTree tables ([#2634](https://github.com/ClickHouse/ClickHouse/pull/2634)). - Support for arbitrary types for the `uniq*` family of aggregate functions ([#2010](https://github.com/ClickHouse/ClickHouse/issues/2010)). @@ -581,7 +581,7 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Faster conversions of floating-point numbers to a string ([Amos Bird](https://github.com/ClickHouse/ClickHouse/pull/2664)). - If some rows were skipped during an insert due to parsing errors (this is possible with the `input_allow_errors_num` and `input_allow_errors_ratio` settings enabled), the number of skipped rows is now written to the server log ([Leonardo Cecchi](https://github.com/ClickHouse/ClickHouse/pull/2669)). -#### Bug Fixes: {#bug-fixes-17} +#### Bug fixes: {#bug-fixes-17} - Fixed the TRUNCATE command for temporary tables ([Amos Bird](https://github.com/ClickHouse/ClickHouse/pull/2624)). - Fixed a rare deadlock in the ZooKeeper client library that occurred when there was a network error while reading the response ([c315200](https://github.com/ClickHouse/ClickHouse/commit/c315200e64b87e44bdf740707fc857d1fdf7e947)). @@ -592,20 +592,20 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Fixed incompatibility between servers with different versions in distributed queries that use a `CAST` function that isn't in uppercase letters ([fe8c4d6](https://github.com/ClickHouse/ClickHouse/commit/fe8c4d64e434cacd4ceef34faa9005129f2190a5)). - Added missing quoting of identifiers for queries to an external DBMS ([#2635](https://github.com/ClickHouse/ClickHouse/issues/2635)). -#### Backward Incompatible Changes: {#backward-incompatible-changes-6} +#### Backward incompatible changes: {#backward-incompatible-changes-6} - Converting a string containing the number zero to DateTime does not work. Example: `SELECT toDateTime('0')`. This is also the reason that `DateTime DEFAULT '0'` does not work in tables, as well as `0` in dictionaries. Solution: replace `0` with `0000-00-00 00:00:00`. -## ClickHouse Release 1.1 {#clickhouse-release-1-1} +## ClickHouse release 1.1 {#clickhouse-release-1-1} -### ClickHouse Release 1.1.54394, 2018-07-12 {#clickhouse-release-1-1-54394-2018-07-12} +### ClickHouse release 1.1.54394, 2018-07-12 {#clickhouse-release-1-1-54394-2018-07-12} -#### New Features: {#new-features-10} +#### New features: {#new-features-10} - Added the `histogram` aggregate function ([Mikhail Surin](https://github.com/ClickHouse/ClickHouse/pull/2521)). - Now `OPTIMIZE TABLE ... FINAL` can be used without specifying partitions for `ReplicatedMergeTree` ([Amos Bird](https://github.com/ClickHouse/ClickHouse/pull/2600)). -#### Bug Fixes: {#bug-fixes-18} +#### Bug fixes: {#bug-fixes-18} - Fixed a problem with a very small timeout for sockets (one second) for reading and writing when sending and downloading replicated data, which made it impossible to download larger parts if there is a load on the network or disk (it resulted in cyclical attempts to download parts). This error occurred in version 1.1.54388. - Fixed issues when using chroot in ZooKeeper if you inserted duplicate data blocks in the table. @@ -614,9 +614,9 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Fixed how an empty `TinyLog` table works after inserting an empty data block ([#2563](https://github.com/ClickHouse/ClickHouse/issues/2563)). - The `system.zookeeper` table works if the value of the node in ZooKeeper is NULL. -### ClickHouse Release 1.1.54390, 2018-07-06 {#clickhouse-release-1-1-54390-2018-07-06} +### ClickHouse release 1.1.54390, 2018-07-06 {#clickhouse-release-1-1-54390-2018-07-06} -#### New Features: {#new-features-11} +#### New features: {#new-features-11} - Queries can be sent in `multipart/form-data` format (in the `query` field), which is useful if external data is also sent for query processing ([Olga Hvostikova](https://github.com/ClickHouse/ClickHouse/pull/2490)). - Added the ability to enable or disable processing single or double quotes when reading data in CSV format. You can configure this in the `format_csv_allow_single_quotes` and `format_csv_allow_double_quotes` settings ([Amos Bird](https://github.com/ClickHouse/ClickHouse/pull/2574)). @@ -632,7 +632,7 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Faster selection of data parts for merging in `ReplicatedMergeTree` tables. Faster recovery of the ZooKeeper session ([#2597](https://github.com/ClickHouse/ClickHouse/pull/2597)). - The `format_version.txt` file for `MergeTree` tables is re-created if it is missing, which makes sense if ClickHouse is launched after copying the directory structure without files ([Ciprian Hacman](https://github.com/ClickHouse/ClickHouse/pull/2593)). -#### Bug Fixes: {#bug-fixes-19} +#### Bug fixes: {#bug-fixes-19} - Fixed a bug when working with ZooKeeper that could make it impossible to recover the session and readonly states of tables before restarting the server. - Fixed a bug when working with ZooKeeper that could result in old nodes not being deleted if the session is interrupted. @@ -642,13 +642,13 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Fixed switching to the default database when reconnecting the client ([#2583](https://github.com/ClickHouse/ClickHouse/pull/2583)). - Fixed a bug that occurred when the `use_index_for_in_with_subqueries` setting was disabled. -#### Security Fix: {#security-fix-1} +#### Security fix: {#security-fix-1} - Sending files is no longer possible when connected to MySQL (`LOAD DATA LOCAL INFILE`). -### ClickHouse Release 1.1.54388, 2018-06-28 {#clickhouse-release-1-1-54388-2018-06-28} +### ClickHouse release 1.1.54388, 2018-06-28 {#clickhouse-release-1-1-54388-2018-06-28} -#### New Features: {#new-features-12} +#### New features: {#new-features-12} - Support for the `ALTER TABLE t DELETE WHERE` query for replicated tables. Added the `system.mutations` table to track progress of this type of queries. - Support for the `ALTER TABLE t [REPLACE|ATTACH] PARTITION` query for \*MergeTree tables. @@ -666,12 +666,12 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Added the `date_time_input_format` setting. If you switch this setting to `'best_effort'`, DateTime values will be read in a wide range of formats. - Added the `clickhouse-obfuscator` utility for data obfuscation. Usage example: publishing data used in performance tests. -#### Experimental Features: {#experimental-features-2} +#### Experimental features: {#experimental-features-2} - Added the ability to calculate `and` arguments only where they are needed ([Anastasia Tsarkova](https://github.com/ClickHouse/ClickHouse/pull/2272)) - JIT compilation to native code is now available for some expressions ([pyos](https://github.com/ClickHouse/ClickHouse/pull/2277)). -#### Bug Fixes: {#bug-fixes-20} +#### Bug fixes: {#bug-fixes-20} - Duplicates no longer appear for a query with `DISTINCT` and `ORDER BY`. - Queries with `ARRAY JOIN` and `arrayFilter` no longer return an incorrect result. @@ -706,7 +706,7 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - When calculating the number of available CPU cores, limits on cgroups are now taken into account ([Atri Sharma](https://github.com/ClickHouse/ClickHouse/pull/2325)). - Added chown for config directories in the systemd config file ([Mikhail Shiryaev](https://github.com/ClickHouse/ClickHouse/pull/2421)). -#### Build Changes: {#build-changes-4} +#### Build changes: {#build-changes-4} - The gcc8 compiler can be used for builds. - Added the ability to build llvm from submodule. @@ -717,32 +717,32 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Added the ability to use the libtinfo library instead of libtermcap ([Georgy Kondratiev](https://github.com/ClickHouse/ClickHouse/pull/2519)). - Fixed a header file conflict in Fedora Rawhide ([#2520](https://github.com/ClickHouse/ClickHouse/issues/2520)). -#### Backward Incompatible Changes: {#backward-incompatible-changes-7} +#### Backward incompatible changes: {#backward-incompatible-changes-7} - Removed escaping in `Vertical` and `Pretty*` formats and deleted the `VerticalRaw` format. - If servers with version 1.1.54388 (or newer) and servers with an older version are used simultaneously in a distributed query and the query has the `cast(x, 'Type')` expression without the `AS` keyword and does not have the word `cast` in uppercase, an exception will be thrown with a message like `Not found column cast(0, 'UInt8') in block`. Solution: Update the server on the entire cluster. -### ClickHouse Release 1.1.54385, 2018-06-01 {#clickhouse-release-1-1-54385-2018-06-01} +### ClickHouse release 1.1.54385, 2018-06-01 {#clickhouse-release-1-1-54385-2018-06-01} -#### Bug Fixes: {#bug-fixes-21} +#### Bug fixes: {#bug-fixes-21} - Fixed an error that in some cases caused ZooKeeper operations to block. -### ClickHouse Release 1.1.54383, 2018-05-22 {#clickhouse-release-1-1-54383-2018-05-22} +### ClickHouse release 1.1.54383, 2018-05-22 {#clickhouse-release-1-1-54383-2018-05-22} -#### Bug Fixes: {#bug-fixes-22} +#### Bug fixes: {#bug-fixes-22} - Fixed a slowdown of replication queue if a table has many replicas. -### ClickHouse Release 1.1.54381, 2018-05-14 {#clickhouse-release-1-1-54381-2018-05-14} +### ClickHouse release 1.1.54381, 2018-05-14 {#clickhouse-release-1-1-54381-2018-05-14} -#### Bug Fixes: {#bug-fixes-23} +#### Bug fixes: {#bug-fixes-23} - Fixed a nodes leak in ZooKeeper when ClickHouse loses connection to ZooKeeper server. -### ClickHouse Release 1.1.54380, 2018-04-21 {#clickhouse-release-1-1-54380-2018-04-21} +### ClickHouse release 1.1.54380, 2018-04-21 {#clickhouse-release-1-1-54380-2018-04-21} -#### New Features: {#new-features-13} +#### New features: {#new-features-13} - Added the table function `file(path, format, structure)`. An example reading bytes from `/dev/urandom`: ``` ln -s /dev/urandom /var/lib/clickhouse/user_files/random``clickhouse-client -q "SELECT * FROM file('random', 'RowBinary', 'd UInt8') LIMIT 10" ```. @@ -751,7 +751,7 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Subqueries can be wrapped in `()` brackets to enhance query readability. For example: `(SELECT 1) UNION ALL (SELECT 1)`. - Simple `SELECT` queries from the `system.processes` table are not included in the `max_concurrent_queries` limit. -#### Bug Fixes: {#bug-fixes-24} +#### Bug fixes: {#bug-fixes-24} - Fixed incorrect behavior of the `IN` operator when select from `MATERIALIZED VIEW`. - Fixed incorrect filtering by partition index in expressions like `partition_key_column IN (...)`. @@ -760,13 +760,13 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Fixed freezing of `KILL QUERY`. - Fixed an error in ZooKeeper client library which led to loss of watches, freezing of distributed DDL queue, and slowdowns in the replication queue if a non-empty `chroot` prefix is used in the ZooKeeper configuration. -#### Backward Incompatible Changes: {#backward-incompatible-changes-8} +#### Backward incompatible changes: {#backward-incompatible-changes-8} - Removed support for expressions like `(a, b) IN (SELECT (a, b))` (you can use the equivalent expression `(a, b) IN (SELECT a, b)`). In previous releases, these expressions led to undetermined `WHERE` filtering or caused errors. -### ClickHouse Release 1.1.54378, 2018-04-16 {#clickhouse-release-1-1-54378-2018-04-16} +### ClickHouse release 1.1.54378, 2018-04-16 {#clickhouse-release-1-1-54378-2018-04-16} -#### New Features: {#new-features-14} +#### New features: {#new-features-14} - Logging level can be changed without restarting the server. - Added the `SHOW CREATE DATABASE` query. @@ -791,7 +791,7 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - More robust crash recovery for asynchronous insertion into `Distributed` tables. - The return type of the `countEqual` function changed from `UInt32` to `UInt64` (谢磊). -#### Bug Fixes: {#bug-fixes-25} +#### Bug fixes: {#bug-fixes-25} - Fixed an error with `IN` when the left side of the expression is `Nullable`. - Correct results are now returned when using tuples with `IN` when some of the tuple components are in the table index. @@ -807,19 +807,19 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - `SummingMergeTree` now works correctly for summation of nested data structures with a composite key. - Fixed the possibility of a race condition when choosing the leader for `ReplicatedMergeTree` tables. -#### Build Changes: {#build-changes-5} +#### Build changes: {#build-changes-5} - The build supports `ninja` instead of `make` and uses `ninja` by default for building releases. - Renamed packages: `clickhouse-server-base` in `clickhouse-common-static`; `clickhouse-server-common` in `clickhouse-server`; `clickhouse-common-dbg` in `clickhouse-common-static-dbg`. To install, use `clickhouse-server clickhouse-client`. Packages with the old names will still load in the repositories for backward compatibility. -#### Backward Incompatible Changes: {#backward-incompatible-changes-9} +#### Backward incompatible changes: {#backward-incompatible-changes-9} - Removed the special interpretation of an IN expression if an array is specified on the left side. Previously, the expression `arr IN (set)` was interpreted as "at least one `arr` element belongs to the `set`". To get the same behavior in the new version, write `arrayExists(x -> x IN (set), arr)`. - Disabled the incorrect use of the socket option `SO_REUSEPORT`, which was incorrectly enabled by default in the Poco library. Note that on Linux there is no longer any reason to simultaneously specify the addresses `::` and `0.0.0.0` for listen – use just `::`, which allows listening to the connection both over IPv4 and IPv6 (with the default kernel config settings). You can also revert to the behavior from previous versions by specifying `1` in the config. -### ClickHouse Release 1.1.54370, 2018-03-16 {#clickhouse-release-1-1-54370-2018-03-16} +### ClickHouse release 1.1.54370, 2018-03-16 {#clickhouse-release-1-1-54370-2018-03-16} -#### New Features: {#new-features-15} +#### New features: {#new-features-15} - Added the `system.macros` table and auto updating of macros when the config file is changed. - Added the `SYSTEM RELOAD CONFIG` query. @@ -830,7 +830,7 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - When inserting data in a `Replicated` table, fewer requests are made to `ZooKeeper` (and most of the user-level errors have disappeared from the `ZooKeeper` log). - Added the ability to create aliases for data sets. Example: `WITH (1, 2, 3) AS set SELECT number IN set FROM system.numbers LIMIT 10`. -#### Bug Fixes: {#bug-fixes-26} +#### Bug fixes: {#bug-fixes-26} - Fixed the `Illegal PREWHERE` error when reading from Merge tables for `Distributed`tables. - Added fixes that allow you to start clickhouse-server in IPv4-only Docker containers. @@ -844,9 +844,9 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Restored the behavior for queries like `SELECT * FROM remote('server2', default.table) WHERE col IN (SELECT col2 FROM default.table)` when the right side of the `IN` should use a remote `default.table` instead of a local one. This behavior was broken in version 1.1.54358. - Removed extraneous error-level logging of `Not found column ... in block`. -### ClickHouse Release 1.1.54362, 2018-03-11 {#clickhouse-release-1-1-54362-2018-03-11} +### ClickHouse release 1.1.54362, 2018-03-11 {#clickhouse-release-1-1-54362-2018-03-11} -#### New Features: {#new-features-16} +#### New features: {#new-features-16} - Aggregation without `GROUP BY` for an empty set (such as `SELECT count(*) FROM table WHERE 0`) now returns a result with one row with null values for aggregate functions, in compliance with the SQL standard. To restore the old behavior (return an empty result), set `empty_result_for_aggregation_by_empty_set` to 1. - Added type conversion for `UNION ALL`. Different alias names are allowed in `SELECT` positions in `UNION ALL`, in compliance with the SQL standard. @@ -900,7 +900,7 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - `Enum` values can be used in `min`, `max`, `sum` and some other functions. In these cases, it uses the corresponding numeric values. This feature was previously available but was lost in the release 1.1.54337. - Added `max_expanded_ast_elements` to restrict the size of the AST after recursively expanding aliases. -#### Bug Fixes: {#bug-fixes-27} +#### Bug fixes: {#bug-fixes-27} - Fixed cases when unnecessary columns were removed from subqueries in error, or not removed from subqueries containing `UNION ALL`. - Fixed a bug in merges for `ReplacingMergeTree` tables. @@ -928,19 +928,19 @@ keywords: ['ClickHouse 2018', 'changelog 2018', 'release notes', 'version histor - Prohibited the use of queries with `UNION ALL` in a `MATERIALIZED VIEW`. - Fixed an error during initialization of the `part_log` system table when the server starts (by default, `part_log` is disabled). -#### Backward Incompatible Changes: {#backward-incompatible-changes-10} +#### Backward incompatible changes: {#backward-incompatible-changes-10} - Removed the `distributed_ddl_allow_replicated_alter` option. This behavior is enabled by default. - Removed the `strict_insert_defaults` setting. If you were using this functionality, write to `feedback@clickhouse.com`. - Removed the `UnsortedMergeTree` engine. -### ClickHouse Release 1.1.54343, 2018-02-05 {#clickhouse-release-1-1-54343-2018-02-05} +### ClickHouse release 1.1.54343, 2018-02-05 {#clickhouse-release-1-1-54343-2018-02-05} - Added macros support for defining cluster names in distributed DDL queries and constructors of Distributed tables: `CREATE TABLE distr ON CLUSTER '{cluster}' (...) ENGINE = Distributed('{cluster}', 'db', 'table')`. - Now queries like `SELECT ... FROM table WHERE expr IN (subquery)` are processed using the `table` index. - Improved processing of duplicates when inserting to Replicated tables, so they no longer slow down execution of the replication queue. -### ClickHouse Release 1.1.54342, 2018-01-22 {#clickhouse-release-1-1-54342-2018-01-22} +### ClickHouse release 1.1.54342, 2018-01-22 {#clickhouse-release-1-1-54342-2018-01-22} This release contains bug fixes for the previous release 1.1.54337: @@ -952,9 +952,9 @@ This release contains bug fixes for the previous release 1.1.54337: - Buffer tables now work correctly when MATERIALIZED columns are present in the destination table (by zhang2014). - Fixed a bug in implementation of NULL. -### ClickHouse Release 1.1.54337, 2018-01-18 {#clickhouse-release-1-1-54337-2018-01-18} +### ClickHouse release 1.1.54337, 2018-01-18 {#clickhouse-release-1-1-54337-2018-01-18} -#### New Features: {#new-features-17} +#### New features: {#new-features-17} - Added support for storage of multi-dimensional arrays and tuples (`Tuple` data type) in tables. - Support for table functions for `DESCRIBE` and `INSERT` queries. Added support for subqueries in `DESCRIBE`. Examples: `DESC TABLE remote('host', default.hits)`; `DESC TABLE (SELECT 1)`; `INSERT INTO TABLE FUNCTION remote('host', default.hits)`. Support for `INSERT INTO TABLE` in addition to `INSERT INTO`. @@ -985,7 +985,7 @@ This release contains bug fixes for the previous release 1.1.54337: - Added the `--silent` option for the `clickhouse-local` tool. It suppresses printing query execution info in stderr. - Added support for reading values of type `Date` from text in a format where the month and/or day of the month is specified using a single digit instead of two digits (Amos Bird). -#### Performance Optimizations: {#performance-optimizations} +#### Performance optimizations: {#performance-optimizations} - Improved performance of aggregate functions `min`, `max`, `any`, `anyLast`, `anyHeavy`, `argMin`, `argMax` from string arguments. - Improved performance of the functions `isInfinite`, `isFinite`, `isNaN`, `roundToExp2`. @@ -994,7 +994,7 @@ This release contains bug fixes for the previous release 1.1.54337: - Lowered memory usage for `JOIN` in the case when the left and right parts have columns with identical names that are not contained in `USING` . - Improved performance of aggregate functions `varSamp`, `varPop`, `stddevSamp`, `stddevPop`, `covarSamp`, `covarPop`, `corr` by reducing computational stability. The old functions are available under the names `varSampStable`, `varPopStable`, `stddevSampStable`, `stddevPopStable`, `covarSampStable`, `covarPopStable`, `corrStable`. -#### Bug Fixes: {#bug-fixes-28} +#### Bug fixes: {#bug-fixes-28} - Fixed data deduplication after running a `DROP` or `DETACH PARTITION` query. In the previous version, dropping a partition and inserting the same data again was not working because inserted blocks were considered duplicates. - Fixed a bug that could lead to incorrect interpretation of the `WHERE` clause for `CREATE MATERIALIZED VIEW` queries with `POPULATE` . @@ -1033,7 +1033,7 @@ This release contains bug fixes for the previous release 1.1.54337: - Fixed the `SYSTEM DROP DNS CACHE` query: the cache was flushed but addresses of cluster nodes were not updated. - Fixed the behavior of `MATERIALIZED VIEW` after executing `DETACH TABLE` for the table under the view (Marek Vavruša). -#### Build Improvements: {#build-improvements-4} +#### Build improvements: {#build-improvements-4} - The `pbuilder` tool is used for builds. The build process is almost completely independent of the build host environment. - A single build is used for different OS versions. Packages and binaries have been made compatible with a wide range of Linux systems. @@ -1047,7 +1047,7 @@ This release contains bug fixes for the previous release 1.1.54337: - Removed usage of GNU extensions from the code. Enabled the `-Wextra` option. When building with `clang` the default is `libc++` instead of `libstdc++`. - Extracted `clickhouse_parsers` and `clickhouse_common_io` libraries to speed up builds of various tools. -#### Backward Incompatible Changes: {#backward-incompatible-changes-11} +#### Backward incompatible changes: {#backward-incompatible-changes-11} - The format for marks in `Log` type tables that contain `Nullable` columns was changed in a backward incompatible way. If you have these tables, you should convert them to the `TinyLog` type before starting up the new server version. To do this, replace `ENGINE = Log` with `ENGINE = TinyLog` in the corresponding `.sql` file in the `metadata` directory. If your table does not have `Nullable` columns or if the type of your table is not `Log`, then you do not need to do anything. - Removed the `experimental_allow_extended_storage_definition_syntax` setting. Now this feature is enabled by default. @@ -1058,7 +1058,7 @@ This release contains bug fixes for the previous release 1.1.54337: - In previous server versions there was an undocumented feature: if an aggregate function depends on parameters, you can still specify it without parameters in the AggregateFunction data type. Example: `AggregateFunction(quantiles, UInt64)` instead of `AggregateFunction(quantiles(0.5, 0.9), UInt64)`. This feature was lost. Although it was undocumented, we plan to support it again in future releases. - Enum data types cannot be used in min/max aggregate functions. This ability will be returned in the next release. -#### Please Note When Upgrading: {#please-note-when-upgrading} +#### Please note when upgrading: {#please-note-when-upgrading} - When doing a rolling update on a cluster, at the point when some of the replicas are running the old version of ClickHouse and some are running the new version, replication is temporarily stopped and the message `unknown parameter 'shard'` appears in the log. Replication will continue after all replicas of the cluster are updated. - If different versions of ClickHouse are running on the cluster servers, it is possible that distributed queries using the following functions will have incorrect results: `varSamp`, `varPop`, `stddevSamp`, `stddevPop`, `covarSamp`, `covarPop`, `corr`. You should update all cluster nodes. diff --git a/docs/whats-new/changelog/2019.md b/docs/whats-new/changelog/2019.md index da35397b1b0..de06e12e443 100644 --- a/docs/whats-new/changelog/2019.md +++ b/docs/whats-new/changelog/2019.md @@ -8,11 +8,11 @@ doc_type: 'changelog' keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version history', 'legacy releases'] --- -## ClickHouse Release 19.17 {#clickhouse-release-v19-17} +## ClickHouse release 19.17 {#clickhouse-release-v19-17} -### ClickHouse Release 19.17.6.36, 2019-12-27 {#clickhouse-release-v19-17-6-36-2019-12-27} +### ClickHouse release 19.17.6.36, 2019-12-27 {#clickhouse-release-v19-17-6-36-2019-12-27} -#### Bug Fix {#bug-fix} +#### Bug fix {#bug-fix} - Fixed potential buffer overflow in decompress. Malicious user can pass fabricated compressed data that could cause read after buffer. This issue was found by Eldar Zaitov from Yandex information security team. [#8404](https://github.com/ClickHouse/ClickHouse/pull/8404) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Fixed possible server crash (`std::terminate`) when the server cannot send or write data in JSON or XML format with values of String data type (that require UTF-8 validation) or when compressing result data with Brotli algorithm or in some other rare cases. [#8384](https://github.com/ClickHouse/ClickHouse/pull/8384) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -45,13 +45,13 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Now an exception will be thrown in case of using WITH TIES alongside LIMIT BY. And now it's possible to use TOP with LIMIT BY. [#7637](https://github.com/ClickHouse/ClickHouse/pull/7637) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)) - Fix dictionary reload if it has `invalidate_query`, which stopped updates and some exception on previous update tries. [#8029](https://github.com/ClickHouse/ClickHouse/pull/8029) ([alesapin](https://github.com/alesapin)) -### ClickHouse Release 19.17.4.11, 2019-11-22 {#clickhouse-release-v19-17-4-11-2019-11-22} +### ClickHouse release 19.17.4.11, 2019-11-22 {#clickhouse-release-v19-17-4-11-2019-11-22} -#### Backward Incompatible Change {#backward-incompatible-change} +#### Backward incompatible change {#backward-incompatible-change} - Using column instead of AST to store scalar subquery results for better performance. Setting `enable_scalar_subquery_optimization` was added in 19.17 and it was enabled by default. It leads to errors like [this](https://github.com/ClickHouse/ClickHouse/issues/7851) during upgrade to 19.17.2 or 19.17.3 from previous versions. This setting was disabled by default in 19.17.4, to make possible upgrading from 19.16 and older versions without errors. [#7392](https://github.com/ClickHouse/ClickHouse/pull/7392) ([Amos Bird](https://github.com/amosbird)) -#### New Feature {#new-feature} +#### New feature {#new-feature} - Add the ability to create dictionaries with DDL queries. [#7360](https://github.com/ClickHouse/ClickHouse/pull/7360) ([alesapin](https://github.com/alesapin)) - Make `bloom_filter` type of index supporting `LowCardinality` and `Nullable` [#7363](https://github.com/ClickHouse/ClickHouse/issues/7363) [#7561](https://github.com/ClickHouse/ClickHouse/pull/7561) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) @@ -65,11 +65,11 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Implemented `javaHashUTF16LE()` function [#7651](https://github.com/ClickHouse/ClickHouse/pull/7651) ([achimbab](https://github.com/achimbab)) - Add `_shard_num` virtual column for the Distributed engine [#7624](https://github.com/ClickHouse/ClickHouse/pull/7624) ([Azat Khuzhin](https://github.com/azat)) -#### Experimental Feature {#experimental-feature} +#### Experimental feature {#experimental-feature} - Support for processors (new query execution pipeline) in `MergeTree`. [#7181](https://github.com/ClickHouse/ClickHouse/pull/7181) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) -#### Bug Fix {#bug-fix-1} +#### Bug fix {#bug-fix-1} - Fix incorrect float parsing in `Values` [#7817](https://github.com/ClickHouse/ClickHouse/issues/7817) [#7870](https://github.com/ClickHouse/ClickHouse/pull/7870) ([tavplubix](https://github.com/tavplubix)) - Fix rare deadlock which can happen when trace_log is enabled. [#7838](https://github.com/ClickHouse/ClickHouse/pull/7838) ([filimonov](https://github.com/filimonov)) @@ -107,7 +107,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Better Null format for tcp handler, so that it's possible to use `select ignore() from table format Null` for perf measure via clickhouse-client [#7606](https://github.com/ClickHouse/ClickHouse/pull/7606) ([Amos Bird](https://github.com/amosbird)) - Queries like `CREATE TABLE ... AS (SELECT (1, 2))` are parsed correctly [#7542](https://github.com/ClickHouse/ClickHouse/pull/7542) ([hcz](https://github.com/hczhcz)) -#### Performance Improvement {#performance-improvement} +#### Performance improvement {#performance-improvement} - The performance of aggregation over short string keys is improved. [#6243](https://github.com/ClickHouse/ClickHouse/pull/6243) ([Alexander Kuzmenkov](https://github.com/akuzm), [Amos Bird](https://github.com/amosbird)) - Run another pass of syntax/expression analysis to get potential optimizations after constant predicates are folded. [#7497](https://github.com/ClickHouse/ClickHouse/pull/7497) ([Amos Bird](https://github.com/amosbird)) @@ -115,7 +115,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Vectorize processing `arrayReduce` similar to Aggregator `addBatch`. [#7608](https://github.com/ClickHouse/ClickHouse/pull/7608) ([Amos Bird](https://github.com/amosbird)) - Minor improvements in performance of `Kafka` consumption [#7475](https://github.com/ClickHouse/ClickHouse/pull/7475) ([Ivan](https://github.com/abyss7)) -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement} - Add support for cross-compiling to the CPU architecture AARCH64. Refactor packager script. [#7370](https://github.com/ClickHouse/ClickHouse/pull/7370) [#7539](https://github.com/ClickHouse/ClickHouse/pull/7539) ([Ivan](https://github.com/abyss7)) - Unpack darwin-x86_64 and linux-aarch64 toolchains into mounted Docker volume when building packages [#7534](https://github.com/ClickHouse/ClickHouse/pull/7534) ([Ivan](https://github.com/abyss7)) @@ -131,13 +131,13 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Added ANTLR4 grammar for ClickHouse SQL dialect [#7595](https://github.com/ClickHouse/ClickHouse/issues/7595) [#7596](https://github.com/ClickHouse/ClickHouse/pull/7596) ([alexey-milovidov](https://github.com/alexey-milovidov)) -## ClickHouse Release 19.16 {#clickhouse-release-v19-16} +## ClickHouse release 19.16 {#clickhouse-release-v19-16} -#### ClickHouse Release 19.16.14.65, 2020-03-25 {#clickhouse-release-v19-16-14-65-2020-03-25} +#### ClickHouse release 19.16.14.65, 2020-03-25 {#clickhouse-release-v19-16-14-65-2020-03-25} - Fixed up a bug in batched calculations of ternary logical OPs on multiple arguments (more than 10). [#8718](https://github.com/ClickHouse/ClickHouse/pull/8718) ([Alexander Kazakov](https://github.com/Akazz)) This bugfix was backported to version 19.16 by a special request from Altinity. -#### ClickHouse Release 19.16.14.65, 2020-03-05 {#clickhouse-release-v19-16-14-65-2020-03-05} +#### ClickHouse release 19.16.14.65, 2020-03-05 {#clickhouse-release-v19-16-14-65-2020-03-05} - Fix distributed subqueries incompatibility with older CH versions. Fixes [#7851](https://github.com/ClickHouse/ClickHouse/issues/7851) [(tabplubix)](https://github.com/tavplubix) @@ -152,14 +152,14 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Allow using `MaterializedView` with subqueries above `Kafka` tables. [#8197](https://github.com/ClickHouse/ClickHouse/pull/8197) ([filimonov](https://github.com/filimonov)) -#### New Feature {#new-feature-1} +#### New feature {#new-feature-1} - Add `deduplicate_blocks_in_dependent_materialized_views` option to control the behaviour of idempotent inserts into tables with materialized views. This new feature was added to the bugfix release by a special request from Altinity. [#9070](https://github.com/ClickHouse/ClickHouse/pull/9070) [(urykhy)](https://github.com/urykhy) -### ClickHouse Release 19.16.2.2, 2019-10-30 {#clickhouse-release-v19-16-2-2-2019-10-30} +### ClickHouse release 19.16.2.2, 2019-10-30 {#clickhouse-release-v19-16-2-2-2019-10-30} -#### Backward Incompatible Change {#backward-incompatible-change-1} +#### Backward incompatible change {#backward-incompatible-change-1} - Add missing arity validation for count/counIf. [#7095](https://github.com/ClickHouse/ClickHouse/issues/7095) @@ -171,7 +171,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor [#7118](https://github.com/ClickHouse/ClickHouse/pull/7118) ([tavplubix](https://github.com/tavplubix)) -#### New Feature {#new-feature-2} +#### New feature {#new-feature-2} - Introduce uniqCombined64() to calculate cardinality greater than UINT_MAX. [#7213](https://github.com/ClickHouse/ClickHouse/pull/7213), @@ -213,7 +213,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Support Redis as source of external dictionary. [#4361](https://github.com/ClickHouse/ClickHouse/pull/4361) [#6962](https://github.com/ClickHouse/ClickHouse/pull/6962) ([comunodi](https://github.com/comunodi), [Anton Popov](https://github.com/CurtizJ)) -#### Bug Fix {#bug-fix-2} +#### Bug fix {#bug-fix-2} - Fix wrong query result if it has `WHERE IN (SELECT ...)` section and `optimize_read_in_order` is used. [#7371](https://github.com/ClickHouse/ClickHouse/pull/7371) ([Anton @@ -299,7 +299,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor memory). Load data back when needed. [#7186](https://github.com/ClickHouse/ClickHouse/pull/7186) ([Artem Zuikov](https://github.com/4ertus2)) -#### Performance Improvement {#performance-improvement-1} +#### Performance improvement {#performance-improvement-1} - Speed up joinGet with const arguments by avoiding data duplication. [#7359](https://github.com/ClickHouse/ClickHouse/pull/7359) ([Amos @@ -310,7 +310,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor [#6781](https://github.com/ClickHouse/ClickHouse/pull/6781) ([tavplubix](https://github.com/tavplubix)) -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-1} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-1} - Disable some contribs for cross-compilation to Mac OS. [#7101](https://github.com/ClickHouse/ClickHouse/pull/7101) ([Ivan](https://github.com/abyss7)) @@ -379,7 +379,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor [#7063](https://github.com/ClickHouse/ClickHouse/pull/7063) ([proller](https://github.com/proller)) -#### Code Cleanup {#code-cleanup} +#### Code cleanup {#code-cleanup} - Generalize configuration repository to prepare for DDL for Dictionaries. [#7155](https://github.com/ClickHouse/ClickHouse/pull/7155) ([alesapin](https://github.com/alesapin)) @@ -414,11 +414,11 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor [#7350](https://github.com/ClickHouse/ClickHouse/pull/7350) ([tavplubix](https://github.com/tavplubix)) -## ClickHouse Release 19.15 {#clickhouse-release-19-15} +## ClickHouse release 19.15 {#clickhouse-release-19-15} -### ClickHouse Release 19.15.4.10, 2019-10-31 {#clickhouse-release-19-15-4-10-2019-10-31} +### ClickHouse release 19.15.4.10, 2019-10-31 {#clickhouse-release-19-15-4-10-2019-10-31} -#### Bug Fix {#bug-fix-3} +#### Bug fix {#bug-fix-3} - Added handling of SQL_TINYINT and SQL_BIGINT, and fix handling of SQL_FLOAT data source types in ODBC Bridge. [#7491](https://github.com/ClickHouse/ClickHouse/pull/7491) ([Denis Glazachev](https://github.com/traceon)) @@ -444,9 +444,9 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor [#7158](https://github.com/ClickHouse/ClickHouse/pull/7158) ([Azat Khuzhin](https://github.com/azat)) - Added example config with macros for tests ([alexey-milovidov](https://github.com/alexey-milovidov)) -### ClickHouse Release 19.15.3.6, 2019-10-09 {#clickhouse-release-19-15-3-6-2019-10-09} +### ClickHouse release 19.15.3.6, 2019-10-09 {#clickhouse-release-19-15-3-6-2019-10-09} -#### Bug Fix {#bug-fix-4} +#### Bug fix {#bug-fix-4} - Fixed bad_variant in hashed dictionary. ([alesapin](https://github.com/alesapin)) @@ -459,9 +459,9 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Serialize NULL values correctly in min/max indexes of MergeTree parts. [#7234](https://github.com/ClickHouse/ClickHouse/pull/7234) ([Alexander Kuzmenkov](https://github.com/akuzm)) -### ClickHouse Release 19.15.2.2, 2019-10-01 {#clickhouse-release-19-15-2-2-2019-10-01} +### ClickHouse release 19.15.2.2, 2019-10-01 {#clickhouse-release-19-15-2-2-2019-10-01} -#### New Feature {#new-feature-3} +#### New feature {#new-feature-3} - Tiered storage: support to use multiple storage volumes for tables with MergeTree engine. It's possible to store fresh data on SSD and automatically move old data to HDD. ([example](https://clickhouse.github.io/clickhouse-presentations/meetup30/new_features/#12)). [#4918](https://github.com/ClickHouse/ClickHouse/pull/4918) ([Igr](https://github.com/ObjatieGroba)) [#6489](https://github.com/ClickHouse/ClickHouse/pull/6489) ([alesapin](https://github.com/alesapin)) - Add table function `input` for reading incoming data in `INSERT SELECT` query. [#5450](https://github.com/ClickHouse/ClickHouse/pull/5450) ([palasonic1](https://github.com/palasonic1)) [#6832](https://github.com/ClickHouse/ClickHouse/pull/6832) ([Anton Popov](https://github.com/CurtizJ)) @@ -472,7 +472,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Add `bitmapMin` and `bitmapMax` functions. [#6970](https://github.com/ClickHouse/ClickHouse/pull/6970) ([Zhichang Yu](https://github.com/yuzhichang)) - Add function `repeat` related to [issue-6648](https://github.com/ClickHouse/ClickHouse/issues/6648) [#6999](https://github.com/ClickHouse/ClickHouse/pull/6999) ([flynn](https://github.com/ucasFL)) -#### Experimental Feature {#experimental-feature-1} +#### Experimental feature {#experimental-feature-1} - Implement (in memory) Merge Join variant that does not change current pipeline. Result is partially sorted by merge key. Set `partial_merge_join = 1` to use this feature. The Merge Join is still in development. [#6940](https://github.com/ClickHouse/ClickHouse/pull/6940) ([Artem Zuikov](https://github.com/4ertus2)) - Add `S3` engine and table function. It is still in development (no authentication support yet). [#5596](https://github.com/ClickHouse/ClickHouse/pull/5596) ([Vladimir Chebotarev](https://github.com/excitoon)) @@ -489,7 +489,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Add automatically cast type `T` to `LowCardinality(T)` while inserting data in column of type `LowCardinality(T)` in Native format via HTTP. [#6891](https://github.com/ClickHouse/ClickHouse/pull/6891) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) - Add ability to use function `hex` without using `reinterpretAsString` for `Float32`, `Float64`. [#7024](https://github.com/ClickHouse/ClickHouse/pull/7024) ([Mikhail Korotov](https://github.com/millb)) -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-2} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-2} - Add gdb-index to clickhouse binary with debug info. It will speed up startup time of `gdb`. [#6947](https://github.com/ClickHouse/ClickHouse/pull/6947) ([alesapin](https://github.com/alesapin)) - Speed up deb packaging with patched dpkg-deb which uses `pigz`. [#6960](https://github.com/ClickHouse/ClickHouse/pull/6960) ([alesapin](https://github.com/alesapin)) @@ -498,7 +498,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Add build with MemorySanitizer to CI. [#7066](https://github.com/ClickHouse/ClickHouse/pull/7066) ([Alexander Kuzmenkov](https://github.com/akuzm)) - Replace `libsparsehash` with `sparsehash-c11` [#6965](https://github.com/ClickHouse/ClickHouse/pull/6965) ([Azat Khuzhin](https://github.com/azat)) -#### Bug Fix {#bug-fix-5} +#### Bug fix {#bug-fix-5} - Fixed performance degradation of index analysis on complex keys on large tables. This fixes #6924. [#7075](https://github.com/ClickHouse/ClickHouse/pull/7075) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Fix logical error causing segfaults when selecting from Kafka empty topic. [#6909](https://github.com/ClickHouse/ClickHouse/pull/6909) ([Ivan](https://github.com/abyss7)) @@ -510,34 +510,34 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Fix `Unknown identifier` error in ORDER BY and GROUP BY with multiple JOINs [#7022](https://github.com/ClickHouse/ClickHouse/pull/7022) ([Artem Zuikov](https://github.com/4ertus2)) - Fixed `MSan` warning while executing function with `LowCardinality` argument. [#7062](https://github.com/ClickHouse/ClickHouse/pull/7062) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) -#### Backward Incompatible Change {#backward-incompatible-change-2} +#### Backward incompatible change {#backward-incompatible-change-2} - Changed serialization format of bitmap\* aggregate function states to improve performance. Serialized states of bitmap\* from previous versions cannot be read. [#6908](https://github.com/ClickHouse/ClickHouse/pull/6908) ([Zhichang Yu](https://github.com/yuzhichang)) -## ClickHouse Release 19.14 {#clickhouse-release-19-14} +## ClickHouse release 19.14 {#clickhouse-release-19-14} -### ClickHouse Release 19.14.7.15, 2019-10-02 {#clickhouse-release-19-14-7-15-2019-10-02} +### ClickHouse release 19.14.7.15, 2019-10-02 {#clickhouse-release-19-14-7-15-2019-10-02} -#### Bug Fix {#bug-fix-6} +#### Bug fix {#bug-fix-6} - This release also contains all bug fixes from 19.11.12.69. - Fixed compatibility for distributed queries between 19.14 and earlier versions. This fixes [#7068](https://github.com/ClickHouse/ClickHouse/issues/7068). [#7069](https://github.com/ClickHouse/ClickHouse/pull/7069) ([alexey-milovidov](https://github.com/alexey-milovidov)) -### ClickHouse Release 19.14.6.12, 2019-09-19 {#clickhouse-release-19-14-6-12-2019-09-19} +### ClickHouse release 19.14.6.12, 2019-09-19 {#clickhouse-release-19-14-6-12-2019-09-19} -#### Bug Fix {#bug-fix-7} +#### Bug fix {#bug-fix-7} - Fix for function `АrrayEnumerateUniqRanked` with empty arrays in params. [#6928](https://github.com/ClickHouse/ClickHouse/pull/6928) ([proller](https://github.com/proller)) - Fixed subquery name in queries with `ARRAY JOIN` and `GLOBAL IN subquery` with alias. Use subquery alias for external table name if it is specified. [#6934](https://github.com/ClickHouse/ClickHouse/pull/6934) ([Ivan](https://github.com/abyss7)) -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-3} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-3} - Fix [flapping](https://clickhouse-test-reports.s3.yandex.net/6944/aab95fd5175a513413c7395a73a82044bdafb906/functional_stateless_tests_(debug).html) test `00715_fetch_merged_or_mutated_part_zookeeper` by rewriting it to a shell scripts because it needs to wait for mutations to apply. [#6977](https://github.com/ClickHouse/ClickHouse/pull/6977) ([Alexander Kazakov](https://github.com/Akazz)) - Fixed UBSan and MemSan failure in function `groupUniqArray` with emtpy array argument. It was caused by placing of empty `PaddedPODArray` into hash table zero cell because constructor for zero cell value was not called. [#6937](https://github.com/ClickHouse/ClickHouse/pull/6937) ([Amos Bird](https://github.com/amosbird)) -### ClickHouse Release 19.14.3.3, 2019-09-10 {#clickhouse-release-19-14-3-3-2019-09-10} +### ClickHouse release 19.14.3.3, 2019-09-10 {#clickhouse-release-19-14-3-3-2019-09-10} -#### New Feature {#new-feature-4} +#### New feature {#new-feature-4} - `WITH FILL` modifier for `ORDER BY`. (continuation of [#5069](https://github.com/ClickHouse/ClickHouse/issues/5069)) [#6610](https://github.com/ClickHouse/ClickHouse/pull/6610) ([Anton Popov](https://github.com/CurtizJ)) - `WITH TIES` modifier for `LIMIT`. (continuation of [#5069](https://github.com/ClickHouse/ClickHouse/issues/5069)) [#6610](https://github.com/ClickHouse/ClickHouse/pull/6610) ([Anton Popov](https://github.com/CurtizJ)) @@ -564,12 +564,12 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Added support for `_partition` and `_timestamp` virtual columns to Kafka engine. [#6400](https://github.com/ClickHouse/ClickHouse/pull/6400) ([Ivan](https://github.com/abyss7)) - Possibility to remove sensitive data from `query_log`, server logs, process list with regexp-based rules. [#5710](https://github.com/ClickHouse/ClickHouse/pull/5710) ([filimonov](https://github.com/filimonov)) -#### Experimental Feature {#experimental-feature-2} +#### Experimental feature {#experimental-feature-2} - Input and output data format `Template`. It allows to specify custom format string for input and output. [#4354](https://github.com/ClickHouse/ClickHouse/issues/4354) [#6727](https://github.com/ClickHouse/ClickHouse/pull/6727) ([tavplubix](https://github.com/tavplubix)) - Implementation of `LIVE VIEW` tables that were originally proposed in [#2898](https://github.com/ClickHouse/ClickHouse/pull/2898), prepared in [#3925](https://github.com/ClickHouse/ClickHouse/issues/3925), and then updated in [#5541](https://github.com/ClickHouse/ClickHouse/issues/5541). See [#5541](https://github.com/ClickHouse/ClickHouse/issues/5541) for detailed description. [#5541](https://github.com/ClickHouse/ClickHouse/issues/5541) ([vzakaznikov](https://github.com/vzakaznikov)) [#6425](https://github.com/ClickHouse/ClickHouse/pull/6425) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) [#6656](https://github.com/ClickHouse/ClickHouse/pull/6656) ([vzakaznikov](https://github.com/vzakaznikov)) Note that `LIVE VIEW` feature may be removed in next versions. -#### Bug Fix {#bug-fix-8} +#### Bug fix {#bug-fix-8} - This release also contains all bug fixes from 19.13 and 19.11. - Fix segmentation fault when the table has skip indices and vertical merge happens. [#6723](https://github.com/ClickHouse/ClickHouse/pull/6723) ([alesapin](https://github.com/alesapin)) @@ -628,7 +628,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Typo in the error message ( is -\> are ). [#6839](https://github.com/ClickHouse/ClickHouse/pull/6839) ([Denis Zhuravlev](https://github.com/den-crane)) - Fixed error while parsing of columns list from string if type contained a comma (this issue was relevant for `File`, `URL`, `HDFS` storages) [#6217](https://github.com/ClickHouse/ClickHouse/issues/6217). [#6209](https://github.com/ClickHouse/ClickHouse/pull/6209) ([dimarub2000](https://github.com/dimarub2000)) -#### Security Fix {#security-fix} +#### Security fix {#security-fix} - This release also contains all bug security fixes from 19.13 and 19.11. - Fixed the possibility of a fabricated query to cause server crash due to stack overflow in SQL parser. Fixed the possibility of stack overflow in Merge and Distributed tables, materialized views and conditions for row-level security that involve subqueries. [#6433](https://github.com/ClickHouse/ClickHouse/pull/6433) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -695,7 +695,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - `MergeTree` now has an additional option `ttl_only_drop_parts` (disabled by default) to avoid partial pruning of parts, so that they dropped completely when all the rows in a part are expired. [#6191](https://github.com/ClickHouse/ClickHouse/pull/6191) ([Sergi Vladykin](https://github.com/svladykin)) - Type checks for set index functions. Throw exception if function got a wrong type. This fixes fuzz test with UBSan. [#6511](https://github.com/ClickHouse/ClickHouse/pull/6511) ([Nikita Vasilev](https://github.com/nikvas0)) -#### Performance Improvement {#performance-improvement-2} +#### Performance improvement {#performance-improvement-2} - Optimize queries with `ORDER BY expressions` clause, where `expressions` have coinciding prefix with sorting key in `MergeTree` tables. This optimization is controlled by `optimize_read_in_order` setting. [#6054](https://github.com/ClickHouse/ClickHouse/pull/6054) [#6629](https://github.com/ClickHouse/ClickHouse/pull/6629) ([Anton Popov](https://github.com/CurtizJ)) - Allow to use multiple threads during parts loading and removal. [#6372](https://github.com/ClickHouse/ClickHouse/issues/6372) [#6074](https://github.com/ClickHouse/ClickHouse/issues/6074) [#6438](https://github.com/ClickHouse/ClickHouse/pull/6438) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -706,7 +706,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Pre-fault pages when allocating memory with `mmap()`. [#6667](https://github.com/ClickHouse/ClickHouse/pull/6667) ([akuzm](https://github.com/akuzm)) - Fix performance bug in `Decimal` comparison. [#6380](https://github.com/ClickHouse/ClickHouse/pull/6380) ([Artem Zuikov](https://github.com/4ertus2)) -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-4} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-4} - Remove Compiler (runtime template instantiation) because we've win over it's performance. [#6646](https://github.com/ClickHouse/ClickHouse/pull/6646) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Added performance test to show degradation of performance in gcc-9 in more isolated way. [#6302](https://github.com/ClickHouse/ClickHouse/pull/6302) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -784,22 +784,22 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Fix "splitted" build. [#6618](https://github.com/ClickHouse/ClickHouse/pull/6618) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Other build fixes: [#6186](https://github.com/ClickHouse/ClickHouse/pull/6186) ([Amos Bird](https://github.com/amosbird)) [#6486](https://github.com/ClickHouse/ClickHouse/pull/6486) [#6348](https://github.com/ClickHouse/ClickHouse/pull/6348) ([vxider](https://github.com/Vxider)) [#6744](https://github.com/ClickHouse/ClickHouse/pull/6744) ([Ivan](https://github.com/abyss7)) [#6016](https://github.com/ClickHouse/ClickHouse/pull/6016) [#6421](https://github.com/ClickHouse/ClickHouse/pull/6421) [#6491](https://github.com/ClickHouse/ClickHouse/pull/6491) ([proller](https://github.com/proller)) -#### Backward Incompatible Change {#backward-incompatible-change-3} +#### Backward incompatible change {#backward-incompatible-change-3} - Removed rarely used table function `catBoostPool` and storage `CatBoostPool`. If you have used this table function, please write email to `feedback@clickhouse.com`. Note that CatBoost integration remains and will be supported. [#6279](https://github.com/ClickHouse/ClickHouse/pull/6279) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Disable `ANY RIGHT JOIN` and `ANY FULL JOIN` by default. Set `any_join_distinct_right_table_keys` setting to enable them. [#5126](https://github.com/ClickHouse/ClickHouse/issues/5126) [#6351](https://github.com/ClickHouse/ClickHouse/pull/6351) ([Artem Zuikov](https://github.com/4ertus2)) -## ClickHouse Release 19.13 {#clickhouse-release-19-13} +## ClickHouse release 19.13 {#clickhouse-release-19-13} -### ClickHouse Release 19.13.6.51, 2019-10-02 {#clickhouse-release-19-13-6-51-2019-10-02} +### ClickHouse release 19.13.6.51, 2019-10-02 {#clickhouse-release-19-13-6-51-2019-10-02} -#### Bug Fix {#bug-fix-9} +#### Bug fix {#bug-fix-9} - This release also contains all bug fixes from 19.11.12.69. -### ClickHouse Release 19.13.5.44, 2019-09-20 {#clickhouse-release-19-13-5-44-2019-09-20} +### ClickHouse release 19.13.5.44, 2019-09-20 {#clickhouse-release-19-13-5-44-2019-09-20} -#### Bug Fix {#bug-fix-10} +#### Bug fix {#bug-fix-10} - This release also contains all bug fixes from 19.14.6.12. - Fixed possible inconsistent state of table while executing `DROP` query for replicated table while zookeeper is not accessible. [#6045](https://github.com/ClickHouse/ClickHouse/issues/6045) [#6413](https://github.com/ClickHouse/ClickHouse/pull/6413) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)) @@ -812,9 +812,9 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Fixed parsing of `AggregateFunction` values embedded in query. [#6575](https://github.com/ClickHouse/ClickHouse/issues/6575) [#6773](https://github.com/ClickHouse/ClickHouse/pull/6773) ([Zhichang Yu](https://github.com/yuzhichang)) - Fixed wrong behaviour of `trim` functions family. [#6647](https://github.com/ClickHouse/ClickHouse/pull/6647) ([alexey-milovidov](https://github.com/alexey-milovidov)) -### ClickHouse Release 19.13.4.32, 2019-09-10 {#clickhouse-release-19-13-4-32-2019-09-10} +### ClickHouse release 19.13.4.32, 2019-09-10 {#clickhouse-release-19-13-4-32-2019-09-10} -#### Bug Fix {#bug-fix-11} +#### Bug fix {#bug-fix-11} - This release also contains all bug security fixes from 19.11.9.52 and 19.11.10.54. - Fixed data race in `system.parts` table and `ALTER` query. [#6245](https://github.com/ClickHouse/ClickHouse/issues/6245) [#6513](https://github.com/ClickHouse/ClickHouse/pull/6513) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -826,13 +826,13 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Query transformation for `MySQL`, `ODBC`, `JDBC` table functions now works properly for `SELECT WHERE` queries with multiple `AND` expressions. [#6381](https://github.com/ClickHouse/ClickHouse/issues/6381) [#6676](https://github.com/ClickHouse/ClickHouse/pull/6676) ([dimarub2000](https://github.com/dimarub2000)) - Added previous declaration checks for MySQL 8 integration. [#6569](https://github.com/ClickHouse/ClickHouse/pull/6569) ([Rafael David Tinoco](https://github.com/rafaeldtinoco)) -#### Security Fix {#security-fix-1} +#### Security fix {#security-fix-1} - Fix two vulnerabilities in codecs in decompression phase (malicious user can fabricate compressed data that will lead to buffer overflow in decompression). [#6670](https://github.com/ClickHouse/ClickHouse/pull/6670) ([Artem Zuikov](https://github.com/4ertus2)) -### ClickHouse Release 19.13.3.26, 2019-08-22 {#clickhouse-release-19-13-3-26-2019-08-22} +### ClickHouse release 19.13.3.26, 2019-08-22 {#clickhouse-release-19-13-3-26-2019-08-22} -#### Bug Fix {#bug-fix-12} +#### Bug fix {#bug-fix-12} - Fix `ALTER TABLE ... UPDATE` query for tables with `enable_mixed_granularity_parts=1`. [#6543](https://github.com/ClickHouse/ClickHouse/pull/6543) ([alesapin](https://github.com/alesapin)) - Fix NPE when using IN clause with a subquery with a tuple. [#6125](https://github.com/ClickHouse/ClickHouse/issues/6125) [#6550](https://github.com/ClickHouse/ClickHouse/pull/6550) ([tavplubix](https://github.com/tavplubix)) @@ -842,13 +842,13 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Fixed wrong code in mutations that may lead to memory corruption. Fixed segfault with read of address `0x14c0` that may happed due to concurrent `DROP TABLE` and `SELECT` from `system.parts` or `system.parts_columns`. Fixed race condition in preparation of mutation queries. Fixed deadlock caused by `OPTIMIZE` of Replicated tables and concurrent modification operations like ALTERs. [#6514](https://github.com/ClickHouse/ClickHouse/pull/6514) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Fixed possible data loss after `ALTER DELETE` query on table with skipping index. [#6224](https://github.com/ClickHouse/ClickHouse/issues/6224) [#6282](https://github.com/ClickHouse/ClickHouse/pull/6282) ([Nikita Vasilev](https://github.com/nikvas0)) -#### Security Fix {#security-fix-2} +#### Security fix {#security-fix-2} - If the attacker has write access to ZooKeeper and is able to run custom server available from the network where ClickHouse run, it can create custom-built malicious server that will act as ClickHouse replica and register it in ZooKeeper. When another replica will fetch data part from malicious replica, it can force clickhouse-server to write to arbitrary path on filesystem. Found by Eldar Zaitov, information security team at Yandex. [#6247](https://github.com/ClickHouse/ClickHouse/pull/6247) ([alexey-milovidov](https://github.com/alexey-milovidov)) -### ClickHouse Release 19.13.2.19, 2019-08-14 {#clickhouse-release-19-13-2-19-2019-08-14} +### ClickHouse release 19.13.2.19, 2019-08-14 {#clickhouse-release-19-13-2-19-2019-08-14} -#### New Feature {#new-feature-5} +#### New feature {#new-feature-5} - Sampling profiler on query level. [Example](https://gist.github.com/alexey-milovidov/92758583dd41c24c360fdb8d6a4da194). [#4247](https://github.com/ClickHouse/ClickHouse/issues/4247) ([laplab](https://github.com/laplab)) [#6124](https://github.com/ClickHouse/ClickHouse/pull/6124) ([alexey-milovidov](https://github.com/alexey-milovidov)) [#6250](https://github.com/ClickHouse/ClickHouse/pull/6250) [#6283](https://github.com/ClickHouse/ClickHouse/pull/6283) [#6386](https://github.com/ClickHouse/ClickHouse/pull/6386) - Allow to specify a list of columns with `COLUMNS('regexp')` expression that works like a more sophisticated variant of `*` asterisk. [#5951](https://github.com/ClickHouse/ClickHouse/pull/5951) ([mfridental](https://github.com/mfridental)), ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -858,15 +858,15 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - `RENAME` queries now work with all storages. [#5953](https://github.com/ClickHouse/ClickHouse/pull/5953) ([Ivan](https://github.com/abyss7)) - Now client receive logs from server with any desired level by setting `send_logs_level` regardless to the log level specified in server settings. [#5964](https://github.com/ClickHouse/ClickHouse/pull/5964) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)) -#### Backward Incompatible Change {#backward-incompatible-change-4} +#### Backward incompatible change {#backward-incompatible-change-4} - The setting `input_format_defaults_for_omitted_fields` is enabled by default. Inserts in Distributed tables need this setting to be the same on cluster (you need to set it before rolling update). It enables calculation of complex default expressions for omitted fields in `JSONEachRow` and `CSV*` formats. It should be the expected behavior but may lead to negligible performance difference. [#6043](https://github.com/ClickHouse/ClickHouse/pull/6043) ([Artem Zuikov](https://github.com/4ertus2)), [#5625](https://github.com/ClickHouse/ClickHouse/pull/5625) ([akuzm](https://github.com/akuzm)) -#### Experimental Features {#experimental-features} +#### Experimental features {#experimental-features} - New query processing pipeline. Use `experimental_use_processors=1` option to enable it. Use for your own trouble. [#4914](https://github.com/ClickHouse/ClickHouse/pull/4914) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) -#### Bug Fix {#bug-fix-13} +#### Bug fix {#bug-fix-13} - Kafka integration has been fixed in this version. - Fixed `DoubleDelta` encoding of `Int64` for large `DoubleDelta` values, improved `DoubleDelta` encoding for random data for `Int32`. [#5998](https://github.com/ClickHouse/ClickHouse/pull/5998) ([Vasily Nemkov](https://github.com/Enmk)) @@ -876,46 +876,46 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Throws an exception if `config.d` file does not have the corresponding root element as the config file [#6123](https://github.com/ClickHouse/ClickHouse/pull/6123) ([dimarub2000](https://github.com/dimarub2000)) -#### Performance Improvement {#performance-improvement-3} +#### Performance improvement {#performance-improvement-3} - Optimize `count()`. Now it uses the smallest column (if possible). [#6028](https://github.com/ClickHouse/ClickHouse/pull/6028) ([Amos Bird](https://github.com/amosbird)) -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-5} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-5} - Report memory usage in performance tests. [#5899](https://github.com/ClickHouse/ClickHouse/pull/5899) ([akuzm](https://github.com/akuzm)) - Fix build with external `libcxx` [#6010](https://github.com/ClickHouse/ClickHouse/pull/6010) ([Ivan](https://github.com/abyss7)) - Fix shared build with `rdkafka` library [#6101](https://github.com/ClickHouse/ClickHouse/pull/6101) ([Ivan](https://github.com/abyss7)) -## ClickHouse Release 19.11 {#clickhouse-release-19-11} +## ClickHouse release 19.11 {#clickhouse-release-19-11} -### ClickHouse Release 19.11.13.74, 2019-11-01 {#clickhouse-release-19-11-13-74-2019-11-01} +### ClickHouse release 19.11.13.74, 2019-11-01 {#clickhouse-release-19-11-13-74-2019-11-01} -#### Bug Fix {#bug-fix-14} +#### Bug fix {#bug-fix-14} - Fixed rare crash in `ALTER MODIFY COLUMN` and vertical merge when one of merged/altered parts is empty (0 rows). [#6780](https://github.com/ClickHouse/ClickHouse/pull/6780) ([alesapin](https://github.com/alesapin)) - Manual update of `SIMDJSON`. This fixes possible flooding of stderr files with bogus json diagnostic messages. [#7548](https://github.com/ClickHouse/ClickHouse/pull/7548) ([Alexander Kazakov](https://github.com/Akazz)) - Fixed bug with `mrk` file extension for mutations ([alesapin](https://github.com/alesapin)) -### ClickHouse Release 19.11.12.69, 2019-10-02 {#clickhouse-release-19-11-12-69-2019-10-02} +### ClickHouse release 19.11.12.69, 2019-10-02 {#clickhouse-release-19-11-12-69-2019-10-02} -#### Bug Fix {#bug-fix-15} +#### Bug fix {#bug-fix-15} - Fixed performance degradation of index analysis on complex keys on large tables. This fixes [#6924](https://github.com/ClickHouse/ClickHouse/issues/6924). [#7075](https://github.com/ClickHouse/ClickHouse/pull/7075) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Avoid rare SIGSEGV while sending data in tables with Distributed engine (`Failed to send batch: file with index XXXXX is absent`). [#7032](https://github.com/ClickHouse/ClickHouse/pull/7032) ([Azat Khuzhin](https://github.com/azat)) - Fix `Unknown identifier` with multiple joins. This fixes [#5254](https://github.com/ClickHouse/ClickHouse/issues/5254). [#7022](https://github.com/ClickHouse/ClickHouse/pull/7022) ([Artem Zuikov](https://github.com/4ertus2)) -### ClickHouse Release 19.11.11.57, 2019-09-13 {#clickhouse-release-19-11-11-57-2019-09-13} +### ClickHouse release 19.11.11.57, 2019-09-13 {#clickhouse-release-19-11-11-57-2019-09-13} - Fix logical error causing segfaults when selecting from Kafka empty topic. [#6902](https://github.com/ClickHouse/ClickHouse/issues/6902) [#6909](https://github.com/ClickHouse/ClickHouse/pull/6909) ([Ivan](https://github.com/abyss7)) - Fix for function `АrrayEnumerateUniqRanked` with empty arrays in params. [#6928](https://github.com/ClickHouse/ClickHouse/pull/6928) ([proller](https://github.com/proller)) -### ClickHouse Release 19.11.10.54, 2019-09-10 {#clickhouse-release-19-11-10-54-2019-09-10} +### ClickHouse release 19.11.10.54, 2019-09-10 {#clickhouse-release-19-11-10-54-2019-09-10} -#### Bug Fix {#bug-fix-16} +#### Bug fix {#bug-fix-16} - Do store offsets for Kafka messages manually to be able to commit them all at once for all partitions. Fixes potential duplication in "one consumer - many partitions" scenario. [#6872](https://github.com/ClickHouse/ClickHouse/pull/6872) ([Ivan](https://github.com/abyss7)) -### ClickHouse Release 19.11.9.52, 2019-09-6 {#clickhouse-release-19-11-9-52-2019-09-6} +### ClickHouse release 19.11.9.52, 2019-09-6 {#clickhouse-release-19-11-9-52-2019-09-6} - Improve error handling in cache dictionaries. [#6737](https://github.com/ClickHouse/ClickHouse/pull/6737) ([Vitaly Baranov](https://github.com/vitlibar)) - Fixed bug in function `arrayEnumerateUniqRanked`. [#6779](https://github.com/ClickHouse/ClickHouse/pull/6779) ([proller](https://github.com/proller)) @@ -929,13 +929,13 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Fixed error with processing "timezone" in server configuration file. [#6709](https://github.com/ClickHouse/ClickHouse/pull/6709) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Fix kafka tests. [#6805](https://github.com/ClickHouse/ClickHouse/pull/6805) ([Ivan](https://github.com/abyss7)) -#### Security Fix {#security-fix-3} +#### Security fix {#security-fix-3} - If the attacker has write access to ZooKeeper and is able to run custom server available from the network where ClickHouse runs, it can create custom-built malicious server that will act as ClickHouse replica and register it in ZooKeeper. When another replica will fetch data part from malicious replica, it can force clickhouse-server to write to arbitrary path on filesystem. Found by Eldar Zaitov, information security team at Yandex. [#6247](https://github.com/ClickHouse/ClickHouse/pull/6247) ([alexey-milovidov](https://github.com/alexey-milovidov)) -### ClickHouse Release 19.11.8.46, 2019-08-22 {#clickhouse-release-19-11-8-46-2019-08-22} +### ClickHouse release 19.11.8.46, 2019-08-22 {#clickhouse-release-19-11-8-46-2019-08-22} -#### Bug Fix {#bug-fix-17} +#### Bug fix {#bug-fix-17} - Fix `ALTER TABLE ... UPDATE` query for tables with `enable_mixed_granularity_parts=1`. [#6543](https://github.com/ClickHouse/ClickHouse/pull/6543) ([alesapin](https://github.com/alesapin)) - Fix NPE when using IN clause with a subquery with a tuple. [#6125](https://github.com/ClickHouse/ClickHouse/issues/6125) [#6550](https://github.com/ClickHouse/ClickHouse/pull/6550) ([tavplubix](https://github.com/tavplubix)) @@ -944,9 +944,9 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Fixed data race in system.parts table and ALTER query. This fixes [#6245](https://github.com/ClickHouse/ClickHouse/issues/6245). [#6513](https://github.com/ClickHouse/ClickHouse/pull/6513) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Fixed wrong code in mutations that may lead to memory corruption. Fixed segfault with read of address `0x14c0` that may happed due to concurrent `DROP TABLE` and `SELECT` from `system.parts` or `system.parts_columns`. Fixed race condition in preparation of mutation queries. Fixed deadlock caused by `OPTIMIZE` of Replicated tables and concurrent modification operations like ALTERs. [#6514](https://github.com/ClickHouse/ClickHouse/pull/6514) ([alexey-milovidov](https://github.com/alexey-milovidov)) -### ClickHouse Release 19.11.7.40, 2019-08-14 {#clickhouse-release-19-11-7-40-2019-08-14} +### ClickHouse release 19.11.7.40, 2019-08-14 {#clickhouse-release-19-11-7-40-2019-08-14} -#### Bug Fix {#bug-fix-18} +#### Bug fix {#bug-fix-18} - Kafka integration has been fixed in this version. - Fix segfault when using `arrayReduce` for constant arguments. [#6326](https://github.com/ClickHouse/ClickHouse/pull/6326) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -966,9 +966,9 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Allow user to override `poll_interval` and `idle_connection_timeout` settings on connection. [#6230](https://github.com/ClickHouse/ClickHouse/pull/6230) ([alexey-milovidov](https://github.com/alexey-milovidov)) -### ClickHouse Release 19.11.5.28, 2019-08-05 {#clickhouse-release-19-11-5-28-2019-08-05} +### ClickHouse release 19.11.5.28, 2019-08-05 {#clickhouse-release-19-11-5-28-2019-08-05} -#### Bug Fix {#bug-fix-19} +#### Bug fix {#bug-fix-19} - Fixed the possibility of hanging queries when server is overloaded. [#6301](https://github.com/ClickHouse/ClickHouse/pull/6301) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Fix FPE in yandexConsistentHash function. This fixes [#6304](https://github.com/ClickHouse/ClickHouse/issues/6304). [#6126](https://github.com/ClickHouse/ClickHouse/pull/6126) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -977,13 +977,13 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Fix rare bug with incompatible stream headers in queries to `Distributed` table over `MergeTree` table when part of `WHERE` moves to `PREWHERE`. [#6236](https://github.com/ClickHouse/ClickHouse/pull/6236) ([alesapin](https://github.com/alesapin)) - Fixed overflow in integer division of signed type to unsigned type. This fixes [#6214](https://github.com/ClickHouse/ClickHouse/issues/6214). [#6233](https://github.com/ClickHouse/ClickHouse/pull/6233) ([alexey-milovidov](https://github.com/alexey-milovidov)) -#### Backward Incompatible Change {#backward-incompatible-change-5} +#### Backward incompatible change {#backward-incompatible-change-5} - `Kafka` still broken. -### ClickHouse Release 19.11.4.24, 2019-08-01 {#clickhouse-release-19-11-4-24-2019-08-01} +### ClickHouse release 19.11.4.24, 2019-08-01 {#clickhouse-release-19-11-4-24-2019-08-01} -#### Bug Fix {#bug-fix-20} +#### Bug fix {#bug-fix-20} - Fix bug with writing secondary indices marks with adaptive granularity. [#6126](https://github.com/ClickHouse/ClickHouse/pull/6126) ([alesapin](https://github.com/alesapin)) - Fix `WITH ROLLUP` and `WITH CUBE` modifiers of `GROUP BY` with two-level aggregation. [#6225](https://github.com/ClickHouse/ClickHouse/pull/6225) ([Anton Popov](https://github.com/CurtizJ)) @@ -999,19 +999,19 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Clearing the Kafka data buffer from the previous read operation that was completed with an error [#6026](https://github.com/ClickHouse/ClickHouse/pull/6026) ([Nikolay](https://github.com/bopohaa)) Note that Kafka is broken in this version. - Since `StorageMergeTree::background_task_handle` is initialized in `startup()` the `MergeTreeBlockOutputStream::write()` may try to use it before initialization. Just check if it is initialized. [#6080](https://github.com/ClickHouse/ClickHouse/pull/6080) ([Ivan](https://github.com/abyss7)) -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-6} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-6} - Added official `rpm` packages. [#5740](https://github.com/ClickHouse/ClickHouse/pull/5740) ([proller](https://github.com/proller)) ([alesapin](https://github.com/alesapin)) - Add an ability to build `.rpm` and `.tgz` packages with `packager` script. [#5769](https://github.com/ClickHouse/ClickHouse/pull/5769) ([alesapin](https://github.com/alesapin)) - Fixes for "Arcadia" build system. [#6223](https://github.com/ClickHouse/ClickHouse/pull/6223) ([proller](https://github.com/proller)) -#### Backward Incompatible Change {#backward-incompatible-change-6} +#### Backward incompatible change {#backward-incompatible-change-6} - `Kafka` is broken in this version. -### ClickHouse Release 19.11.3.11, 2019-07-18 {#clickhouse-release-19-11-3-11-2019-07-18} +### ClickHouse release 19.11.3.11, 2019-07-18 {#clickhouse-release-19-11-3-11-2019-07-18} -#### New Feature {#new-feature-6} +#### New feature {#new-feature-6} - Added support for prepared statements. [#5331](https://github.com/ClickHouse/ClickHouse/pull/5331/) ([Alexander](https://github.com/sanych73)) [#5630](https://github.com/ClickHouse/ClickHouse/pull/5630) ([alexey-milovidov](https://github.com/alexey-milovidov)) - `DoubleDelta` and `Gorilla` column codecs [#5600](https://github.com/ClickHouse/ClickHouse/pull/5600) ([Vasily Nemkov](https://github.com/Enmk)) @@ -1022,7 +1022,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Add synonim `arrayFlatten` \<-\> `flatten` [#5764](https://github.com/ClickHouse/ClickHouse/pull/5764) ([hcz](https://github.com/hczhcz)) - Intergate H3 function `geoToH3` from Uber. [#4724](https://github.com/ClickHouse/ClickHouse/pull/4724) ([Remen Ivan](https://github.com/BHYCHIK)) [#5805](https://github.com/ClickHouse/ClickHouse/pull/5805) ([alexey-milovidov](https://github.com/alexey-milovidov)) -#### Bug Fix {#bug-fix-21} +#### Bug fix {#bug-fix-21} - Implement DNS cache with asynchronous update. Separate thread resolves all hosts and updates DNS cache with period (setting `dns_cache_update_period`). It should help, when ip of hosts changes frequently. [#5857](https://github.com/ClickHouse/ClickHouse/pull/5857) ([Anton Popov](https://github.com/CurtizJ)) - Fix segfault in `Delta` codec which affects columns with values less than 32 bits size. The bug led to random memory corruption. [#5786](https://github.com/ClickHouse/ClickHouse/pull/5786) ([alesapin](https://github.com/alesapin)) @@ -1072,11 +1072,11 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Update default value of `max_ast_elements parameter` [#5933](https://github.com/ClickHouse/ClickHouse/pull/5933) ([Artem Konovalov](https://github.com/izebit)) - Added a notion of obsolete settings. The obsolete setting `allow_experimental_low_cardinality_type` can be used with no effect. [0f15c01c6802f7ce1a1494c12c846be8c98944cd](https://github.com/ClickHouse/ClickHouse/commit/0f15c01c6802f7ce1a1494c12c846be8c98944cd) [Alexey Milovidov](https://github.com/alexey-milovidov) -#### Performance Improvement {#performance-improvement-4} +#### Performance improvement {#performance-improvement-4} - Increase number of streams to SELECT from Merge table for more uniform distribution of threads. Added setting `max_streams_multiplier_for_merge_tables`. This fixes [#5797](https://github.com/ClickHouse/ClickHouse/issues/5797) [#5915](https://github.com/ClickHouse/ClickHouse/pull/5915) ([alexey-milovidov](https://github.com/alexey-milovidov)) -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-7} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-7} - Add a backward compatibility test for client-server interaction with different versions of clickhouse. [#5868](https://github.com/ClickHouse/ClickHouse/pull/5868) ([alesapin](https://github.com/alesapin)) - Test coverage information in every commit and pull request. [#5896](https://github.com/ClickHouse/ClickHouse/pull/5896) ([alesapin](https://github.com/alesapin)) @@ -1114,17 +1114,17 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Performance test concerning the new JIT feature with bigger dataset, as requested here [#5263](https://github.com/ClickHouse/ClickHouse/issues/5263) [#5887](https://github.com/ClickHouse/ClickHouse/pull/5887) ([Guillaume Tassery](https://github.com/YiuRULE)) - Run stateful tests in stress test [12693e568722f11e19859742f56428455501fd2a](https://github.com/ClickHouse/ClickHouse/commit/12693e568722f11e19859742f56428455501fd2a) ([alesapin](https://github.com/alesapin)) -#### Backward Incompatible Change {#backward-incompatible-change-7} +#### Backward incompatible change {#backward-incompatible-change-7} - `Kafka` is broken in this version. - Enable `adaptive_index_granularity` = 10MB by default for new `MergeTree` tables. If you created new MergeTree tables on version 19.11+, downgrade to versions prior to 19.6 will be impossible. [#5628](https://github.com/ClickHouse/ClickHouse/pull/5628) ([alesapin](https://github.com/alesapin)) - Removed obsolete undocumented embedded dictionaries that were used by Yandex.Metrica. The functions `OSIn`, `SEIn`, `OSToRoot`, `SEToRoot`, `OSHierarchy`, `SEHierarchy` are no longer available. If you are using these functions, write email to clickhouse-feedback@yandex-team.com. Note: at the last moment we decided to keep these functions for a while. [#5780](https://github.com/ClickHouse/ClickHouse/pull/5780) ([alexey-milovidov](https://github.com/alexey-milovidov)) -## ClickHouse Release 19.10 {#clickhouse-release-19-10} +## ClickHouse release 19.10 {#clickhouse-release-19-10} -### ClickHouse Release 19.10.1.5, 2019-07-12 {#clickhouse-release-19-10-1-5-2019-07-12} +### ClickHouse release 19.10.1.5, 2019-07-12 {#clickhouse-release-19-10-1-5-2019-07-12} -#### New Feature {#new-feature-7} +#### New feature {#new-feature-7} - Add new column codec: `T64`. Made for (U)IntX/EnumX/Data(Time)/DecimalX columns. It should be good for columns with constant or small range values. Codec itself allows enlarge or shrink data type without re-compression. [#5557](https://github.com/ClickHouse/ClickHouse/pull/5557) ([Artem Zuikov](https://github.com/4ertus2)) - Add database engine `MySQL` that allow to view all the tables in remote MySQL server [#5599](https://github.com/ClickHouse/ClickHouse/pull/5599) ([Winter Zhang](https://github.com/zhang2014)) @@ -1132,7 +1132,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Support for `crc32` function (with behaviour exactly as in MySQL or PHP). Do not use it if you need a hash function. [#5661](https://github.com/ClickHouse/ClickHouse/pull/5661) ([Remen Ivan](https://github.com/BHYCHIK)) - Implemented `SYSTEM START/STOP DISTRIBUTED SENDS` queries to control asynchronous inserts into `Distributed` tables. [#4935](https://github.com/ClickHouse/ClickHouse/pull/4935) ([Winter Zhang](https://github.com/zhang2014)) -#### Bug Fix {#bug-fix-22} +#### Bug fix {#bug-fix-22} - Ignore query execution limits and max parts size for merge limits while executing mutations. [#5659](https://github.com/ClickHouse/ClickHouse/pull/5659) ([Anton Popov](https://github.com/CurtizJ)) - Fix bug which may lead to deduplication of normal blocks (extremely rare) and insertion of duplicate blocks (more often). [#5549](https://github.com/ClickHouse/ClickHouse/pull/5549) ([alesapin](https://github.com/alesapin)) @@ -1155,24 +1155,24 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Support `` section in `clickhouse-local` config file. [#5540](https://github.com/ClickHouse/ClickHouse/pull/5540) ([proller](https://github.com/proller)) - Allow run query with `remote` table function in `clickhouse-local` [#5627](https://github.com/ClickHouse/ClickHouse/pull/5627) ([proller](https://github.com/proller)) -#### Performance Improvement {#performance-improvement-5} +#### Performance improvement {#performance-improvement-5} - Add the possibility to write the final mark at the end of MergeTree columns. It allows to avoid useless reads for keys that are out of table data range. It is enabled only if adaptive index granularity is in use. [#5624](https://github.com/ClickHouse/ClickHouse/pull/5624) ([alesapin](https://github.com/alesapin)) - Improved performance of MergeTree tables on very slow filesystems by reducing number of `stat` syscalls. [#5648](https://github.com/ClickHouse/ClickHouse/pull/5648) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Fixed performance degradation in reading from MergeTree tables that was introduced in version 19.6. Fixes #5631. [#5633](https://github.com/ClickHouse/ClickHouse/pull/5633) ([alexey-milovidov](https://github.com/alexey-milovidov)) -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-8} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-8} - Implemented `TestKeeper` as an implementation of ZooKeeper interface used for testing [#5643](https://github.com/ClickHouse/ClickHouse/pull/5643) ([alexey-milovidov](https://github.com/alexey-milovidov)) ([levushkin aleksej](https://github.com/alexey-milovidov)) - From now on `.sql` tests can be run isolated by server, in parallel, with random database. It allows to run them faster, add new tests with custom server configurations, and be sure that different tests does not affect each other. [#5554](https://github.com/ClickHouse/ClickHouse/pull/5554) ([Ivan](https://github.com/abyss7)) - Remove `` and `` from performance tests [#5672](https://github.com/ClickHouse/ClickHouse/pull/5672) ([Olga Khvostikova](https://github.com/stavrolia)) - Fixed "select_format" performance test for `Pretty` formats [#5642](https://github.com/ClickHouse/ClickHouse/pull/5642) ([alexey-milovidov](https://github.com/alexey-milovidov)) -## ClickHouse Release 19.9 {#clickhouse-release-19-9} +## ClickHouse release 19.9 {#clickhouse-release-19-9} -### ClickHouse Release 19.9.3.31, 2019-07-05 {#clickhouse-release-19-9-3-31-2019-07-05} +### ClickHouse release 19.9.3.31, 2019-07-05 {#clickhouse-release-19-9-3-31-2019-07-05} -#### Bug Fix {#bug-fix-23} +#### Bug fix {#bug-fix-23} - Fix segfault in Delta codec which affects columns with values less than 32 bits size. The bug led to random memory corruption. [#5786](https://github.com/ClickHouse/ClickHouse/pull/5786) ([alesapin](https://github.com/alesapin)) - Fix rare bug in checking of part with LowCardinality column. [#5832](https://github.com/ClickHouse/ClickHouse/pull/5832) ([alesapin](https://github.com/alesapin)) @@ -1183,9 +1183,9 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Fix race condition, which cause that some queries may not appear in query_log instantly after SYSTEM FLUSH LOGS query. [#5685](https://github.com/ClickHouse/ClickHouse/pull/5685) ([Anton Popov](https://github.com/CurtizJ)) - Added missing support for constant arguments to `evalMLModel` function. [#5820](https://github.com/ClickHouse/ClickHouse/pull/5820) ([alexey-milovidov](https://github.com/alexey-milovidov)) -### ClickHouse Release 19.9.2.4, 2019-06-24 {#clickhouse-release-19-9-2-4-2019-06-24} +### ClickHouse release 19.9.2.4, 2019-06-24 {#clickhouse-release-19-9-2-4-2019-06-24} -#### New Feature {#new-feature-8} +#### New feature {#new-feature-8} - Print information about frozen parts in `system.parts` table. [#5471](https://github.com/ClickHouse/ClickHouse/pull/5471) ([proller](https://github.com/proller)) - Ask client password on clickhouse-client start on tty if not set in arguments [#5092](https://github.com/ClickHouse/ClickHouse/pull/5092) ([proller](https://github.com/proller)) @@ -1199,7 +1199,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Rename and improve regression methods. [#5492](https://github.com/ClickHouse/ClickHouse/pull/5492) ([Quid37](https://github.com/Quid37)) - Clearer interfaces of string searchers. [#5586](https://github.com/ClickHouse/ClickHouse/pull/5586) ([Danila Kutenin](https://github.com/danlark1)) -#### Bug Fix {#bug-fix-24} +#### Bug fix {#bug-fix-24} - Fix potential data loss in Kafka [#5445](https://github.com/ClickHouse/ClickHouse/pull/5445) ([Ivan](https://github.com/abyss7)) - Fix potential infinite loop in `PrettySpace` format when called with zero columns [#5560](https://github.com/ClickHouse/ClickHouse/pull/5560) ([Olga Khvostikova](https://github.com/stavrolia)) @@ -1220,7 +1220,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Throw an exception on wrong integers in `dictGetT` functions instead of crash. [#5446](https://github.com/ClickHouse/ClickHouse/pull/5446) ([Artem Zuikov](https://github.com/4ertus2)) - Fix wrong element_count and load_factor for hashed dictionary in `system.dictionaries` table. [#5440](https://github.com/ClickHouse/ClickHouse/pull/5440) ([Azat Khuzhin](https://github.com/azat)) -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-9} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-9} - Fixed build without `Brotli` HTTP compression support (`ENABLE_BROTLI=OFF` cmake variable). [#5521](https://github.com/ClickHouse/ClickHouse/pull/5521) ([Anton Yuzhaninov](https://github.com/citrin)) - Include roaring.h as roaring/roaring.h [#5523](https://github.com/ClickHouse/ClickHouse/pull/5523) ([Orivej Desh](https://github.com/orivej)) @@ -1236,11 +1236,11 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Fix build clickhouse as submodule [#5574](https://github.com/ClickHouse/ClickHouse/pull/5574) ([proller](https://github.com/proller)) - Improve JSONExtract performance tests [#5444](https://github.com/ClickHouse/ClickHouse/pull/5444) ([Vitaly Baranov](https://github.com/vitlibar)) -## ClickHouse Release 19.8 {#clickhouse-release-19-8} +## ClickHouse release 19.8 {#clickhouse-release-19-8} -### ClickHouse Release 19.8.3.8, 2019-06-11 {#clickhouse-release-19-8-3-8-2019-06-11} +### ClickHouse release 19.8.3.8, 2019-06-11 {#clickhouse-release-19-8-3-8-2019-06-11} -#### New Features {#new-features} +#### New features {#new-features} - Added functions to work with JSON [#4686](https://github.com/ClickHouse/ClickHouse/pull/4686) ([hcz](https://github.com/hczhcz)) [#5124](https://github.com/ClickHouse/ClickHouse/pull/5124). ([Vitaly Baranov](https://github.com/vitlibar)) - Add a function basename, with a similar behaviour to a basename function, which exists in a lot of languages (`os.path.basename` in python, `basename` in PHP, etc...). Work with both an UNIX-like path or a Windows path. [#5136](https://github.com/ClickHouse/ClickHouse/pull/5136) ([Guillaume Tassery](https://github.com/YiuRULE)) @@ -1287,7 +1287,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Respect query settings in asynchronous INSERTs into Distributed tables. [#4936](https://github.com/ClickHouse/ClickHouse/pull/4936) ([TCeason](https://github.com/TCeason)) - Renamed functions `leastSqr` to `simpleLinearRegression`, `LinearRegression` to `linearRegression`, `LogisticRegression` to `logisticRegression`. [#5391](https://github.com/ClickHouse/ClickHouse/pull/5391) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) -#### Performance Improvements {#performance-improvements} +#### Performance improvements {#performance-improvements} - Parallelize processing of parts of non-replicated MergeTree tables in ALTER MODIFY query. [#4639](https://github.com/ClickHouse/ClickHouse/pull/4639) ([Ivan Kush](https://github.com/IvanKush)) - Optimizations in regular expressions extraction. [#5193](https://github.com/ClickHouse/ClickHouse/pull/5193) [#5191](https://github.com/ClickHouse/ClickHouse/pull/5191) ([Danila Kutenin](https://github.com/danlark1)) @@ -1298,7 +1298,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Upgrade our LZ4 implementation with reference one to have faster decompression. [#5070](https://github.com/ClickHouse/ClickHouse/pull/5070) ([Danila Kutenin](https://github.com/danlark1)) - Implemented MSD radix sort (based on kxsort), and partial sorting. [#5129](https://github.com/ClickHouse/ClickHouse/pull/5129) ([Evgenii Pravda](https://github.com/kvinty)) -#### Bug Fixes {#bug-fixes} +#### Bug fixes {#bug-fixes} - Fix push require columns with join [#5192](https://github.com/ClickHouse/ClickHouse/pull/5192) ([Winter Zhang](https://github.com/zhang2014)) - Fixed bug, when ClickHouse is run by systemd, the command `sudo service clickhouse-server forcerestart` was not working as expected. [#5204](https://github.com/ClickHouse/ClickHouse/pull/5204) ([proller](https://github.com/proller)) @@ -1318,7 +1318,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor [#5328](https://github.com/ClickHouse/ClickHouse/pull/5328) ([never lee](https://github.com/neverlee)) -#### Build/Testing/Packaging Improvements {#buildtestingpackaging-improvements} +#### Build/Testing/Packaging improvements {#buildtestingpackaging-improvements} - Fix some sanitizer reports that show probable use-after-free.[#5139](https://github.com/ClickHouse/ClickHouse/pull/5139) [#5143](https://github.com/ClickHouse/ClickHouse/pull/5143) [#5393](https://github.com/ClickHouse/ClickHouse/pull/5393) ([Ivan](https://github.com/abyss7)) - Move performance tests out of separate directories for convenience. [#5158](https://github.com/ClickHouse/ClickHouse/pull/5158) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -1328,21 +1328,21 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Add small instruction how to write performance tests. [#5408](https://github.com/ClickHouse/ClickHouse/pull/5408) ([alesapin](https://github.com/alesapin)) - Add ability to make substitutions in create, fill and drop query in performance tests [#5367](https://github.com/ClickHouse/ClickHouse/pull/5367) ([Olga Khvostikova](https://github.com/stavrolia)) -## ClickHouse Release 19.7 {#clickhouse-release-19-7} +## ClickHouse release 19.7 {#clickhouse-release-19-7} -### ClickHouse Release 19.7.5.29, 2019-07-05 {#clickhouse-release-19-7-5-29-2019-07-05} +### ClickHouse release 19.7.5.29, 2019-07-05 {#clickhouse-release-19-7-5-29-2019-07-05} -#### Bug Fix {#bug-fix-25} +#### Bug fix {#bug-fix-25} - Fix performance regression in some queries with JOIN. [#5192](https://github.com/ClickHouse/ClickHouse/pull/5192) ([Winter Zhang](https://github.com/zhang2014)) -### ClickHouse Release 19.7.5.27, 2019-06-09 {#clickhouse-release-19-7-5-27-2019-06-09} +### ClickHouse release 19.7.5.27, 2019-06-09 {#clickhouse-release-19-7-5-27-2019-06-09} -#### New Features {#new-features-1} +#### New features {#new-features-1} - Added bitmap related functions `bitmapHasAny` and `bitmapHasAll` analogous to `hasAny` and `hasAll` functions for arrays. [#5279](https://github.com/ClickHouse/ClickHouse/pull/5279) ([Sergi Vladykin](https://github.com/svladykin)) -#### Bug Fixes {#bug-fixes-1} +#### Bug fixes {#bug-fixes-1} - Fix segfault on `minmax` INDEX with Null value. [#5246](https://github.com/ClickHouse/ClickHouse/pull/5246) ([Nikita Vasilev](https://github.com/nikvas0)) - Mark all input columns in LIMIT BY as required output. It fixes 'Not found column' error in some distributed queries. [#5407](https://github.com/ClickHouse/ClickHouse/pull/5407) ([Constantin S. Pan](https://github.com/kvap)) @@ -1364,9 +1364,9 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor did not process it, but already get list of children, will terminate the DDLWorker thread. [#5489](https://github.com/ClickHouse/ClickHouse/pull/5489) ([Azat Khuzhin](https://github.com/azat)) - Fix INSERT into Distributed() table with MATERIALIZED column. [#5429](https://github.com/ClickHouse/ClickHouse/pull/5429) ([Azat Khuzhin](https://github.com/azat)) -### ClickHouse Release 19.7.3.9, 2019-05-30 {#clickhouse-release-19-7-3-9-2019-05-30} +### ClickHouse release 19.7.3.9, 2019-05-30 {#clickhouse-release-19-7-3-9-2019-05-30} -#### New Features {#new-features-2} +#### New features {#new-features-2} - Allow to limit the range of a setting that can be specified by user. These constraints can be set up in user settings profile. @@ -1383,7 +1383,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor [#5081](https://github.com/ClickHouse/ClickHouse/pull/5081) ([Alexander](https://github.com/Akazz)) -#### Bug Fixes {#bug-fixes-2} +#### Bug fixes {#bug-fixes-2} - Crash with uncompressed_cache + JOIN during merge (#5197) [#5133](https://github.com/ClickHouse/ClickHouse/pull/5133) ([Danila @@ -1396,7 +1396,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor ([Ivan](https://github.com/abyss7)) - Fixed very rare data race condition that could happen when executing a query with UNION ALL involving at least two SELECTs from system.columns, system.tables, system.parts, system.parts_tables or tables of Merge family and performing ALTER of columns of the related tables concurrently. [#5189](https://github.com/ClickHouse/ClickHouse/pull/5189) ([alexey-milovidov](https://github.com/alexey-milovidov)) -#### Performance Improvements {#performance-improvements-1} +#### Performance improvements {#performance-improvements-1} - Use radix sort for sorting by single numeric column in `ORDER BY` without `LIMIT`. [#5106](https://github.com/ClickHouse/ClickHouse/pull/5106), @@ -1414,7 +1414,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor [#5068](https://github.com/ClickHouse/ClickHouse/pull/5068) ([never lee](https://github.com/neverlee)) -#### Build/Testing/Packaging Improvements {#buildtestingpackaging-improvements-1} +#### Build/Testing/Packaging improvements {#buildtestingpackaging-improvements-1} - Print UTF-8 characters properly in `clickhouse-test`. [#5084](https://github.com/ClickHouse/ClickHouse/pull/5084) @@ -1431,11 +1431,11 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor [#5110](https://github.com/ClickHouse/ClickHouse/pull/5110) ([proller](https://github.com/proller)) -## ClickHouse Release 19.6 {#clickhouse-release-19-6} +## ClickHouse release 19.6 {#clickhouse-release-19-6} -### ClickHouse Release 19.6.3.18, 2019-06-13 {#clickhouse-release-19-6-3-18-2019-06-13} +### ClickHouse release 19.6.3.18, 2019-06-13 {#clickhouse-release-19-6-3-18-2019-06-13} -#### Bug Fixes {#bug-fixes-3} +#### Bug fixes {#bug-fixes-3} - Fixed IN condition pushdown for queries from table functions `mysql` and `odbc` and corresponding table engines. This fixes #3540 and #2384. [#5313](https://github.com/ClickHouse/ClickHouse/pull/5313) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Fix deadlock in Zookeeper. [#5297](https://github.com/ClickHouse/ClickHouse/pull/5297) ([github1youlc](https://github.com/github1youlc)) @@ -1444,16 +1444,16 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Fix data race in rename query. [#5247](https://github.com/ClickHouse/ClickHouse/pull/5247) ([Winter Zhang](https://github.com/zhang2014)) - Temporarily disable LFAlloc. Usage of LFAlloc might lead to a lot of MAP_FAILED in allocating UncompressedCache and in a result to crashes of queries at high loaded servers. [cfdba93](https://github.com/ClickHouse/ClickHouse/commit/cfdba938ce22f16efeec504f7f90206a515b1280)([Danila Kutenin](https://github.com/danlark1)) -### ClickHouse Release 19.6.2.11, 2019-05-13 {#clickhouse-release-19-6-2-11-2019-05-13} +### ClickHouse release 19.6.2.11, 2019-05-13 {#clickhouse-release-19-6-2-11-2019-05-13} -#### New Features {#new-features-3} +#### New features {#new-features-3} - TTL expressions for columns and tables. [#4212](https://github.com/ClickHouse/ClickHouse/pull/4212) ([Anton Popov](https://github.com/CurtizJ)) - Added support for `brotli` compression for HTTP responses (Accept-Encoding: br) [#4388](https://github.com/ClickHouse/ClickHouse/pull/4388) ([Mikhail](https://github.com/fandyushin)) - Added new function `isValidUTF8` for checking whether a set of bytes is correctly utf-8 encoded. [#4934](https://github.com/ClickHouse/ClickHouse/pull/4934) ([Danila Kutenin](https://github.com/danlark1)) - Add new load balancing policy `first_or_random` which sends queries to the first specified host and if it's inaccessible send queries to random hosts of shard. Useful for cross-replication topology setups. [#5012](https://github.com/ClickHouse/ClickHouse/pull/5012) ([nvartolomei](https://github.com/nvartolomei)) -#### Experimental Features {#experimental-features-1} +#### Experimental features {#experimental-features-1} - Add setting `index_granularity_bytes` (adaptive index granularity) for MergeTree\* tables family. [#4826](https://github.com/ClickHouse/ClickHouse/pull/4826) ([alesapin](https://github.com/alesapin)) @@ -1466,15 +1466,15 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Support asterisks and qualified asterisks for multiple joins without subqueries [#4898](https://github.com/ClickHouse/ClickHouse/pull/4898) ([Artem Zuikov](https://github.com/4ertus2)) - Make missing column error message more user friendly. [#4915](https://github.com/ClickHouse/ClickHouse/pull/4915) ([Artem Zuikov](https://github.com/4ertus2)) -#### Performance Improvements {#performance-improvements-2} +#### Performance improvements {#performance-improvements-2} - Significant speedup of ASOF JOIN [#4924](https://github.com/ClickHouse/ClickHouse/pull/4924) ([Martijn Bakker](https://github.com/Gladdy)) -#### Backward Incompatible Changes {#backward-incompatible-changes} +#### Backward incompatible changes {#backward-incompatible-changes} - HTTP header `Query-Id` was renamed to `X-ClickHouse-Query-Id` for consistency. [#4972](https://github.com/ClickHouse/ClickHouse/pull/4972) ([Mikhail](https://github.com/fandyushin)) -#### Bug Fixes {#bug-fixes-4} +#### Bug fixes {#bug-fixes-4} - Fixed potential null pointer dereference in `clickhouse-copier`. [#4900](https://github.com/ClickHouse/ClickHouse/pull/4900) ([proller](https://github.com/proller)) - Fixed error on query with JOIN + ARRAY JOIN [#4938](https://github.com/ClickHouse/ClickHouse/pull/4938) ([Artem Zuikov](https://github.com/4ertus2)) @@ -1483,7 +1483,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Fix potentially wrong result for `SELECT DISTINCT` with `JOIN` [#5001](https://github.com/ClickHouse/ClickHouse/pull/5001) ([Artem Zuikov](https://github.com/4ertus2)) - Fixed very rare data race condition that could happen when executing a query with UNION ALL involving at least two SELECTs from system.columns, system.tables, system.parts, system.parts_tables or tables of Merge family and performing ALTER of columns of the related tables concurrently. [#5189](https://github.com/ClickHouse/ClickHouse/pull/5189) ([alexey-milovidov](https://github.com/alexey-milovidov)) -#### Build/Testing/Packaging Improvements {#buildtestingpackaging-improvements-2} +#### Build/Testing/Packaging improvements {#buildtestingpackaging-improvements-2} - Fixed test failures when running clickhouse-server on different host [#4713](https://github.com/ClickHouse/ClickHouse/pull/4713) ([Vasily Nemkov](https://github.com/Enmk)) - clickhouse-test: Disable color control sequences in non tty environment. [#4937](https://github.com/ClickHouse/ClickHouse/pull/4937) ([alesapin](https://github.com/alesapin)) @@ -1492,26 +1492,26 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Yandex LFAlloc was added to ClickHouse to allocate MarkCache and UncompressedCache data in different ways to catch segfaults more reliable [#4995](https://github.com/ClickHouse/ClickHouse/pull/4995) ([Danila Kutenin](https://github.com/danlark1)) - Python util to help with backports and changelogs. [#4949](https://github.com/ClickHouse/ClickHouse/pull/4949) ([Ivan](https://github.com/abyss7)) -## ClickHouse Release 19.5 {#clickhouse-release-19-5} +## ClickHouse release 19.5 {#clickhouse-release-19-5} -### ClickHouse Release 19.5.4.22, 2019-05-13 {#clickhouse-release-19-5-4-22-2019-05-13} +### ClickHouse release 19.5.4.22, 2019-05-13 {#clickhouse-release-19-5-4-22-2019-05-13} -#### Bug Fixes {#bug-fixes-5} +#### Bug fixes {#bug-fixes-5} - Fixed possible crash in bitmap\* functions [#5220](https://github.com/ClickHouse/ClickHouse/pull/5220) [#5228](https://github.com/ClickHouse/ClickHouse/pull/5228) ([Andy Yang](https://github.com/andyyzh)) - Fixed very rare data race condition that could happen when executing a query with UNION ALL involving at least two SELECTs from system.columns, system.tables, system.parts, system.parts_tables or tables of Merge family and performing ALTER of columns of the related tables concurrently. [#5189](https://github.com/ClickHouse/ClickHouse/pull/5189) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Fixed error `Set for IN is not created yet in case of using single LowCardinality column in the left part of IN`. This error happened if LowCardinality column was the part of primary key. #5031 [#5154](https://github.com/ClickHouse/ClickHouse/pull/5154) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) - Modification of retention function: If a row satisfies both the first and NTH condition, only the first satisfied condition is added to the data state. Now all conditions that satisfy in a row of data are added to the data state. [#5119](https://github.com/ClickHouse/ClickHouse/pull/5119) ([小路](https://github.com/nicelulu)) -### ClickHouse Release 19.5.3.8, 2019-04-18 {#clickhouse-release-19-5-3-8-2019-04-18} +### ClickHouse release 19.5.3.8, 2019-04-18 {#clickhouse-release-19-5-3-8-2019-04-18} -#### Bug Fixes {#bug-fixes-6} +#### Bug fixes {#bug-fixes-6} - Fixed type of setting `max_partitions_per_insert_block` from boolean to UInt64. [#5028](https://github.com/ClickHouse/ClickHouse/pull/5028) ([Mohammad Hossein Sekhavat](https://github.com/mhsekhavat)) -### ClickHouse Release 19.5.2.6, 2019-04-15 {#clickhouse-release-19-5-2-6-2019-04-15} +### ClickHouse release 19.5.2.6, 2019-04-15 {#clickhouse-release-19-5-2-6-2019-04-15} -#### New Features {#new-features-4} +#### New features {#new-features-4} - [Hyperscan](https://github.com/intel/hyperscan) multiple regular expression matching was added (functions `multiMatchAny`, `multiMatchAnyIndex`, `multiFuzzyMatchAny`, `multiFuzzyMatchAnyIndex`). [#4780](https://github.com/ClickHouse/ClickHouse/pull/4780), [#4841](https://github.com/ClickHouse/ClickHouse/pull/4841) ([Danila Kutenin](https://github.com/danlark1)) - `multiSearchFirstPosition` function was added. [#4780](https://github.com/ClickHouse/ClickHouse/pull/4780) ([Danila Kutenin](https://github.com/danlark1)) @@ -1537,7 +1537,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Improved data skipping indices calculation. [#4640](https://github.com/ClickHouse/ClickHouse/pull/4640) ([Nikita Vasilev](https://github.com/nikvas0)) - Keep ordinary, `DEFAULT`, `MATERIALIZED` and `ALIAS` columns in a single list (fixes issue [#2867](https://github.com/ClickHouse/ClickHouse/issues/2867)). [#4707](https://github.com/ClickHouse/ClickHouse/pull/4707) ([Alex Zatelepin](https://github.com/ztlpn)) -#### Bug Fix {#bug-fix-26} +#### Bug fix {#bug-fix-26} - Avoid `std::terminate` in case of memory allocation failure. Now `std::bad_alloc` exception is thrown as expected. [#4665](https://github.com/ClickHouse/ClickHouse/pull/4665) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Fixes `capnproto` reading from buffer. Sometimes files wasn't loaded successfully by HTTP. [#4674](https://github.com/ClickHouse/ClickHouse/pull/4674) ([Vladislav](https://github.com/smirnov-vs)) @@ -1577,18 +1577,18 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Fix function `toISOWeek` result for year 1970. [#4988](https://github.com/ClickHouse/ClickHouse/pull/4988) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Fix `DROP`, `TRUNCATE` and `OPTIMIZE` queries duplication, when executed on `ON CLUSTER` for `ReplicatedMergeTree*` tables family. [#4991](https://github.com/ClickHouse/ClickHouse/pull/4991) ([alesapin](https://github.com/alesapin)) -#### Backward Incompatible Change {#backward-incompatible-change-8} +#### Backward incompatible change {#backward-incompatible-change-8} - Rename setting `insert_sample_with_metadata` to setting `input_format_defaults_for_omitted_fields`. [#4771](https://github.com/ClickHouse/ClickHouse/pull/4771) ([Artem Zuikov](https://github.com/4ertus2)) - Added setting `max_partitions_per_insert_block` (with value 100 by default). If inserted block contains larger number of partitions, an exception is thrown. Set it to 0 if you want to remove the limit (not recommended). [#4845](https://github.com/ClickHouse/ClickHouse/pull/4845) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Multi-search functions were renamed (`multiPosition` to `multiSearchAllPositions`, `multiSearch` to `multiSearchAny`, `firstMatch` to `multiSearchFirstIndex`). [#4780](https://github.com/ClickHouse/ClickHouse/pull/4780) ([Danila Kutenin](https://github.com/danlark1)) -#### Performance Improvement {#performance-improvement-6} +#### Performance improvement {#performance-improvement-6} - Optimize Volnitsky searcher by inlining, giving about 5-10% search improvement for queries with many needles or many similar bigrams. [#4862](https://github.com/ClickHouse/ClickHouse/pull/4862) ([Danila Kutenin](https://github.com/danlark1)) - Fix performance issue when setting `use_uncompressed_cache` is greater than zero, which appeared when all read data contained in cache. [#4913](https://github.com/ClickHouse/ClickHouse/pull/4913) ([alesapin](https://github.com/alesapin)) -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-10} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-10} - Hardening debug build: more granular memory mappings and ASLR; add memory protection for mark cache and index. This allows to find more memory stomping bugs in case when ASan and MSan cannot do it. [#4632](https://github.com/ClickHouse/ClickHouse/pull/4632) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Add support for cmake variables `ENABLE_PROTOBUF`, `ENABLE_PARQUET` and `ENABLE_BROTLI` which allows to enable/disable the above features (same as we can do for librdkafka, mysql, etc). [#4669](https://github.com/ClickHouse/ClickHouse/pull/4669) ([Silviu Caragea](https://github.com/silviucpp)) @@ -1603,11 +1603,11 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Disable usage of `mremap` when compiled with Thread Sanitizer. Surprisingly enough, TSan does not intercept `mremap` (though it does intercept `mmap`, `munmap`) that leads to false positives. Fixed TSan report in stateful tests. [#4859](https://github.com/ClickHouse/ClickHouse/pull/4859) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Add test checking using format schema via HTTP interface. [#4864](https://github.com/ClickHouse/ClickHouse/pull/4864) ([Vitaly Baranov](https://github.com/vitlibar)) -## ClickHouse Release 19.4 {#clickhouse-release-19-4} +## ClickHouse release 19.4 {#clickhouse-release-19-4} -### ClickHouse Release 19.4.4.33, 2019-04-17 {#clickhouse-release-19-4-4-33-2019-04-17} +### ClickHouse release 19.4.4.33, 2019-04-17 {#clickhouse-release-19-4-4-33-2019-04-17} -#### Bug Fixes {#bug-fixes-7} +#### Bug fixes {#bug-fixes-7} - Avoid `std::terminate` in case of memory allocation failure. Now `std::bad_alloc` exception is thrown as expected. [#4665](https://github.com/ClickHouse/ClickHouse/pull/4665) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Fixes `capnproto` reading from buffer. Sometimes files wasn't loaded successfully by HTTP. [#4674](https://github.com/ClickHouse/ClickHouse/pull/4674) ([Vladislav](https://github.com/smirnov-vs)) @@ -1646,32 +1646,32 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Keep ordinary, `DEFAULT`, `MATERIALIZED` and `ALIAS` columns in a single list (fixes issue [#2867](https://github.com/ClickHouse/ClickHouse/issues/2867)). [#4707](https://github.com/ClickHouse/ClickHouse/pull/4707) ([Alex Zatelepin](https://github.com/ztlpn)) -### ClickHouse Release 19.4.3.11, 2019-04-02 {#clickhouse-release-19-4-3-11-2019-04-02} +### ClickHouse release 19.4.3.11, 2019-04-02 {#clickhouse-release-19-4-3-11-2019-04-02} -#### Bug Fixes {#bug-fixes-8} +#### Bug fixes {#bug-fixes-8} - Fix crash in `FULL/RIGHT JOIN` when we joining on nullable vs not nullable. [#4855](https://github.com/ClickHouse/ClickHouse/pull/4855) ([Artem Zuikov](https://github.com/4ertus2)) - Fix segmentation fault in `clickhouse-copier`. [#4835](https://github.com/ClickHouse/ClickHouse/pull/4835) ([proller](https://github.com/proller)) -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-11} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-11} - Add a way to launch clickhouse-server image from a custom user. [#4753](https://github.com/ClickHouse/ClickHouse/pull/4753) ([Mikhail f. Shiryaev](https://github.com/Felixoid)) -### ClickHouse Release 19.4.2.7, 2019-03-30 {#clickhouse-release-19-4-2-7-2019-03-30} +### ClickHouse release 19.4.2.7, 2019-03-30 {#clickhouse-release-19-4-2-7-2019-03-30} -#### Bug Fixes {#bug-fixes-9} +#### Bug fixes {#bug-fixes-9} - Fixed reading from `Array(LowCardinality)` column in rare case when column contained a long sequence of empty arrays. [#4850](https://github.com/ClickHouse/ClickHouse/pull/4850) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) -### ClickHouse Release 19.4.1.3, 2019-03-19 {#clickhouse-release-19-4-1-3-2019-03-19} +### ClickHouse release 19.4.1.3, 2019-03-19 {#clickhouse-release-19-4-1-3-2019-03-19} -#### Bug Fixes {#bug-fixes-10} +#### Bug fixes {#bug-fixes-10} - Fixed remote queries which contain both `LIMIT BY` and `LIMIT`. Previously, if `LIMIT BY` and `LIMIT` were used for remote query, `LIMIT` could happen before `LIMIT BY`, which led to too filtered result. [#4708](https://github.com/ClickHouse/ClickHouse/pull/4708) ([Constantin S. Pan](https://github.com/kvap)) -### ClickHouse Release 19.4.0.49, 2019-03-09 {#clickhouse-release-19-4-0-49-2019-03-09} +### ClickHouse release 19.4.0.49, 2019-03-09 {#clickhouse-release-19-4-0-49-2019-03-09} -#### New Features {#new-features-5} +#### New features {#new-features-5} - Added full support for `Protobuf` format (input and output, nested data structures). [#4174](https://github.com/ClickHouse/ClickHouse/pull/4174) [#4493](https://github.com/ClickHouse/ClickHouse/pull/4493) ([Vitaly Baranov](https://github.com/vitlibar)) - Added bitmap functions with Roaring Bitmaps. [#4207](https://github.com/ClickHouse/ClickHouse/pull/4207) ([Andy Yang](https://github.com/andyyzh)) [#4568](https://github.com/ClickHouse/ClickHouse/pull/4568) ([Vitaly Baranov](https://github.com/vitlibar)) @@ -1683,7 +1683,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Added functions `arrayEnumerateDenseRanked` and `arrayEnumerateUniqRanked` (it's like `arrayEnumerateUniq` but allows to fine tune array depth to look inside multidimensional arrays). [#4475](https://github.com/ClickHouse/ClickHouse/pull/4475) ([proller](https://github.com/proller)) [#4601](https://github.com/ClickHouse/ClickHouse/pull/4601) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Multiple JOINS with some restrictions: no asterisks, no complex aliases in ON/WHERE/GROUP BY/… [#4462](https://github.com/ClickHouse/ClickHouse/pull/4462) ([Artem Zuikov](https://github.com/4ertus2)) -#### Bug Fixes {#bug-fixes-11} +#### Bug fixes {#bug-fixes-11} - This release also contains all bug fixes from 19.3 and 19.1. - Fixed bug in data skipping indices: order of granules after INSERT was incorrect. [#4407](https://github.com/ClickHouse/ClickHouse/pull/4407) ([Nikita Vasilev](https://github.com/nikvas0)) @@ -1712,7 +1712,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Result of multiple JOINs need correct result names to be used in subselects. Replace flat aliases with source names in result. [#4474](https://github.com/ClickHouse/ClickHouse/pull/4474) ([Artem Zuikov](https://github.com/4ertus2)) - Improve push-down logic for joined statements. [#4387](https://github.com/ClickHouse/ClickHouse/pull/4387) ([Ivan](https://github.com/abyss7)) -#### Performance Improvements {#performance-improvements-3} +#### Performance improvements {#performance-improvements-3} - Improved heuristics of "move to PREWHERE" optimization. [#4405](https://github.com/ClickHouse/ClickHouse/pull/4405) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Use proper lookup tables that uses HashTable's API for 8-bit and 16-bit keys. [#4536](https://github.com/ClickHouse/ClickHouse/pull/4536) ([Amos Bird](https://github.com/amosbird)) @@ -1720,7 +1720,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Cleanup distributed DDL queue in a separate thread so that it does not slow down the main loop that processes distributed DDL tasks. [#4502](https://github.com/ClickHouse/ClickHouse/pull/4502) ([Alex Zatelepin](https://github.com/ztlpn)) - When `min_bytes_to_use_direct_io` is set to 1, not every file was opened with O_DIRECT mode because the data size to read was sometimes underestimated by the size of one compressed block. [#4526](https://github.com/ClickHouse/ClickHouse/pull/4526) ([alexey-milovidov](https://github.com/alexey-milovidov)) -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-12} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-12} - Added support for clang-9 [#4604](https://github.com/ClickHouse/ClickHouse/pull/4604) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Fix wrong `__asm__` instructions (again) [#4621](https://github.com/ClickHouse/ClickHouse/pull/4621) ([Konstantin Podshumok](https://github.com/podshumok)) @@ -1733,29 +1733,29 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Fix compilation on Mac. [#4371](https://github.com/ClickHouse/ClickHouse/pull/4371) ([Vitaly Baranov](https://github.com/vitlibar)) - Build fixes for FreeBSD and various unusual build configurations. [#4444](https://github.com/ClickHouse/ClickHouse/pull/4444) ([proller](https://github.com/proller)) -## ClickHouse Release 19.3 {#clickhouse-release-19-3} +## ClickHouse release 19.3 {#clickhouse-release-19-3} -### ClickHouse Release 19.3.9.1, 2019-04-02 {#clickhouse-release-19-3-9-1-2019-04-02} +### ClickHouse release 19.3.9.1, 2019-04-02 {#clickhouse-release-19-3-9-1-2019-04-02} -#### Bug Fixes {#bug-fixes-12} +#### Bug fixes {#bug-fixes-12} - Fix crash in `FULL/RIGHT JOIN` when we joining on nullable vs not nullable. [#4855](https://github.com/ClickHouse/ClickHouse/pull/4855) ([Artem Zuikov](https://github.com/4ertus2)) - Fix segmentation fault in `clickhouse-copier`. [#4835](https://github.com/ClickHouse/ClickHouse/pull/4835) ([proller](https://github.com/proller)) - Fixed reading from `Array(LowCardinality)` column in rare case when column contained a long sequence of empty arrays. [#4850](https://github.com/ClickHouse/ClickHouse/pull/4850) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-13} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-13} - Add a way to launch clickhouse-server image from a custom user [#4753](https://github.com/ClickHouse/ClickHouse/pull/4753) ([Mikhail f. Shiryaev](https://github.com/Felixoid)) -### ClickHouse Release 19.3.7, 2019-03-12 {#clickhouse-release-19-3-7-2019-03-12} +### ClickHouse release 19.3.7, 2019-03-12 {#clickhouse-release-19-3-7-2019-03-12} -#### Bug Fixes {#bug-fixes-13} +#### Bug fixes {#bug-fixes-13} - Fixed error in #3920. This error manifests itself as random cache corruption (messages `Unknown codec family code`, `Cannot seek through file`) and segfaults. This bug first appeared in version 19.1 and is present in versions up to 19.1.10 and 19.3.6. [#4623](https://github.com/ClickHouse/ClickHouse/pull/4623) ([alexey-milovidov](https://github.com/alexey-milovidov)) -### ClickHouse Release 19.3.6, 2019-03-02 {#clickhouse-release-19-3-6-2019-03-02} +### ClickHouse release 19.3.6, 2019-03-02 {#clickhouse-release-19-3-6-2019-03-02} -#### Bug Fixes {#bug-fixes-14} +#### Bug fixes {#bug-fixes-14} - When there are more than 1000 threads in a thread pool, `std::terminate` may happen on thread exit. [Azat Khuzhin](https://github.com/azat) [#4485](https://github.com/ClickHouse/ClickHouse/pull/4485) [#4505](https://github.com/ClickHouse/ClickHouse/pull/4505) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Now it's possible to create `ReplicatedMergeTree*` tables with comments on columns without defaults and tables with columns codecs without comments and defaults. Also fix comparison of codecs. [#4523](https://github.com/ClickHouse/ClickHouse/pull/4523) ([alesapin](https://github.com/alesapin)) @@ -1764,22 +1764,22 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Fixed hangup on server shutdown if distributed DDLs were used. [#4472](https://github.com/ClickHouse/ClickHouse/pull/4472) ([Alex Zatelepin](https://github.com/ztlpn)) - Incorrect column numbers were printed in error message about text format parsing for columns with number greater than 10. [#4484](https://github.com/ClickHouse/ClickHouse/pull/4484) ([alexey-milovidov](https://github.com/alexey-milovidov)) -#### Build/Testing/Packaging Improvements {#buildtestingpackaging-improvements-3} +#### Build/Testing/Packaging improvements {#buildtestingpackaging-improvements-3} - Fixed build with AVX enabled. [#4527](https://github.com/ClickHouse/ClickHouse/pull/4527) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Enable extended accounting and IO accounting based on good known version instead of kernel under which it is compiled. [#4541](https://github.com/ClickHouse/ClickHouse/pull/4541) ([nvartolomei](https://github.com/nvartolomei)) - Allow to skip setting of core_dump.size_limit, warning instead of throw if limit set fail. [#4473](https://github.com/ClickHouse/ClickHouse/pull/4473) ([proller](https://github.com/proller)) - Removed the `inline` tags of `void readBinary(...)` in `Field.cpp`. Also merged redundant `namespace DB` blocks. [#4530](https://github.com/ClickHouse/ClickHouse/pull/4530) ([hcz](https://github.com/hczhcz)) -### ClickHouse Release 19.3.5, 2019-02-21 {#clickhouse-release-19-3-5-2019-02-21} +### ClickHouse release 19.3.5, 2019-02-21 {#clickhouse-release-19-3-5-2019-02-21} -#### Bug Fixes {#bug-fixes-15} +#### Bug fixes {#bug-fixes-15} - Fixed bug with large http insert queries processing. [#4454](https://github.com/ClickHouse/ClickHouse/pull/4454) ([alesapin](https://github.com/alesapin)) - Fixed backward incompatibility with old versions due to wrong implementation of `send_logs_level` setting. [#4445](https://github.com/ClickHouse/ClickHouse/pull/4445) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Fixed backward incompatibility of table function `remote` introduced with column comments. [#4446](https://github.com/ClickHouse/ClickHouse/pull/4446) ([alexey-milovidov](https://github.com/alexey-milovidov)) -### ClickHouse Release 19.3.4, 2019-02-16 {#clickhouse-release-19-3-4-2019-02-16} +### ClickHouse release 19.3.4, 2019-02-16 {#clickhouse-release-19-3-4-2019-02-16} #### Improvements {#improvements-4} @@ -1788,20 +1788,20 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Allow to repair abandoned replica even if it already has huge number of nodes in its queue. [#4399](https://github.com/ClickHouse/ClickHouse/pull/4399) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Add one required argument to `SET` index (max stored rows number). [#4386](https://github.com/ClickHouse/ClickHouse/pull/4386) ([Nikita Vasilev](https://github.com/nikvas0)) -#### Bug Fixes {#bug-fixes-16} +#### Bug fixes {#bug-fixes-16} - Fixed `WITH ROLLUP` result for group by single `LowCardinality` key. [#4384](https://github.com/ClickHouse/ClickHouse/pull/4384) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) - Fixed bug in the set index (dropping a granule if it contains more than `max_rows` rows). [#4386](https://github.com/ClickHouse/ClickHouse/pull/4386) ([Nikita Vasilev](https://github.com/nikvas0)) - A lot of FreeBSD build fixes. [#4397](https://github.com/ClickHouse/ClickHouse/pull/4397) ([proller](https://github.com/proller)) - Fixed aliases substitution in queries with subquery containing same alias (issue [#4110](https://github.com/ClickHouse/ClickHouse/issues/4110)). [#4351](https://github.com/ClickHouse/ClickHouse/pull/4351) ([Artem Zuikov](https://github.com/4ertus2)) -#### Build/Testing/Packaging Improvements {#buildtestingpackaging-improvements-4} +#### Build/Testing/Packaging improvements {#buildtestingpackaging-improvements-4} - Add ability to run `clickhouse-server` for stateless tests in docker image. [#4347](https://github.com/ClickHouse/ClickHouse/pull/4347) ([Vasily Nemkov](https://github.com/Enmk)) -### ClickHouse Release 19.3.3, 2019-02-13 {#clickhouse-release-19-3-3-2019-02-13} +### ClickHouse release 19.3.3, 2019-02-13 {#clickhouse-release-19-3-3-2019-02-13} -#### New Features {#new-features-6} +#### New features {#new-features-6} - Added the `KILL MUTATION` statement that allows removing mutations that are for some reasons stuck. Added `latest_failed_part`, `latest_fail_time`, `latest_fail_reason` fields to the `system.mutations` table for easier troubleshooting. [#4287](https://github.com/ClickHouse/ClickHouse/pull/4287) ([Alex Zatelepin](https://github.com/ztlpn)) - Added aggregate function `entropy` which computes Shannon entropy. [#4238](https://github.com/ClickHouse/ClickHouse/pull/4238) ([Quid37](https://github.com/Quid37)) @@ -1822,12 +1822,12 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Added hints while user make typo in function name or type in command line client. [#4239](https://github.com/ClickHouse/ClickHouse/pull/4239) ([Danila Kutenin](https://github.com/danlark1)) - Added `Query-Id` to Server's HTTP Response header. [#4231](https://github.com/ClickHouse/ClickHouse/pull/4231) ([Mikhail](https://github.com/fandyushin)) -#### Experimental Features {#experimental-features-2} +#### Experimental features {#experimental-features-2} - Added `minmax` and `set` data skipping indices for MergeTree table engines family. [#4143](https://github.com/ClickHouse/ClickHouse/pull/4143) ([Nikita Vasilev](https://github.com/nikvas0)) - Added conversion of `CROSS JOIN` to `INNER JOIN` if possible. [#4221](https://github.com/ClickHouse/ClickHouse/pull/4221) [#4266](https://github.com/ClickHouse/ClickHouse/pull/4266) ([Artem Zuikov](https://github.com/4ertus2)) -#### Bug Fixes {#bug-fixes-17} +#### Bug fixes {#bug-fixes-17} - Fixed `Not found column` for duplicate columns in `JOIN ON` section. [#4279](https://github.com/ClickHouse/ClickHouse/pull/4279) ([Artem Zuikov](https://github.com/4ertus2)) - Make `START REPLICATED SENDS` command start replicated sends. [#4229](https://github.com/ClickHouse/ClickHouse/pull/4229) ([nvartolomei](https://github.com/nvartolomei)) @@ -1857,7 +1857,7 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Fixed rare race condition when removing of old data parts can fail with `File not found` error. [#4378](https://github.com/ClickHouse/ClickHouse/pull/4378) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Fix install package with missing /etc/clickhouse-server/config.xml. [#4343](https://github.com/ClickHouse/ClickHouse/pull/4343) ([proller](https://github.com/proller)) -#### Build/Testing/Packaging Improvements {#buildtestingpackaging-improvements-5} +#### Build/Testing/Packaging improvements {#buildtestingpackaging-improvements-5} - Debian package: correct /etc/clickhouse-server/preprocessed link according to config. [#4205](https://github.com/ClickHouse/ClickHouse/pull/4205) ([proller](https://github.com/proller)) - Various build fixes for FreeBSD. [#4225](https://github.com/ClickHouse/ClickHouse/pull/4225) ([proller](https://github.com/proller)) @@ -1878,14 +1878,14 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Added checking SSE and AVX instruction at start. [#4234](https://github.com/ClickHouse/ClickHouse/pull/4234) ([Igr](https://github.com/igron99)) - Init script will wait server until start. [#4281](https://github.com/ClickHouse/ClickHouse/pull/4281) ([proller](https://github.com/proller)) -#### Backward Incompatible Changes {#backward-incompatible-changes-1} +#### Backward incompatible changes {#backward-incompatible-changes-1} - Removed `allow_experimental_low_cardinality_type` setting. `LowCardinality` data types are production ready. [#4323](https://github.com/ClickHouse/ClickHouse/pull/4323) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Reduce mark cache size and uncompressed cache size accordingly to available memory amount. [#4240](https://github.com/ClickHouse/ClickHouse/pull/4240) ([Lopatin Konstantin](https://github.com/k-lopatin) - Added keyword `INDEX` in `CREATE TABLE` query. A column with name `index` must be quoted with backticks or double quotes: `` `index` ``. [#4143](https://github.com/ClickHouse/ClickHouse/pull/4143) ([Nikita Vasilev](https://github.com/nikvas0)) - `sumMap` now promote result type instead of overflow. The old `sumMap` behavior can be obtained by using `sumMapWithOverflow` function. [#4151](https://github.com/ClickHouse/ClickHouse/pull/4151) ([Léo Ercolanelli](https://github.com/ercolanelli-leo)) -#### Performance Improvements {#performance-improvements-4} +#### Performance improvements {#performance-improvements-4} - `std::sort` replaced by `pdqsort` for queries without `LIMIT`. [#4236](https://github.com/ClickHouse/ClickHouse/pull/4236) ([Evgenii Pravda](https://github.com/kvinty)) - Now server reuse threads from global thread pool. This affects performance in some corner cases. [#4150](https://github.com/ClickHouse/ClickHouse/pull/4150) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -1906,40 +1906,40 @@ keywords: ['ClickHouse 2019', 'changelog 2019', 'release notes', 'version histor - Improved server shutdown time and ALTERs waiting time. [#4372](https://github.com/ClickHouse/ClickHouse/pull/4372) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Added info about the replicated_can_become_leader setting to system.replicas and add logging if the replica won't try to become leader. [#4379](https://github.com/ClickHouse/ClickHouse/pull/4379) ([Alex Zatelepin](https://github.com/ztlpn)) -## ClickHouse Release 19.1 {#clickhouse-release-19-1} +## ClickHouse release 19.1 {#clickhouse-release-19-1} -### ClickHouse Release 19.1.14, 2019-03-14 {#clickhouse-release-19-1-14-2019-03-14} +### ClickHouse release 19.1.14, 2019-03-14 {#clickhouse-release-19-1-14-2019-03-14} - Fixed error `Column ... queried more than once` that may happen if the setting `asterisk_left_columns_only` is set to 1 in case of using `GLOBAL JOIN` with `SELECT *` (rare case). The issue does not exist in 19.3 and newer. [6bac7d8d](https://github.com/ClickHouse/ClickHouse/pull/4692/commits/6bac7d8d11a9b0d6de0b32b53c47eb2f6f8e7062) ([Artem Zuikov](https://github.com/4ertus2)) -### ClickHouse Release 19.1.13, 2019-03-12 {#clickhouse-release-19-1-13-2019-03-12} +### ClickHouse release 19.1.13, 2019-03-12 {#clickhouse-release-19-1-13-2019-03-12} This release contains exactly the same set of patches as 19.3.7. -### ClickHouse Release 19.1.10, 2019-03-03 {#clickhouse-release-19-1-10-2019-03-03} +### ClickHouse release 19.1.10, 2019-03-03 {#clickhouse-release-19-1-10-2019-03-03} This release contains exactly the same set of patches as 19.3.6. -## ClickHouse Release 19.1 {#clickhouse-release-19-1-1} +## ClickHouse release 19.1 {#clickhouse-release-19-1-1} -### ClickHouse Release 19.1.9, 2019-02-21 {#clickhouse-release-19-1-9-2019-02-21} +### ClickHouse release 19.1.9, 2019-02-21 {#clickhouse-release-19-1-9-2019-02-21} -#### Bug Fixes {#bug-fixes-18} +#### Bug fixes {#bug-fixes-18} - Fixed backward incompatibility with old versions due to wrong implementation of `send_logs_level` setting. [#4445](https://github.com/ClickHouse/ClickHouse/pull/4445) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Fixed backward incompatibility of table function `remote` introduced with column comments. [#4446](https://github.com/ClickHouse/ClickHouse/pull/4446) ([alexey-milovidov](https://github.com/alexey-milovidov)) -### ClickHouse Release 19.1.8, 2019-02-16 {#clickhouse-release-19-1-8-2019-02-16} +### ClickHouse release 19.1.8, 2019-02-16 {#clickhouse-release-19-1-8-2019-02-16} -#### Bug Fixes {#bug-fixes-19} +#### Bug fixes {#bug-fixes-19} - Fix install package with missing /etc/clickhouse-server/config.xml. [#4343](https://github.com/ClickHouse/ClickHouse/pull/4343) ([proller](https://github.com/proller)) -## ClickHouse Release 19.1 {#clickhouse-release-19-1-2} +## ClickHouse release 19.1 {#clickhouse-release-19-1-2} -### ClickHouse Release 19.1.7, 2019-02-15 {#clickhouse-release-19-1-7-2019-02-15} +### ClickHouse release 19.1.7, 2019-02-15 {#clickhouse-release-19-1-7-2019-02-15} -#### Bug Fixes {#bug-fixes-20} +#### Bug fixes {#bug-fixes-20} - Correctly return the right type and properly handle locks in `joinGet` function. [#4153](https://github.com/ClickHouse/ClickHouse/pull/4153) ([Amos Bird](https://github.com/amosbird)) - Fixed error when system logs are tried to create again at server shutdown. [#4254](https://github.com/ClickHouse/ClickHouse/pull/4254) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -1966,9 +1966,9 @@ This release contains exactly the same set of patches as 19.3.6. - Fixed bug with incorrect `Date` and `DateTime` comparison. [#4237](https://github.com/ClickHouse/ClickHouse/pull/4237) ([valexey](https://github.com/valexey)) - Fixed incorrect result when `Date` and `DateTime` arguments are used in branches of conditional operator (function `if`). Added generic case for function `if`. [#4243](https://github.com/ClickHouse/ClickHouse/pull/4243) ([alexey-milovidov](https://github.com/alexey-milovidov)) -### ClickHouse Release 19.1.6, 2019-01-24 {#clickhouse-release-19-1-6-2019-01-24} +### ClickHouse release 19.1.6, 2019-01-24 {#clickhouse-release-19-1-6-2019-01-24} -#### New Features {#new-features-7} +#### New features {#new-features-7} - Custom per column compression codecs for tables. [#3899](https://github.com/ClickHouse/ClickHouse/pull/3899) [#4111](https://github.com/ClickHouse/ClickHouse/pull/4111) ([alesapin](https://github.com/alesapin), [Winter Zhang](https://github.com/zhang2014), [Anatoly](https://github.com/Sindbag)) - Added compression codec `Delta`. [#4052](https://github.com/ClickHouse/ClickHouse/pull/4052) ([alesapin](https://github.com/alesapin)) @@ -1985,11 +1985,11 @@ This release contains exactly the same set of patches as 19.3.6. - Added hashing functions `javaHash`, `hiveHash`. [#3811](https://github.com/ClickHouse/ClickHouse/pull/3811) ([shangshujie365](https://github.com/shangshujie365)) - Added table function `remoteSecure`. Function works as `remote`, but uses secure connection. [#4088](https://github.com/ClickHouse/ClickHouse/pull/4088) ([proller](https://github.com/proller)) -#### Experimental Features {#experimental-features-3} +#### Experimental features {#experimental-features-3} - Added multiple JOINs emulation (`allow_experimental_multiple_joins_emulation` setting). [#3946](https://github.com/ClickHouse/ClickHouse/pull/3946) ([Artem Zuikov](https://github.com/4ertus2)) -#### Bug Fixes {#bug-fixes-21} +#### Bug fixes {#bug-fixes-21} - Make `compiled_expression_cache_size` setting limited by default to lower memory consumption. [#4041](https://github.com/ClickHouse/ClickHouse/pull/4041) ([alesapin](https://github.com/alesapin)) - Fix a bug that led to hangups in threads that perform ALTERs of Replicated tables and in the thread that updates configuration from ZooKeeper. [#2947](https://github.com/ClickHouse/ClickHouse/issues/2947) [#3891](https://github.com/ClickHouse/ClickHouse/issues/3891) [#3934](https://github.com/ClickHouse/ClickHouse/pull/3934) ([Alex Zatelepin](https://github.com/ztlpn)) @@ -2033,7 +2033,7 @@ This release contains exactly the same set of patches as 19.3.6. - Add check that `SET send_logs_level = 'value'` query accept appropriate value. [#3873](https://github.com/ClickHouse/ClickHouse/pull/3873) ([Sabyanin Maxim](https://github.com/s-mx)) - Fixed data type check in type conversion functions. [#3896](https://github.com/ClickHouse/ClickHouse/pull/3896) ([Winter Zhang](https://github.com/zhang2014)) -#### Performance Improvements {#performance-improvements-5} +#### Performance improvements {#performance-improvements-5} - Add a MergeTree setting `use_minimalistic_part_header_in_zookeeper`. If enabled, Replicated tables will store compact part metadata in a single part znode. This can dramatically reduce ZooKeeper snapshot size (especially if the tables have a lot of columns). Note that after enabling this setting you will not be able to downgrade to a version that does not support it. [#3960](https://github.com/ClickHouse/ClickHouse/pull/3960) ([Alex Zatelepin](https://github.com/ztlpn)) - Add an DFA-based implementation for functions `sequenceMatch` and `sequenceCount` in case pattern does not contain time. [#4004](https://github.com/ClickHouse/ClickHouse/pull/4004) ([Léo Ercolanelli](https://github.com/ercolanelli-leo)) @@ -2041,13 +2041,13 @@ This release contains exactly the same set of patches as 19.3.6. - Zero left padding PODArray so that -1 element is always valid and zeroed. It's used for branchless calculation of offsets. [#3920](https://github.com/ClickHouse/ClickHouse/pull/3920) ([Amos Bird](https://github.com/amosbird)) - Reverted `jemalloc` version which lead to performance degradation. [#4018](https://github.com/ClickHouse/ClickHouse/pull/4018) ([alexey-milovidov](https://github.com/alexey-milovidov)) -#### Backward Incompatible Changes {#backward-incompatible-changes-2} +#### Backward incompatible changes {#backward-incompatible-changes-2} - Removed undocumented feature `ALTER MODIFY PRIMARY KEY` because it was superseded by the `ALTER MODIFY ORDER BY` command. [#3887](https://github.com/ClickHouse/ClickHouse/pull/3887) ([Alex Zatelepin](https://github.com/ztlpn)) - Removed function `shardByHash`. [#3833](https://github.com/ClickHouse/ClickHouse/pull/3833) ([alexey-milovidov](https://github.com/alexey-milovidov)) - Forbid using scalar subqueries with result of type `AggregateFunction`. [#3865](https://github.com/ClickHouse/ClickHouse/pull/3865) ([Ivan](https://github.com/abyss7)) -#### Build/Testing/Packaging Improvements {#buildtestingpackaging-improvements-6} +#### Build/Testing/Packaging improvements {#buildtestingpackaging-improvements-6} - Added support for PowerPC (`ppc64le`) build. [#4132](https://github.com/ClickHouse/ClickHouse/pull/4132) ([Danila Kutenin](https://github.com/danlark1)) - Stateful functional tests are run on public available dataset. [#3969](https://github.com/ClickHouse/ClickHouse/pull/3969) ([alexey-milovidov](https://github.com/alexey-milovidov)) diff --git a/docs/whats-new/changelog/2020.md b/docs/whats-new/changelog/2020.md index ee9c17ecddb..dc9012b81b7 100644 --- a/docs/whats-new/changelog/2020.md +++ b/docs/whats-new/changelog/2020.md @@ -12,7 +12,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor ### ClickHouse release v20.12.5.14-stable, 2020-12-28 {#clickhouse-release-v2012514-stable-2020-12-28} -#### Bug Fix {#bug-fix} +#### Bug fix {#bug-fix} * Disable write with AIO during merges because it can lead to extremely rare data corruption of primary key columns during merge. [#18481](https://github.com/ClickHouse/ClickHouse/pull/18481) ([alesapin](https://github.com/alesapin)). * Fixed `value is too short` error when executing `toType(...)` functions (`toDate`, `toUInt32`, etc) with argument of type `Nullable(String)`. Now such functions return `NULL` on parsing errors instead of throwing exception. Fixes [#7673](https://github.com/ClickHouse/ClickHouse/issues/7673). [#18445](https://github.com/ClickHouse/ClickHouse/pull/18445) ([tavplubix](https://github.com/tavplubix)). @@ -21,13 +21,13 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Fix possible crashes in aggregate functions with combinator `Distinct`, while using two-level aggregation. Fixes [#17682](https://github.com/ClickHouse/ClickHouse/issues/17682). [#18365](https://github.com/ClickHouse/ClickHouse/pull/18365) ([Anton Popov](https://github.com/CurtizJ)). * Fix error when query `MODIFY COLUMN ... REMOVE TTL` does not actually remove column TTL. [#18130](https://github.com/ClickHouse/ClickHouse/pull/18130) ([alesapin](https://github.com/alesapin)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement} * Update timezones info to 2020e. [#18531](https://github.com/ClickHouse/ClickHouse/pull/18531) ([alesapin](https://github.com/alesapin)). ### ClickHouse release v20.12.4.5-stable, 2020-12-24 {#clickhouse-release-v201245-stable-2020-12-24} -#### Bug Fix {#bug-fix-1} +#### Bug fix {#bug-fix-1} * Fixed issue when `clickhouse-odbc-bridge` process is unreachable by server on machines with dual IPv4/IPv6 stack; - Fixed issue when ODBC dictionary updates are performed using malformed queries and/or cause crashes; Possibly closes [#14489](https://github.com/ClickHouse/ClickHouse/issues/14489). [#18278](https://github.com/ClickHouse/ClickHouse/pull/18278) ([Denis Glazachev](https://github.com/traceon)). * Fixed key comparison between Enum and Int types. This fixes [#17989](https://github.com/ClickHouse/ClickHouse/issues/17989). [#18214](https://github.com/ClickHouse/ClickHouse/pull/18214) ([Amos Bird](https://github.com/amosbird)). @@ -40,12 +40,12 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor ### ClickHouse release v20.12.3.3-stable, 2020-12-13 {#clickhouse-release-v201233-stable-2020-12-13} -#### Backward Incompatible Change {#backward-incompatible-change} +#### Backward incompatible change {#backward-incompatible-change} * Enable `use_compact_format_in_distributed_parts_names` by default (see the documentation for the reference). [#16728](https://github.com/ClickHouse/ClickHouse/pull/16728) ([Azat Khuzhin](https://github.com/azat)). * Accept user settings related to file formats (e.g. `format_csv_delimiter`) in the `SETTINGS` clause when creating a table that uses `File` engine, and use these settings in all `INSERT`s and `SELECT`s. The file format settings changed in the current user session, or in the `SETTINGS` clause of a DML query itself, no longer affect the query. [#16591](https://github.com/ClickHouse/ClickHouse/pull/16591) ([Alexander Kuzmenkov](https://github.com/akuzm)). -#### New Feature {#new-feature} +#### New feature {#new-feature} * add `*.xz` compression/decompression support.It enables using `*.xz` in `file()` function. This closes [#8828](https://github.com/ClickHouse/ClickHouse/issues/8828). [#16578](https://github.com/ClickHouse/ClickHouse/pull/16578) ([Abi Palagashvili](https://github.com/fibersel)). * Introduce the query `ALTER TABLE ... DROP|DETACH PART 'part_name'`. [#15511](https://github.com/ClickHouse/ClickHouse/pull/15511) ([nvartolomei](https://github.com/nvartolomei)). @@ -60,7 +60,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Updated DateTime, DateTime64 parsing to accept string Date literal format. [#16040](https://github.com/ClickHouse/ClickHouse/pull/16040) ([Maksim Kita](https://github.com/kitaisreal)). * Make it possible to change the path to history file in `clickhouse-client` using the `--history_file` parameter. [#15960](https://github.com/ClickHouse/ClickHouse/pull/15960) ([Maksim Kita](https://github.com/kitaisreal)). -#### Bug Fix {#bug-fix-2} +#### Bug fix {#bug-fix-2} * Fix the issue when server can stop accepting connections in very rare cases. [#17542](https://github.com/ClickHouse/ClickHouse/pull/17542) ([Amos Bird](https://github.com/amosbird)). * Fixed `Function not implemented` error when executing `RENAME` query in `Atomic` database with ClickHouse running on Windows Subsystem for Linux. Fixes [#17661](https://github.com/ClickHouse/ClickHouse/issues/17661). [#17664](https://github.com/ClickHouse/ClickHouse/pull/17664) ([tavplubix](https://github.com/tavplubix)). @@ -135,7 +135,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Improvement for MaterializeMySQL (experimental feature). Throw exception about right sync privileges when MySQL sync user has error privileges. [#15977](https://github.com/ClickHouse/ClickHouse/pull/15977) ([TCeason](https://github.com/TCeason)). * Made `indexOf()` use BloomFilter. [#14977](https://github.com/ClickHouse/ClickHouse/pull/14977) ([achimbab](https://github.com/achimbab)). -#### Performance Improvement {#performance-improvement} +#### Performance improvement {#performance-improvement} * Use Floyd-Rivest algorithm, it is the best for the ClickHouse use case of partial sorting. Bechmarks are in https://github.com/danlark1/miniselect and [here](https://drive.google.com/drive/folders/1DHEaeXgZuX6AJ9eByeZ8iQVQv0ueP8XM). [#16825](https://github.com/ClickHouse/ClickHouse/pull/16825) ([Danila Kutenin](https://github.com/danlark1)). * Now `ReplicatedMergeTree` tree engines family uses a separate thread pool for replicated fetches. Size of the pool limited by setting `background_fetches_pool_size` which can be tuned with a server restart. The default value of the setting is 3 and it means that the maximum amount of parallel fetches is equal to 3 (and it allows to utilize 10G network). Fixes #520. [#16390](https://github.com/ClickHouse/ClickHouse/pull/16390) ([alesapin](https://github.com/alesapin)). @@ -145,11 +145,11 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Fix performance of reading from `Merge` tables over huge number of `MergeTree` tables. Fixes [#7748](https://github.com/ClickHouse/ClickHouse/issues/7748). [#16988](https://github.com/ClickHouse/ClickHouse/pull/16988) ([Anton Popov](https://github.com/CurtizJ)). * Now we can safely prune partitions with exact match. Useful case: Suppose table is partitioned by `intHash64(x) % 100` and the query has condition on `intHash64(x) % 100` verbatim, not on x. [#16253](https://github.com/ClickHouse/ClickHouse/pull/16253) ([Amos Bird](https://github.com/amosbird)). -#### Experimental Feature {#experimental-feature} +#### Experimental feature {#experimental-feature} * Add `EmbeddedRocksDB` table engine (can be used for dictionaries). [#15073](https://github.com/ClickHouse/ClickHouse/pull/15073) ([sundyli](https://github.com/sundy-li)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-1} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-1} * Improvements in test coverage building images. [#17233](https://github.com/ClickHouse/ClickHouse/pull/17233) ([alesapin](https://github.com/alesapin)). * Update embedded timezone data to version 2020d (also update cctz to the latest master). [#17204](https://github.com/ClickHouse/ClickHouse/pull/17204) ([filimonov](https://github.com/filimonov)). @@ -166,7 +166,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Explicitly set uid / gid of clickhouse user & group to the fixed values (101) in clickhouse-server images. [#19096](https://github.com/ClickHouse/ClickHouse/pull/19096) ([filimonov](https://github.com/filimonov)). -#### Bug Fix {#bug-fix-3} +#### Bug fix {#bug-fix-3} * BloomFilter index crash fix. Fixes [#19757](https://github.com/ClickHouse/ClickHouse/issues/19757). [#19884](https://github.com/ClickHouse/ClickHouse/pull/19884) ([Maksim Kita](https://github.com/kitaisreal)). * Deadlock was possible if system.text_log is enabled. This fixes [#19874](https://github.com/ClickHouse/ClickHouse/issues/19874). [#19875](https://github.com/ClickHouse/ClickHouse/pull/19875) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -218,13 +218,13 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Fix indeterministic functions with predicate optimizer. This fixes [#17244](https://github.com/ClickHouse/ClickHouse/issues/17244). [#17273](https://github.com/ClickHouse/ClickHouse/pull/17273) ([Winter Zhang](https://github.com/zhang2014)). * Mutation might hang waiting for some non-existent part after `MOVE` or `REPLACE PARTITION` or, in rare cases, after `DETACH` or `DROP PARTITION`. It's fixed. [#15537](https://github.com/ClickHouse/ClickHouse/pull/15537) ([tavplubix](https://github.com/tavplubix)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-2} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-2} * Update timezones info to 2020e. [#18531](https://github.com/ClickHouse/ClickHouse/pull/18531) ([alesapin](https://github.com/alesapin)). ### ClickHouse release v20.11.6.6-stable, 2020-12-24 {#clickhouse-release-v201166-stable-2020-12-24} -#### Bug Fix {#bug-fix-4} +#### Bug fix {#bug-fix-4} * Fixed issue when `clickhouse-odbc-bridge` process is unreachable by server on machines with dual `IPv4/IPv6 stack` and fixed issue when ODBC dictionary updates are performed using malformed queries and/or cause crashes. This possibly closes [#14489](https://github.com/ClickHouse/ClickHouse/issues/14489). [#18278](https://github.com/ClickHouse/ClickHouse/pull/18278) ([Denis Glazachev](https://github.com/traceon)). * Fixed key comparison between Enum and Int types. This fixes [#17989](https://github.com/ClickHouse/ClickHouse/issues/17989). [#18214](https://github.com/ClickHouse/ClickHouse/pull/18214) ([Amos Bird](https://github.com/amosbird)). @@ -269,13 +269,13 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor ### ClickHouse release v20.11.3.3-stable, 2020-11-13 {#clickhouse-release-v201133-stable-2020-11-13} -#### Bug Fix {#bug-fix-5} +#### Bug fix {#bug-fix-5} * Fix rare silent crashes when query profiler is on and ClickHouse is installed on OS with glibc version that has (supposedly) broken asynchronous unwind tables for some functions. This fixes [#15301](https://github.com/ClickHouse/ClickHouse/issues/15301). This fixes [#13098](https://github.com/ClickHouse/ClickHouse/issues/13098). [#16846](https://github.com/ClickHouse/ClickHouse/pull/16846) ([alexey-milovidov](https://github.com/alexey-milovidov)). ### ClickHouse release v20.11.2.1, 2020-11-11 {#clickhouse-release-v201121-2020-11-11} -#### Backward Incompatible Change {#backward-incompatible-change-1} +#### Backward incompatible change {#backward-incompatible-change-1} * If some `profile` was specified in `distributed_ddl` config section, then this profile could overwrite settings of `default` profile on server startup. It's fixed, now settings of distributed DDL queries should not affect global server settings. [#16635](https://github.com/ClickHouse/ClickHouse/pull/16635) ([tavplubix](https://github.com/tavplubix)). * Restrict to use of non-comparable data types (like `AggregateFunction`) in keys (Sorting key, Primary key, Partition key, and so on). [#16601](https://github.com/ClickHouse/ClickHouse/pull/16601) ([alesapin](https://github.com/alesapin)). @@ -284,7 +284,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Make `rankCorr` function return nan on insufficient data [#16124](https://github.com/ClickHouse/ClickHouse/issues/16124). [#16135](https://github.com/ClickHouse/ClickHouse/pull/16135) ([hexiaoting](https://github.com/hexiaoting)). * When upgrading from versions older than 20.5, if rolling update is performed and cluster contains both versions 20.5 or greater and less than 20.5, if ClickHouse nodes with old versions are restarted and old version has been started up in presence of newer versions, it may lead to `Part ... intersects previous part` errors. To prevent this error, first install newer clickhouse-server packages on all cluster nodes and then do restarts (so, when clickhouse-server is restarted, it will start up with the new version). -#### New Feature {#new-feature-1} +#### New feature {#new-feature-1} * Added support of LDAP as a user directory for locally non-existent users. [#12736](https://github.com/ClickHouse/ClickHouse/pull/12736) ([Denis Glazachev](https://github.com/traceon)). * Add `system.replicated_fetches` table which shows currently running background fetches. [#16428](https://github.com/ClickHouse/ClickHouse/pull/16428) ([alesapin](https://github.com/alesapin)). @@ -308,11 +308,11 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Add function `formatReadableTimeDelta` that format time delta to human readable string ... [#15497](https://github.com/ClickHouse/ClickHouse/pull/15497) ([Filipe Caixeta](https://github.com/filipecaixeta)). * Added `disable_merges` option for volumes in multi-disk configuration. [#13956](https://github.com/ClickHouse/ClickHouse/pull/13956) ([Vladimir Chebotarev](https://github.com/excitoon)). -#### Experimental Feature {#experimental-feature-1} +#### Experimental feature {#experimental-feature-1} * New functions `encrypt`, `aes_encrypt_mysql`, `decrypt`, `aes_decrypt_mysql`. These functions are working slowly, so we consider it as an experimental feature. [#11844](https://github.com/ClickHouse/ClickHouse/pull/11844) ([Vasily Nemkov](https://github.com/Enmk)). -#### Bug Fix {#bug-fix-6} +#### Bug fix {#bug-fix-6} * Mask password in data_path in the `system.distribution_queue`. [#16727](https://github.com/ClickHouse/ClickHouse/pull/16727) ([Azat Khuzhin](https://github.com/azat)). * Fix `IN` operator over several columns and tuples with enabled `transform_null_in` setting. Fixes [#15310](https://github.com/ClickHouse/ClickHouse/issues/15310). [#16722](https://github.com/ClickHouse/ClickHouse/pull/16722) ([Anton Popov](https://github.com/CurtizJ)). @@ -369,7 +369,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Add reconnects to `zookeeper-dump-tree` tool. [#15711](https://github.com/ClickHouse/ClickHouse/pull/15711) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Allow explicitly specify columns list in `CREATE TABLE table AS table_function(...)` query. Fixes [#9249](https://github.com/ClickHouse/ClickHouse/issues/9249) Fixes [#14214](https://github.com/ClickHouse/ClickHouse/issues/14214). [#14295](https://github.com/ClickHouse/ClickHouse/pull/14295) ([tavplubix](https://github.com/tavplubix)). -#### Performance Improvement {#performance-improvement-1} +#### Performance improvement {#performance-improvement-1} * Do not merge parts across partitions in SELECT FINAL. [#15938](https://github.com/ClickHouse/ClickHouse/pull/15938) ([Kruglov Pavel](https://github.com/Avogar)). * Improve performance of `-OrNull` and `-OrDefault` aggregate functions. [#16661](https://github.com/ClickHouse/ClickHouse/pull/16661) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -379,7 +379,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Speedup hashed/sparse_hashed dictionary loading by preallocating the hash table. [#15454](https://github.com/ClickHouse/ClickHouse/pull/15454) ([Azat Khuzhin](https://github.com/azat)). * Now trivial count optimization becomes slightly non-trivial. Predicates that contain exact partition expr can be optimized too. This also fixes [#11092](https://github.com/ClickHouse/ClickHouse/issues/11092) which returns wrong count when `max_parallel_replicas > 1`. [#15074](https://github.com/ClickHouse/ClickHouse/pull/15074) ([Amos Bird](https://github.com/amosbird)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-3} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-3} * Add flaky check for stateless tests. It will detect potentially flaky functional tests in advance, before they are merged. [#16238](https://github.com/ClickHouse/ClickHouse/pull/16238) ([alesapin](https://github.com/alesapin)). * Use proper version for `croaring` instead of amalgamation. [#16285](https://github.com/ClickHouse/ClickHouse/pull/16285) ([sundyli](https://github.com/sundy-li)). @@ -396,7 +396,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor ### ClickHouse release v20.10.7.4-stable, 2020-12-24 {#clickhouse-release-v201074-stable-2020-12-24} -#### Bug Fix {#bug-fix-7} +#### Bug fix {#bug-fix-7} * Fixed issue when `clickhouse-odbc-bridge` process is unreachable by server on machines with dual `IPv4/IPv6` stack and fixed issue when ODBC dictionary updates are performed using malformed queries and/or cause crashes. This possibly closes [#14489](https://github.com/ClickHouse/ClickHouse/issues/14489). [#18278](https://github.com/ClickHouse/ClickHouse/pull/18278) ([Denis Glazachev](https://github.com/traceon)). * Fix key comparison between Enum and Int types. This fixes [#17989](https://github.com/ClickHouse/ClickHouse/issues/17989). [#18214](https://github.com/ClickHouse/ClickHouse/pull/18214) ([Amos Bird](https://github.com/amosbird)). @@ -435,7 +435,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor ### ClickHouse release v20.10.4.1-stable, 2020-11-13 {#clickhouse-release-v201041-stable-2020-11-13} -#### Bug Fix {#bug-fix-8} +#### Bug fix {#bug-fix-8} * Fix rare silent crashes when query profiler is on and ClickHouse is installed on OS with glibc version that has (supposedly) broken asynchronous unwind tables for some functions. This fixes [#15301](https://github.com/ClickHouse/ClickHouse/issues/15301). This fixes [#13098](https://github.com/ClickHouse/ClickHouse/issues/13098). [#16846](https://github.com/ClickHouse/ClickHouse/pull/16846) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix `IN` operator over several columns and tuples with enabled `transform_null_in` setting. Fixes [#15310](https://github.com/ClickHouse/ClickHouse/issues/15310). [#16722](https://github.com/ClickHouse/ClickHouse/pull/16722) ([Anton Popov](https://github.com/CurtizJ)). @@ -452,7 +452,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor ### ClickHouse release v20.10.3.30, 2020-10-28 {#clickhouse-release-v2010330-2020-10-28} -#### Backward Incompatible Change {#backward-incompatible-change-2} +#### Backward incompatible change {#backward-incompatible-change-2} * Make `multiple_joins_rewriter_version` obsolete. Remove first version of joins rewriter. [#15472](https://github.com/ClickHouse/ClickHouse/pull/15472) ([Artem Zuikov](https://github.com/4ertus2)). * Change default value of `format_regexp_escaping_rule` setting (it's related to `Regexp` format) to `Raw` (it means - read whole subpattern as a value) to make the behaviour more like to what users expect. [#15426](https://github.com/ClickHouse/ClickHouse/pull/15426) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -460,7 +460,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Added MergeTree settings (`max_replicated_merges_with_ttl_in_queue` and `max_number_of_merges_with_ttl_in_pool`) to control the number of merges with TTL in the background pool and replicated queue. This change breaks compatibility with older versions only if you use delete TTL. Otherwise, replication will stay compatible. You can avoid incompatibility issues if you update all shard replicas at once or execute `SYSTEM STOP TTL MERGES` until you finish the update of all replicas. If you'll get an incompatible entry in the replication queue, first of all, execute `SYSTEM STOP TTL MERGES` and after `ALTER TABLE ... DETACH PARTITION ...` the partition where incompatible TTL merge was assigned. Attach it back on a single replica. [#14490](https://github.com/ClickHouse/ClickHouse/pull/14490) ([alesapin](https://github.com/alesapin)). * When upgrading from versions older than 20.5, if rolling update is performed and cluster contains both versions 20.5 or greater and less than 20.5, if ClickHouse nodes with old versions are restarted and old version has been started up in presence of newer versions, it may lead to `Part ... intersects previous part` errors. To prevent this error, first install newer clickhouse-server packages on all cluster nodes and then do restarts (so, when clickhouse-server is restarted, it will start up with the new version). -#### New Feature {#new-feature-2} +#### New feature {#new-feature-2} * Background data recompression. Add the ability to specify `TTL ... RECOMPRESS codec_name` for MergeTree table engines family. [#14494](https://github.com/ClickHouse/ClickHouse/pull/14494) ([alesapin](https://github.com/alesapin)). * Add parallel quorum inserts. This closes [#15601](https://github.com/ClickHouse/ClickHouse/issues/15601). [#15601](https://github.com/ClickHouse/ClickHouse/pull/15601) ([Latysheva Alexandra](https://github.com/alexelex)). @@ -493,7 +493,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Add `event_time_microseconds` to `system.asynchronous_metric_log` & `system.metric_log` tables. [#14514](https://github.com/ClickHouse/ClickHouse/pull/14514) ([Bharat Nallan](https://github.com/bharatnc)). * Add `query_start_time_microseconds` field to `system.query_log` & `system.query_thread_log` tables. [#14252](https://github.com/ClickHouse/ClickHouse/pull/14252) ([Bharat Nallan](https://github.com/bharatnc)). -#### Bug Fix {#bug-fix-9} +#### Bug fix {#bug-fix-9} * Fix the case when memory can be overallocated regardless to the limit. This closes [#14560](https://github.com/ClickHouse/ClickHouse/issues/14560). [#16206](https://github.com/ClickHouse/ClickHouse/pull/16206) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix `executable` dictionary source hang. In previous versions, when using some formats (e.g. `JSONEachRow`) data was not feed to a child process before it outputs at least something. This closes [#1697](https://github.com/ClickHouse/ClickHouse/issues/1697). This closes [#2455](https://github.com/ClickHouse/ClickHouse/issues/2455). [#14525](https://github.com/ClickHouse/ClickHouse/pull/14525) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -614,7 +614,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * For INSERTs with inline data in VALUES format in the script mode of `clickhouse-client`, support semicolon as the data terminator, in addition to the new line. Closes [#12288](https://github.com/ClickHouse/ClickHouse/issues/12288). [#13192](https://github.com/ClickHouse/ClickHouse/pull/13192) ([Alexander Kuzmenkov](https://github.com/akuzm)). * Support custom codecs in compact parts. [#12183](https://github.com/ClickHouse/ClickHouse/pull/12183) ([Anton Popov](https://github.com/CurtizJ)). -#### Performance Improvement {#performance-improvement-2} +#### Performance improvement {#performance-improvement-2} * Enable compact parts by default for small parts. This will allow to process frequent inserts slightly more efficiently (4..100 times). [#11913](https://github.com/ClickHouse/ClickHouse/pull/11913) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Improve `quantileTDigest` performance. This fixes [#2668](https://github.com/ClickHouse/ClickHouse/issues/2668). [#15542](https://github.com/ClickHouse/ClickHouse/pull/15542) ([Kruglov Pavel](https://github.com/Avogar)). @@ -628,7 +628,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Only `mlock` code segment when starting clickhouse-server. In previous versions, all mapped regions were locked in memory, including debug info. Debug info is usually splitted to a separate file but if it isn't, it led to +2..3 GiB memory usage. [#14929](https://github.com/ClickHouse/ClickHouse/pull/14929) ([alexey-milovidov](https://github.com/alexey-milovidov)). * ClickHouse binary become smaller due to link time optimization. -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-4} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-4} * Now we use clang-11 for production ClickHouse build. [#15239](https://github.com/ClickHouse/ClickHouse/pull/15239) ([alesapin](https://github.com/alesapin)). * Now we use clang-11 to build ClickHouse in CI. [#14846](https://github.com/ClickHouse/ClickHouse/pull/14846) ([alesapin](https://github.com/alesapin)). @@ -664,11 +664,11 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor ### ClickHouse release v20.9.7.11-stable, 2020-12-07 {#clickhouse-release-v209711-stable-2020-12-07} -#### Performance Improvement {#performance-improvement-3} +#### Performance improvement {#performance-improvement-3} * Fix performance of reading from `Merge` tables over huge number of `MergeTree` tables. Fixes [#7748](https://github.com/ClickHouse/ClickHouse/issues/7748). [#16988](https://github.com/ClickHouse/ClickHouse/pull/16988) ([Anton Popov](https://github.com/CurtizJ)). -#### Bug Fix {#bug-fix-10} +#### Bug fix {#bug-fix-10} * Do not restore parts from WAL if `in_memory_parts_enable_wal` is disabled. [#17802](https://github.com/ClickHouse/ClickHouse/pull/17802) ([detailyang](https://github.com/detailyang)). * Fixed segfault when there is not enough space when inserting into `Distributed` table. [#17737](https://github.com/ClickHouse/ClickHouse/pull/17737) ([tavplubix](https://github.com/tavplubix)). @@ -689,7 +689,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * TODO. [#16866](https://github.com/ClickHouse/ClickHouse/pull/16866) ([tavplubix](https://github.com/tavplubix)). * Return number of affected rows for INSERT queries via MySQL protocol. Previously ClickHouse used to always return 0, it's fixed. Fixes [#16605](https://github.com/ClickHouse/ClickHouse/issues/16605). [#16715](https://github.com/ClickHouse/ClickHouse/pull/16715) ([Winter Zhang](https://github.com/zhang2014)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-5} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-5} * Update embedded timezone data to version 2020d (also update cctz to the latest master). [#17204](https://github.com/ClickHouse/ClickHouse/pull/17204) ([filimonov](https://github.com/filimonov)). @@ -700,7 +700,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Make it possible to connect to `clickhouse-server` secure endpoint which requires SNI. This is possible when `clickhouse-server` is hosted behind TLS proxy. [#16938](https://github.com/ClickHouse/ClickHouse/pull/16938) ([filimonov](https://github.com/filimonov)). * Conditional aggregate functions (for example: `avgIf`, `sumIf`, `maxIf`) should return `NULL` when miss rows and use nullable arguments. [#13964](https://github.com/ClickHouse/ClickHouse/pull/13964) ([Winter Zhang](https://github.com/zhang2014)). -#### Bug Fix {#bug-fix-11} +#### Bug fix {#bug-fix-11} * Fix bug when `ON CLUSTER` queries may hang forever for non-leader ReplicatedMergeTreeTables. [#17089](https://github.com/ClickHouse/ClickHouse/pull/17089) ([alesapin](https://github.com/alesapin)). * Reresolve the IP of the `format_avro_schema_registry_url` in case of errors. [#16985](https://github.com/ClickHouse/ClickHouse/pull/16985) ([filimonov](https://github.com/filimonov)). @@ -716,7 +716,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor ### ClickHouse release v20.9.5.5-stable, 2020-11-13 {#clickhouse-release-v20955-stable-2020-11-13} -#### Bug Fix {#bug-fix-12} +#### Bug fix {#bug-fix-12} * Fix rare silent crashes when query profiler is on and ClickHouse is installed on OS with glibc version that has (supposedly) broken asynchronous unwind tables for some functions. This fixes [#15301](https://github.com/ClickHouse/ClickHouse/issues/15301). This fixes [#13098](https://github.com/ClickHouse/ClickHouse/issues/13098). [#16846](https://github.com/ClickHouse/ClickHouse/pull/16846) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Now when parsing AVRO from input the LowCardinality is removed from type. Fixes [#16188](https://github.com/ClickHouse/ClickHouse/issues/16188). [#16521](https://github.com/ClickHouse/ClickHouse/pull/16521) ([Mike](https://github.com/myrrc)). @@ -728,7 +728,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor ### ClickHouse release v20.9.4.76-stable (2020-10-29) {#clickhouse-release-v209476-stable-2020-10-29} -#### Bug Fix {#bug-fix-13} +#### Bug fix {#bug-fix-13} * Fix double free in case of exception in function `dictGet`. It could have happened if dictionary was loaded with error. [#16429](https://github.com/ClickHouse/ClickHouse/pull/16429) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix group by with totals/rollup/cube modifers and min/max functions over group by keys. Fixes [#16393](https://github.com/ClickHouse/ClickHouse/issues/16393). [#16397](https://github.com/ClickHouse/ClickHouse/pull/16397) ([Anton Popov](https://github.com/CurtizJ)). @@ -761,7 +761,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor ### ClickHouse release v20.9.3.45-stable (2020-10-09) {#clickhouse-release-v209345-stable-2020-10-09} -#### Bug Fix {#bug-fix-14} +#### Bug fix {#bug-fix-14} * Fix error `Cannot find column` which may happen at insertion into `MATERIALIZED VIEW` in case if query for `MV` containes `ARRAY JOIN`. [#15717](https://github.com/ClickHouse/ClickHouse/pull/15717) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix race condition in AMQP-CPP. [#15667](https://github.com/ClickHouse/ClickHouse/pull/15667) ([alesapin](https://github.com/alesapin)). @@ -791,17 +791,17 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor ### ClickHouse release v20.9.2.20, 2020-09-22 {#clickhouse-release-v209220-2020-09-22} -#### Backward Incompatible Change {#backward-incompatible-change-3} +#### Backward incompatible change {#backward-incompatible-change-3} * When upgrading from versions older than 20.5, if rolling update is performed and cluster contains both versions 20.5 or greater and less than 20.5, if ClickHouse nodes with old versions are restarted and old version has been started up in presence of newer versions, it may lead to `Part ... intersects previous part` errors. To prevent this error, first install newer clickhouse-server packages on all cluster nodes and then do restarts (so, when clickhouse-server is restarted, it will start up with the new version). -#### New Feature {#new-feature-3} +#### New feature {#new-feature-3} * Added column transformers `EXCEPT`, `REPLACE`, `APPLY`, which can be applied to the list of selected columns (after `*` or `COLUMNS(...)`). For example, you can write `SELECT * EXCEPT(URL) REPLACE(number + 1 AS number)`. Another example: `select * apply(length) apply(max) from wide_string_table` to find out the maxium length of all string columns. [#14233](https://github.com/ClickHouse/ClickHouse/pull/14233) ([Amos Bird](https://github.com/amosbird)). * Added an aggregate function `rankCorr` which computes a rank correlation coefficient. [#11769](https://github.com/ClickHouse/ClickHouse/pull/11769) ([antikvist](https://github.com/antikvist)) [#14411](https://github.com/ClickHouse/ClickHouse/pull/14411) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Added table function `view` which turns a subquery into a table object. This helps passing queries around. For instance, it can be used in remote/cluster table functions. [#12567](https://github.com/ClickHouse/ClickHouse/pull/12567) ([Amos Bird](https://github.com/amosbird)). -#### Bug Fix {#bug-fix-15} +#### Bug fix {#bug-fix-15} * Fix bug when `ALTER UPDATE` mutation with Nullable column in assignment expression and constant value (like `UPDATE x = 42`) leads to incorrect value in column or segfault. Fixes [#13634](https://github.com/ClickHouse/ClickHouse/issues/13634), [#14045](https://github.com/ClickHouse/ClickHouse/issues/14045). [#14646](https://github.com/ClickHouse/ClickHouse/pull/14646) ([alesapin](https://github.com/alesapin)). * Fix wrong Decimal multiplication result caused wrong decimal scale of result column. [#14603](https://github.com/ClickHouse/ClickHouse/pull/14603) ([Artem Zuikov](https://github.com/4ertus2)). @@ -830,7 +830,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Fix GRANT ALL statement when executed on a non-global level. [#13987](https://github.com/ClickHouse/ClickHouse/pull/13987) ([Vitaly Baranov](https://github.com/vitlibar)). * Fix arrayJoin() capturing in lambda (exception with logical error message was thrown). [#13792](https://github.com/ClickHouse/ClickHouse/pull/13792) ([Azat Khuzhin](https://github.com/azat)). -#### Experimental Feature {#experimental-feature-2} +#### Experimental feature {#experimental-feature-2} * Added `db-generator` tool for random database generation by given SELECT queries. It may faciliate reproducing issues when there is only incomplete bug report from the user. [#14442](https://github.com/ClickHouse/ClickHouse/pull/14442) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)) [#10973](https://github.com/ClickHouse/ClickHouse/issues/10973) ([ZeDRoman](https://github.com/ZeDRoman)). @@ -850,13 +850,13 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Improvements in `RabbitMQ` engine: added connection and channels failure handling, proper commits, insert failures handling, better exchanges, queue durability and queue resume opportunity, new queue settings. Fixed tests. [#12761](https://github.com/ClickHouse/ClickHouse/pull/12761) ([Kseniia Sumarokova](https://github.com/kssenii)). * Support custom codecs in compact parts. [#12183](https://github.com/ClickHouse/ClickHouse/pull/12183) ([Anton Popov](https://github.com/CurtizJ)). -#### Performance Improvement {#performance-improvement-4} +#### Performance improvement {#performance-improvement-4} * Optimize queries with LIMIT/LIMIT BY/ORDER BY for distributed with GROUP BY sharding_key (under `optimize_skip_unused_shards` and `optimize_distributed_group_by_sharding_key`). [#10373](https://github.com/ClickHouse/ClickHouse/pull/10373) ([Azat Khuzhin](https://github.com/azat)). * Creating sets for multiple `JOIN` and `IN` in parallel. It may slightly improve performance for queries with several different `IN subquery` expressions. [#14412](https://github.com/ClickHouse/ClickHouse/pull/14412) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Improve Kafka engine performance by providing independent thread for each consumer. Separate thread pool for streaming engines (like Kafka). [#13939](https://github.com/ClickHouse/ClickHouse/pull/13939) ([fastio](https://github.com/fastio)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-6} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-6} * Lower binary size in debug build by removing debug info from `Functions`. This is needed only for one internal project in Yandex who is using very old linker. [#14549](https://github.com/ClickHouse/ClickHouse/pull/14549) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Prepare for build with clang 11. [#14455](https://github.com/ClickHouse/ClickHouse/pull/14455) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -867,14 +867,14 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor ### ClickHouse release v20.8.12.2-lts, 2021-01-16 {#clickhouse-release-v208122-lts-2021-01-16} -#### Bug Fix {#bug-fix-16} +#### Bug fix {#bug-fix-16} * Fix *If combinator with unary function and Nullable types. [#18806](https://github.com/ClickHouse/ClickHouse/pull/18806) ([Azat Khuzhin](https://github.com/azat)). * Restrict merges from wide to compact parts. In case of vertical merge it led to broken result part. [#18381](https://github.com/ClickHouse/ClickHouse/pull/18381) ([Anton Popov](https://github.com/CurtizJ)). ### ClickHouse release v20.8.11.17-lts, 2020-12-25 {#clickhouse-release-v2081117-lts-2020-12-25} -#### Bug Fix {#bug-fix-17} +#### Bug fix {#bug-fix-17} * Disable write with AIO during merges because it can lead to extremely rare data corruption of primary key columns during merge. [#18481](https://github.com/ClickHouse/ClickHouse/pull/18481) ([alesapin](https://github.com/alesapin)). * Fixed `value is too short` error when executing `toType(...)` functions (`toDate`, `toUInt32`, etc) with argument of type `Nullable(String)`. Now such functions return `NULL` on parsing errors instead of throwing exception. Fixes [#7673](https://github.com/ClickHouse/ClickHouse/issues/7673). [#18445](https://github.com/ClickHouse/ClickHouse/pull/18445) ([tavplubix](https://github.com/tavplubix)). @@ -882,7 +882,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor ### ClickHouse release v20.8.10.13-lts, 2020-12-24 {#clickhouse-release-v2081013-lts-2020-12-24} -#### Bug Fix {#bug-fix-18} +#### Bug fix {#bug-fix-18} * When server log rotation was configured using `logger.size` parameter with numeric value larger than 2^32, the logs were not rotated properly. [#17905](https://github.com/ClickHouse/ClickHouse/pull/17905) ([Alexander Kuzmenkov](https://github.com/akuzm)). * Fixed incorrect initialization of `max_compress_block_size` in MergeTreeWriterSettings with `min_compress_block_size`. [#17833](https://github.com/ClickHouse/ClickHouse/pull/17833) ([flynn](https://github.com/ucasFL)). @@ -906,7 +906,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor ### ClickHouse release v20.8.6.6-lts, 2020-11-13 {#clickhouse-release-v20866-lts-2020-11-13} -#### Bug Fix {#bug-fix-19} +#### Bug fix {#bug-fix-19} * Fix rare silent crashes when query profiler is on and ClickHouse is installed on OS with glibc version that has (supposedly) broken asynchronous unwind tables for some functions. This fixes [#15301](https://github.com/ClickHouse/ClickHouse/issues/15301). This fixes [#13098](https://github.com/ClickHouse/ClickHouse/issues/13098). [#16846](https://github.com/ClickHouse/ClickHouse/pull/16846) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Now when parsing AVRO from input the LowCardinality is removed from type. Fixes [#16188](https://github.com/ClickHouse/ClickHouse/issues/16188). [#16521](https://github.com/ClickHouse/ClickHouse/pull/16521) ([Mike](https://github.com/myrrc)). @@ -918,7 +918,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor ### ClickHouse release v20.8.5.45-lts, 2020-10-29 {#clickhouse-release-v208545-lts-2020-10-29} -#### Bug Fix {#bug-fix-20} +#### Bug fix {#bug-fix-20} * Fix double free in case of exception in function `dictGet`. It could have happened if dictionary was loaded with error. [#16429](https://github.com/ClickHouse/ClickHouse/pull/16429) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix group by with totals/rollup/cube modifers and min/max functions over group by keys. Fixes [#16393](https://github.com/ClickHouse/ClickHouse/issues/16393). [#16397](https://github.com/ClickHouse/ClickHouse/pull/16397) ([Anton Popov](https://github.com/CurtizJ)). @@ -951,7 +951,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor ### ClickHouse release v20.8.4.11-lts, 2020-10-09 {#clickhouse-release-v208411-lts-2020-10-09} -#### Bug Fix {#bug-fix-21} +#### Bug fix {#bug-fix-21} * Fix the order of destruction for resources in `ReadFromStorage` step of query plan. It might cause crashes in rare cases. Possibly connected with [#15610](https://github.com/ClickHouse/ClickHouse/issues/15610). [#15645](https://github.com/ClickHouse/ClickHouse/pull/15645) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fixed `Element ... is not a constant expression` error when using `JSON*` function result in `VALUES`, `LIMIT` or right side of `IN` operator. [#15589](https://github.com/ClickHouse/ClickHouse/pull/15589) ([tavplubix](https://github.com/tavplubix)). @@ -985,7 +985,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor ### ClickHouse release v20.8.3.18-stable, 2020-09-18 {#clickhouse-release-v208318-stable-2020-09-18} -#### Bug Fix {#bug-fix-22} +#### Bug fix {#bug-fix-22} * Fix the issue when some invocations of `extractAllGroups` function may trigger "Memory limit exceeded" error. This fixes [#13383](https://github.com/ClickHouse/ClickHouse/issues/13383). [#14889](https://github.com/ClickHouse/ClickHouse/pull/14889) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix SIGSEGV for an attempt to INSERT into StorageFile(fd). [#14887](https://github.com/ClickHouse/ClickHouse/pull/14887) ([Azat Khuzhin](https://github.com/azat)). @@ -1006,14 +1006,14 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor ### ClickHouse release v20.8.2.3-stable, 2020-09-08 {#clickhouse-release-v20823-stable-2020-09-08} -#### Backward Incompatible Change {#backward-incompatible-change-4} +#### Backward incompatible change {#backward-incompatible-change-4} * Now `OPTIMIZE FINAL` query does not recalculate TTL for parts that were added before TTL was created. Use `ALTER TABLE ... MATERIALIZE TTL` once to calculate them, after that `OPTIMIZE FINAL` will evaluate TTL's properly. This behavior never worked for replicated tables. [#14220](https://github.com/ClickHouse/ClickHouse/pull/14220) ([alesapin](https://github.com/alesapin)). * Extend `parallel_distributed_insert_select` setting, adding an option to run `INSERT` into local table. The setting changes type from `Bool` to `UInt64`, so the values `false` and `true` are no longer supported. If you have these values in server configuration, the server will not start. Please replace them with `0` and `1`, respectively. [#14060](https://github.com/ClickHouse/ClickHouse/pull/14060) ([Azat Khuzhin](https://github.com/azat)). * Remove support for the `ODBCDriver` input/output format. This was a deprecated format once used for communication with the ClickHouse ODBC driver, now long superseded by the `ODBCDriver2` format. Resolves [#13629](https://github.com/ClickHouse/ClickHouse/issues/13629). [#13847](https://github.com/ClickHouse/ClickHouse/pull/13847) ([hexiaoting](https://github.com/hexiaoting)). * When upgrading from versions older than 20.5, if rolling update is performed and cluster contains both versions 20.5 or greater and less than 20.5, if ClickHouse nodes with old versions are restarted and old version has been started up in presence of newer versions, it may lead to `Part ... intersects previous part` errors. To prevent this error, first install newer clickhouse-server packages on all cluster nodes and then do restarts (so, when clickhouse-server is restarted, it will start up with the new version). -#### New Feature {#new-feature-4} +#### New feature {#new-feature-4} * Add the ability to specify `Default` compression codec for columns that correspond to settings specified in `config.xml`. Implements: [#9074](https://github.com/ClickHouse/ClickHouse/issues/9074). [#14049](https://github.com/ClickHouse/ClickHouse/pull/14049) ([alesapin](https://github.com/alesapin)). * Support Kerberos authentication in Kafka, using `krb5` and `cyrus-sasl` libraries. [#12771](https://github.com/ClickHouse/ClickHouse/pull/12771) ([Ilya Golshtein](https://github.com/ilejn)). @@ -1027,7 +1027,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Add `ALTER SAMPLE BY` statement that allows to change table sample clause. [#13280](https://github.com/ClickHouse/ClickHouse/pull/13280) ([Amos Bird](https://github.com/amosbird)). * Function `position` now supports optional `start_pos` argument. [#13237](https://github.com/ClickHouse/ClickHouse/pull/13237) ([vdimir](https://github.com/vdimir)). -#### Bug Fix {#bug-fix-23} +#### Bug fix {#bug-fix-23} * Fix visible data clobbering by progress bar in client in interactive mode. This fixes [#12562](https://github.com/ClickHouse/ClickHouse/issues/12562) and [#13369](https://github.com/ClickHouse/ClickHouse/issues/13369) and [#13584](https://github.com/ClickHouse/ClickHouse/issues/13584) and fixes [#12964](https://github.com/ClickHouse/ClickHouse/issues/12964). [#13691](https://github.com/ClickHouse/ClickHouse/pull/13691) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed incorrect sorting order if `LowCardinality` column when sorting by multiple columns. This fixes [#13958](https://github.com/ClickHouse/ClickHouse/issues/13958). [#14223](https://github.com/ClickHouse/ClickHouse/pull/14223) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). @@ -1103,7 +1103,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Fix readline so it dumps history to file now. [#13600](https://github.com/ClickHouse/ClickHouse/pull/13600) ([Amos Bird](https://github.com/amosbird)). * Create `system` database with `Atomic` engine by default (a preparation to enable `Atomic` database engine by default everywhere). [#13680](https://github.com/ClickHouse/ClickHouse/pull/13680) ([tavplubix](https://github.com/tavplubix)). -#### Performance Improvement {#performance-improvement-5} +#### Performance improvement {#performance-improvement-5} * Slightly optimize very short queries with `LowCardinality`. [#14129](https://github.com/ClickHouse/ClickHouse/pull/14129) ([Anton Popov](https://github.com/CurtizJ)). * Enable parallel INSERTs for table engines `Null`, `Memory`, `Distributed` and `Buffer` when the setting `max_insert_threads` is set. [#14120](https://github.com/ClickHouse/ClickHouse/pull/14120) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -1112,12 +1112,12 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Optimize `has()`, `indexOf()` and `countEqual()` functions for `Array(LowCardinality(T))` and constant right arguments. [#12550](https://github.com/ClickHouse/ClickHouse/pull/12550) ([myrrc](https://github.com/myrrc)). * When performing trivial `INSERT SELECT` queries, automatically set `max_threads` to 1 or `max_insert_threads`, and set `max_block_size` to `min_insert_block_size_rows`. Related to [#5907](https://github.com/ClickHouse/ClickHouse/issues/5907). [#12195](https://github.com/ClickHouse/ClickHouse/pull/12195) ([flynn](https://github.com/ucasFL)). -#### Experimental Feature {#experimental-feature-3} +#### Experimental feature {#experimental-feature-3} * ClickHouse can work as MySQL replica - it is implemented by `MaterializeMySQL` database engine. Implements [#4006](https://github.com/ClickHouse/ClickHouse/issues/4006). [#10851](https://github.com/ClickHouse/ClickHouse/pull/10851) ([Winter Zhang](https://github.com/zhang2014)). * Add types `Int128`, `Int256`, `UInt256` and related functions for them. Extend Decimals with Decimal256 (precision up to 76 digits). New types are under the setting `allow_experimental_bigint_types`. It is working extremely slow and bad. The implementation is incomplete. Please don't use this feature. [#13097](https://github.com/ClickHouse/ClickHouse/pull/13097) ([Artem Zuikov](https://github.com/4ertus2)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-7} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-7} * Added `clickhouse install` script, that is useful if you only have a single binary. [#13528](https://github.com/ClickHouse/ClickHouse/pull/13528) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Allow to run `clickhouse` binary without configuration. [#13515](https://github.com/ClickHouse/ClickHouse/pull/13515) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -1148,7 +1148,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor ### ClickHouse release v20.7.2.30-stable, 2020-08-31 {#clickhouse-release-v207230-stable-2020-08-31} -#### Backward Incompatible Change {#backward-incompatible-change-5} +#### Backward incompatible change {#backward-incompatible-change-5} * Function `modulo` (operator `%`) with at least one floating point number as argument will calculate remainder of division directly on floating point numbers without converting both arguments to integers. It makes behaviour compatible with most of DBMS. This also applicable for Date and DateTime data types. Added alias `mod`. This closes [#7323](https://github.com/ClickHouse/ClickHouse/issues/7323). [#12585](https://github.com/ClickHouse/ClickHouse/pull/12585) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Deprecate special printing of zero Date/DateTime values as `0000-00-00` and `0000-00-00 00:00:00`. [#12442](https://github.com/ClickHouse/ClickHouse/pull/12442) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -1157,7 +1157,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Protect from the cases when user may set `background_pool_size` to value lower than `number_of_free_entries_in_pool_to_execute_mutation` or `number_of_free_entries_in_pool_to_lower_max_size_of_merge`. In these cases ALTERs won't work or the maximum size of merge will be too limited. It will throw exception explaining what to do. This closes [#10897](https://github.com/ClickHouse/ClickHouse/issues/10897). [#12728](https://github.com/ClickHouse/ClickHouse/pull/12728) ([alexey-milovidov](https://github.com/alexey-milovidov)). * When upgrading from versions older than 20.5, if rolling update is performed and cluster contains both versions 20.5 or greater and less than 20.5, if ClickHouse nodes with old versions are restarted and old version has been started up in presence of newer versions, it may lead to `Part ... intersects previous part` errors. To prevent this error, first install newer clickhouse-server packages on all cluster nodes and then do restarts (so, when clickhouse-server is restarted, it will start up with the new version). -#### New Feature {#new-feature-5} +#### New feature {#new-feature-5} * Polygon dictionary type that provides efficient "reverse geocoding" lookups - to find the region by coordinates in a dictionary of many polygons (world map). It is using carefully optimized algorithm with recursive grids to maintain low CPU and memory usage. [#9278](https://github.com/ClickHouse/ClickHouse/pull/9278) ([achulkov2](https://github.com/achulkov2)). * Added support of LDAP authentication for preconfigured users ("Simple Bind" method). [#11234](https://github.com/ClickHouse/ClickHouse/pull/11234) ([Denis Glazachev](https://github.com/traceon)). @@ -1173,7 +1173,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Integration with [COS](https://intl.cloud.tencent.com/product/cos). [#12386](https://github.com/ClickHouse/ClickHouse/pull/12386) ([fastio](https://github.com/fastio)). * Add `mapAdd` and `mapSubtract` functions for adding/subtracting key-mapped values. [#11735](https://github.com/ClickHouse/ClickHouse/pull/11735) ([Ildus Kurbangaliev](https://github.com/ildus)). -#### Bug Fix {#bug-fix-24} +#### Bug fix {#bug-fix-24} * Fix premature `ON CLUSTER` timeouts for queries that must be executed on a single replica. Fixes [#6704](https://github.com/ClickHouse/ClickHouse/issues/6704), [#7228](https://github.com/ClickHouse/ClickHouse/issues/7228), [#13361](https://github.com/ClickHouse/ClickHouse/issues/13361), [#11884](https://github.com/ClickHouse/ClickHouse/issues/11884). [#13450](https://github.com/ClickHouse/ClickHouse/pull/13450) ([alesapin](https://github.com/alesapin)). * Fix crash in mark inclusion search introduced in [#12277](https://github.com/ClickHouse/ClickHouse/pull/12277). [#14225](https://github.com/ClickHouse/ClickHouse/pull/14225) ([Amos Bird](https://github.com/amosbird)). @@ -1285,7 +1285,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * If MergeTree table does not contain ORDER BY or PARTITION BY, it was possible to request ALTER to CLEAR all the columns and ALTER will stuck. Fixed [#7941](https://github.com/ClickHouse/ClickHouse/issues/7941). [#12382](https://github.com/ClickHouse/ClickHouse/pull/12382) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Avoid re-loading completion from the history file after each query (to avoid history overlaps with other client sessions). [#13086](https://github.com/ClickHouse/ClickHouse/pull/13086) ([Azat Khuzhin](https://github.com/azat)). -#### Performance Improvement {#performance-improvement-6} +#### Performance improvement {#performance-improvement-6} * Lower memory usage for some operations up to 2 times. [#12424](https://github.com/ClickHouse/ClickHouse/pull/12424) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Optimize PK lookup for queries that match exact PK range. [#12277](https://github.com/ClickHouse/ClickHouse/pull/12277) ([Ivan Babrou](https://github.com/bobrik)). @@ -1300,7 +1300,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Improve performance of reading from compact parts. Compact parts is an experimental feature. [#12492](https://github.com/ClickHouse/ClickHouse/pull/12492) ([Anton Popov](https://github.com/CurtizJ)). * Attempt to implement streaming optimization in `DiskS3`. DiskS3 is an experimental feature. [#12434](https://github.com/ClickHouse/ClickHouse/pull/12434) ([Vladimir Chebotarev](https://github.com/excitoon)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-8} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-8} * Use `shellcheck` for sh tests linting. [#13200](https://github.com/ClickHouse/ClickHouse/pull/13200) [#13207](https://github.com/ClickHouse/ClickHouse/pull/13207) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Add script which set labels for pull requests in GitHub hook. [#13183](https://github.com/ClickHouse/ClickHouse/pull/13183) ([alesapin](https://github.com/alesapin)). @@ -1341,11 +1341,11 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor ### ClickHouse release v20.6.3.28-stable {#clickhouse-release-v206328-stable} -#### Backward Incompatible Change {#backward-incompatible-change-6} +#### Backward incompatible change {#backward-incompatible-change-6} * When upgrading from versions older than 20.5, if rolling update is performed and cluster contains both versions 20.5 or greater and less than 20.5, if ClickHouse nodes with old versions are restarted and old version has been started up in presence of newer versions, it may lead to `Part ... intersects previous part` errors. To prevent this error, first install newer clickhouse-server packages on all cluster nodes and then do restarts (so, when clickhouse-server is restarted, it will start up with the new version). -#### New Feature {#new-feature-6} +#### New feature {#new-feature-6} * Added an initial implementation of `EXPLAIN` query. Syntax: `EXPLAIN SELECT ...`. This fixes [#1118](https://github.com/ClickHouse/ClickHouse/issues/1118). [#11873](https://github.com/ClickHouse/ClickHouse/pull/11873) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Added storage `RabbitMQ`. [#11069](https://github.com/ClickHouse/ClickHouse/pull/11069) ([Kseniia Sumarokova](https://github.com/kssenii)). @@ -1356,7 +1356,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Added function `parseDateTimeBestEffortUS`. [#12028](https://github.com/ClickHouse/ClickHouse/pull/12028) ([flynn](https://github.com/ucasFL)). * Support format `ORC` for output (was supported only for input). [#11662](https://github.com/ClickHouse/ClickHouse/pull/11662) ([Kruglov Pavel](https://github.com/Avogar)). -#### Bug Fix {#bug-fix-25} +#### Bug fix {#bug-fix-25} * Fixed `aggregate function any(x) is found inside another aggregate function in query` error with `SET optimize_move_functions_out_of_any = 1` and aliases inside `any()`. [#13419](https://github.com/ClickHouse/ClickHouse/pull/13419) ([Artem Zuikov](https://github.com/4ertus2)). * Fixed `PrettyCompactMonoBlock` for clickhouse-local. Fixed extremes/totals with `PrettyCompactMonoBlock`. This fixes [#7746](https://github.com/ClickHouse/ClickHouse/issues/7746). [#13394](https://github.com/ClickHouse/ClickHouse/pull/13394) ([Azat Khuzhin](https://github.com/azat)). @@ -1449,17 +1449,17 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Added replica priority for load_balancing (for manual prioritization of the load balancing). [#11995](https://github.com/ClickHouse/ClickHouse/pull/11995) ([Azat Khuzhin](https://github.com/azat)). * Switched paths in S3 metadata to relative which allows to handle S3 blobs more easily. [#11892](https://github.com/ClickHouse/ClickHouse/pull/11892) ([Vladimir Chebotarev](https://github.com/excitoon)). -#### Performance Improvement {#performance-improvement-7} +#### Performance improvement {#performance-improvement-7} * Improved performace of 'ORDER BY' and 'GROUP BY' by prefix of sorting key (enabled with `optimize_aggregation_in_order` setting, disabled by default). [#11696](https://github.com/ClickHouse/ClickHouse/pull/11696) ([Anton Popov](https://github.com/CurtizJ)). * Removed injective functions inside `uniq*()` if `set optimize_injective_functions_inside_uniq=1`. [#12337](https://github.com/ClickHouse/ClickHouse/pull/12337) ([Ruslan Kamalov](https://github.com/kamalov-ruslan)). * Index not used for IN operator with literals, performance regression introduced around v19.3. This fixes [#10574](https://github.com/ClickHouse/ClickHouse/issues/10574). [#12062](https://github.com/ClickHouse/ClickHouse/pull/12062) ([nvartolomei](https://github.com/nvartolomei)). * Implemented single part uploads for DiskS3 (experimental feature). [#12026](https://github.com/ClickHouse/ClickHouse/pull/12026) ([Vladimir Chebotarev](https://github.com/excitoon)). -#### Experimental Feature {#experimental-feature-4} +#### Experimental feature {#experimental-feature-4} * Added new in-memory format of parts in `MergeTree`-family tables, which stores data in memory. Parts are written on disk at first merge. Part will be created in in-memory format if its size in rows or bytes is below thresholds `min_rows_for_compact_part` and `min_bytes_for_compact_part`. Also optional support of Write-Ahead-Log is available, which is enabled by default and is controlled by setting `in_memory_parts_enable_wal`. [#10697](https://github.com/ClickHouse/ClickHouse/pull/10697) ([Anton Popov](https://github.com/CurtizJ)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-9} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-9} * Implement AST-based query fuzzing mode for clickhouse-client. See [this label](https://github.com/ClickHouse/ClickHouse/issues?q=label%3Afuzz+is%3Aissue) for the list of issues we recently found by fuzzing. Most of them were found by this tool, and a couple by SQLancer and `00746_sql_fuzzy.pl`. [#12111](https://github.com/ClickHouse/ClickHouse/pull/12111) ([Alexander Kuzmenkov](https://github.com/akuzm)). * Add new type of tests based on Testflows framework. [#12090](https://github.com/ClickHouse/ClickHouse/pull/12090) ([vzakaznikov](https://github.com/vzakaznikov)). @@ -1481,7 +1481,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor ### ClickHouse release v20.5.4.40-stable 2020-08-10 {#clickhouse-release-v205440-stable-2020-08-10} -#### Bug Fix {#bug-fix-26} +#### Bug fix {#bug-fix-26} * Fixed wrong index analysis with functions. It could lead to pruning wrong parts, while reading from `MergeTree` tables. Fixes [#13060](https://github.com/ClickHouse/ClickHouse/issues/13060). Fixes [#12406](https://github.com/ClickHouse/ClickHouse/issues/12406). [#13081](https://github.com/ClickHouse/ClickHouse/pull/13081) ([Anton Popov](https://github.com/CurtizJ)). * Fixed unnecessary limiting for the number of threads for selects from local replica. [#12840](https://github.com/ClickHouse/ClickHouse/pull/12840) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -1513,17 +1513,17 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Fixed crash in JOIN with LowCardinality type with `join_algorithm=partial_merge`. [#12035](https://github.com/ClickHouse/ClickHouse/pull/12035) ([Artem Zuikov](https://github.com/4ertus2)). * Fixed wrong result for `if()` with NULLs in condition. [#11807](https://github.com/ClickHouse/ClickHouse/pull/11807) ([Artem Zuikov](https://github.com/4ertus2)). -#### Performance Improvement {#performance-improvement-8} +#### Performance improvement {#performance-improvement-8} * Index not used for IN operator with literals, performance regression introduced around v19.3. This fixes [#10574](https://github.com/ClickHouse/ClickHouse/issues/10574). [#12062](https://github.com/ClickHouse/ClickHouse/pull/12062) ([nvartolomei](https://github.com/nvartolomei)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-10} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-10} * Install `ca-certificates` before the first `apt-get update` in Dockerfile. [#12095](https://github.com/ClickHouse/ClickHouse/pull/12095) ([Ivan Blinkov](https://github.com/blinkov)). ### ClickHouse release v20.5.2.7-stable 2020-07-02 {#clickhouse-release-v20527-stable-2020-07-02} -#### Backward Incompatible Change {#backward-incompatible-change-7} +#### Backward incompatible change {#backward-incompatible-change-7} * Return non-Nullable result from COUNT(DISTINCT), and `uniq` aggregate functions family. If all passed values are NULL, return zero instead. This improves SQL compatibility. [#11661](https://github.com/ClickHouse/ClickHouse/pull/11661) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Added a check for the case when user-level setting is specified in a wrong place. User-level settings should be specified in `users.xml` inside `` section for specific user profile (or in `` for default settings). The server won't start with exception message in log. This fixes [#9051](https://github.com/ClickHouse/ClickHouse/issues/9051). If you want to skip the check, you can either move settings to the appropriate place or add `1` to config.xml. [#11449](https://github.com/ClickHouse/ClickHouse/pull/11449) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -1534,7 +1534,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Several Kafka setting changes their defaults. See [#11388](https://github.com/ClickHouse/ClickHouse/pull/11388). * When upgrading from versions older than 20.5, if rolling update is performed and cluster contains both versions 20.5 or greater and less than 20.5, if ClickHouse nodes with old versions are restarted and old version has been started up in presence of newer versions, it may lead to `Part ... intersects previous part` errors. To prevent this error, first install newer clickhouse-server packages on all cluster nodes and then do restarts (so, when clickhouse-server is restarted, it will start up with the new version). -#### New Feature {#new-feature-7} +#### New feature {#new-feature-7} * `TTL DELETE WHERE` and `TTL GROUP BY` for automatic data coarsening and rollup in tables. [#10537](https://github.com/ClickHouse/ClickHouse/pull/10537) ([expl0si0nn](https://github.com/expl0si0nn)). * Implementation of PostgreSQL wire protocol. [#10242](https://github.com/ClickHouse/ClickHouse/pull/10242) ([Movses](https://github.com/MovElb)). @@ -1585,7 +1585,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * `SimpleAggregateFunction` now also supports `sumMap`. [#10000](https://github.com/ClickHouse/ClickHouse/pull/10000) ([Ildus Kurbangaliev](https://github.com/ildus)). * Support `ALTER RENAME COLUMN` for the distributed table engine. Continuation of [#10727](https://github.com/ClickHouse/ClickHouse/issues/10727). Fixes [#10747](https://github.com/ClickHouse/ClickHouse/issues/10747). [#10887](https://github.com/ClickHouse/ClickHouse/pull/10887) ([alesapin](https://github.com/alesapin)). -#### Bug Fix {#bug-fix-27} +#### Bug fix {#bug-fix-27} * Fix UBSan report in Decimal parse. This fixes [#7540](https://github.com/ClickHouse/ClickHouse/issues/7540). [#10512](https://github.com/ClickHouse/ClickHouse/pull/10512) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix potential floating point exception when parsing DateTime64. This fixes [#11374](https://github.com/ClickHouse/ClickHouse/issues/11374). [#11875](https://github.com/ClickHouse/ClickHouse/pull/11875) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -1775,7 +1775,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Adding support for `INSERT INTO [db.]table WATCH` query. [#10498](https://github.com/ClickHouse/ClickHouse/pull/10498) ([vzakaznikov](https://github.com/vzakaznikov)). * Allow to pass quota_key in clickhouse-client. This closes [#10227](https://github.com/ClickHouse/ClickHouse/issues/10227). [#10270](https://github.com/ClickHouse/ClickHouse/pull/10270) ([alexey-milovidov](https://github.com/alexey-milovidov)). -#### Performance Improvement {#performance-improvement-9} +#### Performance improvement {#performance-improvement-9} * Allow multiple replicas to assign merges, mutations, partition drop, move and replace concurrently. This closes [#10367](https://github.com/ClickHouse/ClickHouse/issues/10367). [#11639](https://github.com/ClickHouse/ClickHouse/pull/11639) ([alexey-milovidov](https://github.com/alexey-milovidov)) [#11795](https://github.com/ClickHouse/ClickHouse/pull/11795) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Optimization of GROUP BY with respect to table sorting key, enabled with `optimize_aggregation_in_order` setting. [#9113](https://github.com/ClickHouse/ClickHouse/pull/9113) ([dimarub2000](https://github.com/dimarub2000)). @@ -1799,13 +1799,13 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Optimize memory usage when reading a response from an S3 HTTP client. [#11561](https://github.com/ClickHouse/ClickHouse/pull/11561) ([Pavel Kovalenko](https://github.com/Jokser)). * Adjust the default Kafka settings for better performance. [#11388](https://github.com/ClickHouse/ClickHouse/pull/11388) ([filimonov](https://github.com/filimonov)). -#### Experimental Feature {#experimental-feature-5} +#### Experimental feature {#experimental-feature-5} * Add data type `Point` (Tuple(Float64, Float64)) and `Polygon` (Array(Array(Tuple(Float64, Float64))). [#10678](https://github.com/ClickHouse/ClickHouse/pull/10678) ([Alexey Ilyukhov](https://github.com/livace)). * Add's a `hasSubstr` function that allows for look for subsequences in arrays. Note: this function is likely to be renamed without further notice. [#11071](https://github.com/ClickHouse/ClickHouse/pull/11071) ([Ryad Zenine](https://github.com/r-zenine)). * Added OpenCL support and bitonic sort algorithm, which can be used for sorting integer types of data in single column. Needs to be build with flag `-DENABLE_OPENCL=1`. For using bitonic sort algorithm instead of others you need to set `bitonic_sort` for Setting's option `special_sort` and make sure that OpenCL is available. This feature does not improve performance or anything else, it is only provided as an example and for demonstration purposes. It is likely to be removed in near future if there will be no further development in this direction. [#10232](https://github.com/ClickHouse/ClickHouse/pull/10232) ([Ri](https://github.com/margaritiko)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-11} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-11} * Enable clang-tidy for programs and utils. [#10991](https://github.com/ClickHouse/ClickHouse/pull/10991) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Remove dependency on `tzdata`: do not fail if `/usr/share/zoneinfo` directory does not exist. Note that all timezones work in ClickHouse even without tzdata installed in system. [#11827](https://github.com/ClickHouse/ClickHouse/pull/11827) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -1878,7 +1878,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor ### ClickHouse release v20.4.8.99-stable 2020-08-10 {#clickhouse-release-v204899-stable-2020-08-10} -#### Bug Fix {#bug-fix-28} +#### Bug fix {#bug-fix-28} * Fixed error in `parseDateTimeBestEffort` function when unix timestamp was passed as an argument. This fixes [#13362](https://github.com/ClickHouse/ClickHouse/issues/13362). [#13441](https://github.com/ClickHouse/ClickHouse/pull/13441) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed potentially low performance and slightly incorrect result for `uniqExact`, `topK`, `sumDistinct` and similar aggregate functions called on Float types with NaN values. It also triggered assert in debug build. This fixes [#12491](https://github.com/ClickHouse/ClickHouse/issues/12491). [#13254](https://github.com/ClickHouse/ClickHouse/pull/13254) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -1941,17 +1941,17 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Fixed memory accounting via HTTP interface (can be significant with `wait_end_of_query=1`). [#11840](https://github.com/ClickHouse/ClickHouse/pull/11840) ([Azat Khuzhin](https://github.com/azat)). * Parse metadata stored in zookeeper before checking for equality. [#11739](https://github.com/ClickHouse/ClickHouse/pull/11739) ([Azat Khuzhin](https://github.com/azat)). -#### Performance Improvement {#performance-improvement-10} +#### Performance improvement {#performance-improvement-10} * Index not used for IN operator with literals, performance regression introduced around v19.3. This fixes [#10574](https://github.com/ClickHouse/ClickHouse/issues/10574). [#12062](https://github.com/ClickHouse/ClickHouse/pull/12062) ([nvartolomei](https://github.com/nvartolomei)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-12} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-12} * Install `ca-certificates` before the first `apt-get update` in Dockerfile. [#12095](https://github.com/ClickHouse/ClickHouse/pull/12095) ([Ivan Blinkov](https://github.com/blinkov)). ### ClickHouse release v20.4.6.53-stable 2020-06-25 {#clickhouse-release-v204653-stable-2020-06-25} -#### Bug Fix {#bug-fix-29} +#### Bug fix {#bug-fix-29} * Fix rare crash caused by using `Nullable` column in prewhere condition. Continuation of [#11608](https://github.com/ClickHouse/ClickHouse/issues/11608). [#11869](https://github.com/ClickHouse/ClickHouse/pull/11869) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Don't allow arrayJoin inside higher order functions. It was leading to broken protocol synchronization. This closes [#3933](https://github.com/ClickHouse/ClickHouse/issues/3933). [#11846](https://github.com/ClickHouse/ClickHouse/pull/11846) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -1983,14 +1983,14 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Get rid of old libunwind patches. https://github.com/ClickHouse-Extras/libunwind/commit/500aa227911bd185a94bfc071d68f4d3b03cb3b1#r39048012 This allows to disable `-fno-omit-frame-pointer` in `clang` builds that improves performance at least by 1% in average. [#10761](https://github.com/ClickHouse/ClickHouse/pull/10761) ([Amos Bird](https://github.com/amosbird)). * Fix usage of primary key wrapped into a function with 'FINAL' modifier and 'ORDER BY' optimization. [#10715](https://github.com/ClickHouse/ClickHouse/pull/10715) ([Anton Popov](https://github.com/CurtizJ)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-13} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-13} * Fix several non significant errors in unit tests. [#11262](https://github.com/ClickHouse/ClickHouse/pull/11262) ([alesapin](https://github.com/alesapin)). * Fix (false) MSan report in MergeTreeIndexFullText. The issue first appeared in [#9968](https://github.com/ClickHouse/ClickHouse/issues/9968). [#10801](https://github.com/ClickHouse/ClickHouse/pull/10801) ([alexey-milovidov](https://github.com/alexey-milovidov)). ### ClickHouse release v20.4.5.36-stable 2020-06-10 {#clickhouse-release-v204536-stable-2020-06-10} -#### Bug Fix {#bug-fix-30} +#### Bug fix {#bug-fix-30} * Fix the error `Data compressed with different methods` that can happen if `min_bytes_to_use_direct_io` is enabled and PREWHERE is active and using SAMPLE or high number of threads. This fixes [#11539](https://github.com/ClickHouse/ClickHouse/issues/11539). [#11540](https://github.com/ClickHouse/ClickHouse/pull/11540) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix return compressed size for codecs. [#11448](https://github.com/ClickHouse/ClickHouse/pull/11448) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -2024,7 +2024,7 @@ keywords: ['ClickHouse 2020', 'changelog 2020', 'release notes', 'version histor * Fix excessive reserving of threads for simple queries (optimization for reducing the number of threads, which was partly broken after changes in pipeline). [#11114](https://github.com/ClickHouse/ClickHouse/pull/11114) ([Azat Khuzhin](https://github.com/azat)). * Fix predicates optimization for distributed queries (`enable_optimize_predicate_expression=1`) for queries with `HAVING` section (i.e. when filtering on the server initiator is required), by preserving the order of expressions (and this is enough to fix), and also force aggregator use column names over indexes. Fixes: [#10613](https://github.com/ClickHouse/ClickHouse/issues/10613), [#11413](https://github.com/ClickHouse/ClickHouse/issues/11413). [#10621](https://github.com/ClickHouse/ClickHouse/pull/10621) ([Azat Khuzhin](https://github.com/azat)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-14} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-14} * Fix several flaky integration tests. [#11355](https://github.com/ClickHouse/ClickHouse/pull/11355) ([alesapin](https://github.com/alesapin)). @@ -2034,7 +2034,7 @@ No changes compared to v20.4.3.16-stable. ### ClickHouse release v20.4.3.16-stable 2020-05-23 {#clickhouse-release-v204316-stable-2020-05-23} -#### Bug Fix {#bug-fix-31} +#### Bug fix {#bug-fix-31} * Removed logging from mutation finalization task if nothing was finalized. [#11109](https://github.com/ClickHouse/ClickHouse/pull/11109) ([alesapin](https://github.com/alesapin)). * Fixed memory leak in registerDiskS3. [#11074](https://github.com/ClickHouse/ClickHouse/pull/11074) ([Pavel Kovalenko](https://github.com/Jokser)). @@ -2077,13 +2077,13 @@ No changes compared to v20.4.3.16-stable. ### ClickHouse release v20.4.2.9, 2020-05-12 {#clickhouse-release-v20429-2020-05-12} -#### Backward Incompatible Change {#backward-incompatible-change-8} +#### Backward incompatible change {#backward-incompatible-change-8} * System tables (e.g. system.query_log, system.trace_log, system.metric_log) are using compact data part format for parts smaller than 10 MiB in size. Compact data part format is supported since version 20.3. If you are going to downgrade to version less than 20.3, you should manually delete table data for system logs in `/var/lib/clickhouse/data/system/`. * When string comparison involves FixedString and compared arguments are of different sizes, do comparison as if smaller string is padded to the length of the larger. This is intented for SQL compatibility if we imagine that FixedString data type corresponds to SQL CHAR. This closes [#9272](https://github.com/ClickHouse/ClickHouse/issues/9272). [#10363](https://github.com/ClickHouse/ClickHouse/pull/10363) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Make SHOW CREATE TABLE multiline. Now it is more readable and more like MySQL. [#10049](https://github.com/ClickHouse/ClickHouse/pull/10049) ([Azat Khuzhin](https://github.com/azat)) * Added a setting `validate_polygons` that is used in `pointInPolygon` function and enabled by default. [#9857](https://github.com/ClickHouse/ClickHouse/pull/9857) ([alexey-milovidov](https://github.com/alexey-milovidov)) -#### New Feature {#new-feature-8} +#### New feature {#new-feature-8} * Add support for secured connection from ClickHouse to Zookeeper [#10184](https://github.com/ClickHouse/ClickHouse/pull/10184) ([Konstantin Lebedev](https://github.com/xzkostyan)) * Support custom HTTP handlers. See [#5436](https://github.com/ClickHouse/ClickHouse/issues/5436) for description. [#7572](https://github.com/ClickHouse/ClickHouse/pull/7572) ([Winter Zhang](https://github.com/zhang2014)) * Add MessagePack Input/Output format. [#9889](https://github.com/ClickHouse/ClickHouse/pull/9889) ([Kruglov Pavel](https://github.com/Avogar)) @@ -2111,11 +2111,11 @@ No changes compared to v20.4.3.16-stable. * Add 'strict_order' option to windowFunnel() [#9773](https://github.com/ClickHouse/ClickHouse/pull/9773) ([achimbab](https://github.com/achimbab)) * Support `DATE` and `TIMESTAMP` SQL operators, e.g. `SELECT date '2001-01-01'` [#9691](https://github.com/ClickHouse/ClickHouse/pull/9691) ([Artem Zuikov](https://github.com/4ertus2)) -#### Experimental Feature {#experimental-feature-6} +#### Experimental feature {#experimental-feature-6} * Added experimental database engine Atomic. It supports non-blocking `DROP` and `RENAME TABLE` queries and atomic `EXCHANGE TABLES t1 AND t2` query [#7512](https://github.com/ClickHouse/ClickHouse/pull/7512) ([tavplubix](https://github.com/tavplubix)) * Initial support for ReplicatedMergeTree over S3 (it works in suboptimal way) [#10126](https://github.com/ClickHouse/ClickHouse/pull/10126) ([Pavel Kovalenko](https://github.com/Jokser)) -#### Bug Fix {#bug-fix-32} +#### Bug fix {#bug-fix-32} * Fixed incorrect scalar results inside inner query of `MATERIALIZED VIEW` in case if this query contained dependent table [#10603](https://github.com/ClickHouse/ClickHouse/pull/10603) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) * Fixed bug, which caused HTTP requests to get stuck on client closing connection when `readonly=2` and `cancel_http_readonly_queries_on_client_close=1`. [#10684](https://github.com/ClickHouse/ClickHouse/pull/10684) ([tavplubix](https://github.com/tavplubix)) * Fix segfault in StorageBuffer when exception is thrown on server startup. Fixes [#10550](https://github.com/ClickHouse/ClickHouse/issues/10550) [#10609](https://github.com/ClickHouse/ClickHouse/pull/10609) ([tavplubix](https://github.com/tavplubix)) @@ -2283,7 +2283,7 @@ No changes compared to v20.4.3.16-stable. * Added support of `MATERIALIZE TTL IN PARTITION`. [#9581](https://github.com/ClickHouse/ClickHouse/pull/9581) ([Vladimir Chebotarev](https://github.com/excitoon)) * Support complex types inside Avro nested fields [#10502](https://github.com/ClickHouse/ClickHouse/pull/10502) ([Andrew Onyshchuk](https://github.com/oandrew)) -#### Performance Improvement {#performance-improvement-11} +#### Performance improvement {#performance-improvement-11} * Better insert logic for right table for Partial MergeJoin. [#10467](https://github.com/ClickHouse/ClickHouse/pull/10467) ([Artem Zuikov](https://github.com/4ertus2)) * Improved performance of row-oriented formats (more than 10% for CSV and more than 35% for Avro in case of narrow tables). [#10503](https://github.com/ClickHouse/ClickHouse/pull/10503) ([Andrew Onyshchuk](https://github.com/oandrew)) * Improved performance of queries with explicitly defined sets at right side of IN operator and tuples on the left side. [#10385](https://github.com/ClickHouse/ClickHouse/pull/10385) ([Anton Popov](https://github.com/CurtizJ)) @@ -2297,7 +2297,7 @@ No changes compared to v20.4.3.16-stable. * Adding more aggregation methods. For example TPC-H query 1 will now pick `FixedHashMap` and gets 25% performance gain [#9829](https://github.com/ClickHouse/ClickHouse/pull/9829) ([Amos Bird](https://github.com/amosbird)) * Use single row counter for multiple streams in pre-limit transform. This helps to avoid uniting pipeline streams in queries with `limit` but without `order by` (like `select f(x) from (select x from t limit 1000000000)`) and use multiple threads for further processing. [#9602](https://github.com/ClickHouse/ClickHouse/pull/9602) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-15} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-15} * Use a fork of AWS SDK libraries from ClickHouse-Extras [#10527](https://github.com/ClickHouse/ClickHouse/pull/10527) ([Pavel Kovalenko](https://github.com/Jokser)) * Add integration tests for new ALTER RENAME COLUMN query. [#10654](https://github.com/ClickHouse/ClickHouse/pull/10654) ([vzakaznikov](https://github.com/vzakaznikov)) * Fix possible signed integer overflow in invocation of function `now64` with wrong arguments. This fixes [#8973](https://github.com/ClickHouse/ClickHouse/issues/8973) [#10511](https://github.com/ClickHouse/ClickHouse/pull/10511) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -2356,7 +2356,7 @@ No changes compared to v20.4.3.16-stable. ### ClickHouse release v20.3.21.2-lts, 2020-11-02 {#clickhouse-release-v203212-lts-2020-11-02} -#### Bug Fix {#bug-fix-33} +#### Bug fix {#bug-fix-33} * Fix dictGet in sharding_key (and similar places, i.e. when the function context is stored permanently). [#16205](https://github.com/ClickHouse/ClickHouse/pull/16205) ([Azat Khuzhin](https://github.com/azat)). * Fix incorrect empty result for query from `Distributed` table if query has `WHERE`, `PREWHERE` and `GLOBAL IN`. Fixes [#15792](https://github.com/ClickHouse/ClickHouse/issues/15792). [#15933](https://github.com/ClickHouse/ClickHouse/pull/15933) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -2364,7 +2364,7 @@ No changes compared to v20.4.3.16-stable. ### ClickHouse release v20.3.20.6-lts, 2020-10-09 {#clickhouse-release-v203206-lts-2020-10-09} -#### Bug Fix {#bug-fix-34} +#### Bug fix {#bug-fix-34} * Mutation might hang waiting for some non-existent part after `MOVE` or `REPLACE PARTITION` or, in rare cases, after `DETACH` or `DROP PARTITION`. It's fixed. [#15724](https://github.com/ClickHouse/ClickHouse/pull/15724), [#15537](https://github.com/ClickHouse/ClickHouse/pull/15537) ([tavplubix](https://github.com/tavplubix)). * Fix hang of queries with a lot of subqueries to same table of `MySQL` engine. Previously, if there were more than 16 subqueries to same `MySQL` table in query, it hang forever. [#15299](https://github.com/ClickHouse/ClickHouse/pull/15299) ([Anton Popov](https://github.com/CurtizJ)). @@ -2374,7 +2374,7 @@ No changes compared to v20.4.3.16-stable. ### ClickHouse release v20.3.19.4-lts, 2020-09-18 {#clickhouse-release-v203194-lts-2020-09-18} -#### Bug Fix {#bug-fix-35} +#### Bug fix {#bug-fix-35} * Fix rare error in `SELECT` queries when the queried column has `DEFAULT` expression which depends on the other column which also has `DEFAULT` and not present in select query and not exists on disk. Partially fixes [#14531](https://github.com/ClickHouse/ClickHouse/issues/14531). [#14845](https://github.com/ClickHouse/ClickHouse/pull/14845) ([alesapin](https://github.com/alesapin)). * Fix bug when `ALTER UPDATE` mutation with Nullable column in assignment expression and constant value (like `UPDATE x = 42`) leads to incorrect value in column or segfault. Fixes [#13634](https://github.com/ClickHouse/ClickHouse/issues/13634), [#14045](https://github.com/ClickHouse/ClickHouse/issues/14045). [#14646](https://github.com/ClickHouse/ClickHouse/pull/14646) ([alesapin](https://github.com/alesapin)). @@ -2386,7 +2386,7 @@ No changes compared to v20.4.3.16-stable. ### ClickHouse release v20.3.18.10-lts, 2020-09-08 {#clickhouse-release-v2031810-lts-2020-09-08} -#### Bug Fix {#bug-fix-36} +#### Bug fix {#bug-fix-36} * Stop query execution if exception happened in `PipelineExecutor` itself. This could prevent rare possible query hung. Continuation of [#14334](https://github.com/ClickHouse/ClickHouse/issues/14334). [#14402](https://github.com/ClickHouse/ClickHouse/pull/14402) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fixed the behaviour when sometimes cache-dictionary returned default value instead of present value from source. [#13624](https://github.com/ClickHouse/ClickHouse/pull/13624) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). @@ -2399,17 +2399,17 @@ No changes compared to v20.4.3.16-stable. * Fix wrong error for long queries. It was possible to get syntax error other than `Max query size exceeded` for correct query. [#13928](https://github.com/ClickHouse/ClickHouse/pull/13928) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Return NULL/zero when value is not parsed completely in parseDateTimeBestEffortOrNull/Zero functions. This fixes [#7876](https://github.com/ClickHouse/ClickHouse/issues/7876). [#11653](https://github.com/ClickHouse/ClickHouse/pull/11653) ([alexey-milovidov](https://github.com/alexey-milovidov)). -#### Performance Improvement {#performance-improvement-12} +#### Performance improvement {#performance-improvement-12} * Slightly optimize very short queries with LowCardinality. [#14129](https://github.com/ClickHouse/ClickHouse/pull/14129) ([Anton Popov](https://github.com/CurtizJ)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-16} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-16} * Fix UBSan report (adding zero to nullptr) in HashTable that appeared after migration to clang-10. [#10638](https://github.com/ClickHouse/ClickHouse/pull/10638) ([alexey-milovidov](https://github.com/alexey-milovidov)). ### ClickHouse release v20.3.17.173-lts, 2020-08-15 {#clickhouse-release-v20317173-lts-2020-08-15} -#### Bug Fix {#bug-fix-37} +#### Bug fix {#bug-fix-37} * Fix crash in JOIN with StorageMerge and `set enable_optimize_predicate_expression=1`. [#13679](https://github.com/ClickHouse/ClickHouse/pull/13679) ([Artem Zuikov](https://github.com/4ertus2)). * Fix invalid return type for comparison of tuples with `NULL` elements. Fixes [#12461](https://github.com/ClickHouse/ClickHouse/issues/12461). [#13420](https://github.com/ClickHouse/ClickHouse/pull/13420) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -2418,7 +2418,7 @@ No changes compared to v20.4.3.16-stable. ### ClickHouse release v20.3.16.165-lts 2020-08-10 {#clickhouse-release-v20316165-lts-2020-08-10} -#### Bug Fix {#bug-fix-38} +#### Bug fix {#bug-fix-38} * Fixed error in `parseDateTimeBestEffort` function when unix timestamp was passed as an argument. This fixes [#13362](https://github.com/ClickHouse/ClickHouse/issues/13362). [#13441](https://github.com/ClickHouse/ClickHouse/pull/13441) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed potentially low performance and slightly incorrect result for `uniqExact`, `topK`, `sumDistinct` and similar aggregate functions called on Float types with `NaN` values. It also triggered assert in debug build. This fixes [#12491](https://github.com/ClickHouse/ClickHouse/issues/12491). [#13254](https://github.com/ClickHouse/ClickHouse/pull/13254) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -2475,13 +2475,13 @@ No changes compared to v20.4.3.16-stable. * Fixed `LIMIT n WITH TIES` usage together with `ORDER BY` statement, which contains aliases. [#11689](https://github.com/ClickHouse/ClickHouse/pull/11689) ([Anton Popov](https://github.com/CurtizJ)). * Fix potential read of uninitialized memory in cache dictionary. [#10834](https://github.com/ClickHouse/ClickHouse/pull/10834) ([alexey-milovidov](https://github.com/alexey-milovidov)). -#### Performance Improvement {#performance-improvement-13} +#### Performance improvement {#performance-improvement-13} * Index not used for IN operator with literals, performance regression introduced around v19.3. This fixes [#10574](https://github.com/ClickHouse/ClickHouse/issues/10574). [#12062](https://github.com/ClickHouse/ClickHouse/pull/12062) ([nvartolomei](https://github.com/nvartolomei)). ### ClickHouse release v20.3.12.112-lts 2020-06-25 {#clickhouse-release-v20312112-lts-2020-06-25} -#### Bug Fix {#bug-fix-39} +#### Bug fix {#bug-fix-39} * Fix rare crash caused by using `Nullable` column in prewhere condition. Continuation of [#11608](https://github.com/ClickHouse/ClickHouse/issues/11608). [#11869](https://github.com/ClickHouse/ClickHouse/pull/11869) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Don't allow arrayJoin inside higher order functions. It was leading to broken protocol synchronization. This closes [#3933](https://github.com/ClickHouse/ClickHouse/issues/3933). [#11846](https://github.com/ClickHouse/ClickHouse/pull/11846) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -2508,11 +2508,11 @@ No changes compared to v20.4.3.16-stable. ### ClickHouse release v20.3.11.97-lts 2020-06-10 {#clickhouse-release-v2031197-lts-2020-06-10} -#### New Feature {#new-feature-9} +#### New feature {#new-feature-9} * Now ClickHouse controls timeouts of dictionary sources on its side. Two new settings added to cache dictionary configuration: `strict_max_lifetime_seconds`, which is `max_lifetime` by default and `query_wait_timeout_milliseconds`, which is one minute by default. The first settings is also useful with `allow_read_expired_keys` settings (to forbid reading very expired keys). [#10337](https://github.com/ClickHouse/ClickHouse/pull/10337) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). -#### Bug Fix {#bug-fix-40} +#### Bug fix {#bug-fix-40} * Fix the error `Data compressed with different methods` that can happen if `min_bytes_to_use_direct_io` is enabled and PREWHERE is active and using SAMPLE or high number of threads. This fixes [#11539](https://github.com/ClickHouse/ClickHouse/issues/11539). [#11540](https://github.com/ClickHouse/ClickHouse/pull/11540) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix return compressed size for codecs. [#11448](https://github.com/ClickHouse/ClickHouse/pull/11448) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -2545,17 +2545,17 @@ No changes compared to v20.4.3.16-stable. * Fix predicates optimization for distributed queries (`enable_optimize_predicate_expression=1`) for queries with `HAVING` section (i.e. when filtering on the server initiator is required), by preserving the order of expressions (and this is enough to fix), and also force aggregator use column names over indexes. Fixes: [#10613](https://github.com/ClickHouse/ClickHouse/issues/10613), [#11413](https://github.com/ClickHouse/ClickHouse/issues/11413). [#10621](https://github.com/ClickHouse/ClickHouse/pull/10621) ([Azat Khuzhin](https://github.com/azat)). * Introduce commit retry logic to decrease the possibility of getting duplicates from Kafka in rare cases when offset commit was failed. [#9884](https://github.com/ClickHouse/ClickHouse/pull/9884) ([filimonov](https://github.com/filimonov)). -#### Performance Improvement {#performance-improvement-14} +#### Performance improvement {#performance-improvement-14} * Get dictionary and check access rights only once per each call of any function reading external dictionaries. [#10928](https://github.com/ClickHouse/ClickHouse/pull/10928) ([Vitaly Baranov](https://github.com/vitlibar)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-17} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-17} * Fix several flaky integration tests. [#11355](https://github.com/ClickHouse/ClickHouse/pull/11355) ([alesapin](https://github.com/alesapin)). ### ClickHouse release v20.3.10.75-lts 2020-05-23 {#clickhouse-release-v2031075-lts-2020-05-23} -#### Bug Fix {#bug-fix-41} +#### Bug fix {#bug-fix-41} * Removed logging from mutation finalization task if nothing was finalized. [#11109](https://github.com/ClickHouse/ClickHouse/pull/11109) ([alesapin](https://github.com/alesapin)). * Fixed `parseDateTime64BestEffort` argument resolution bugs. [#11038](https://github.com/ClickHouse/ClickHouse/pull/11038) ([Vasily Nemkov](https://github.com/Enmk)). @@ -2590,7 +2590,7 @@ No changes compared to v20.4.3.16-stable. * Fixed improper shutdown of `Distributed` storage. [#10491](https://github.com/ClickHouse/ClickHouse/pull/10491) ([Azat Khuzhin](https://github.com/azat)). * Fixed numeric overflow in `simpleLinearRegression` over large integers. [#10474](https://github.com/ClickHouse/ClickHouse/pull/10474) ([hcz](https://github.com/hczhcz)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-18} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-18} * Fix UBSan report in LZ4 library. [#10631](https://github.com/ClickHouse/ClickHouse/pull/10631) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix clang-10 build. [#10238](https://github.com/ClickHouse/ClickHouse/issues/10238). [#10370](https://github.com/ClickHouse/ClickHouse/pull/10370) ([Amos Bird](https://github.com/amosbird)). @@ -2604,7 +2604,7 @@ No changes compared to v20.4.3.16-stable. ### ClickHouse release v20.3.8.53, 2020-04-23 {#clickhouse-release-v203853-2020-04-23} -#### Bug Fix {#bug-fix-43} +#### Bug fix {#bug-fix-43} * Fixed wrong behaviour of datetime functions for timezones that has altered between positive and negative offsets from UTC (e.g. Pacific/Kiritimati). This fixes [#7202](https://github.com/ClickHouse/ClickHouse/issues/7202) [#10369](https://github.com/ClickHouse/ClickHouse/pull/10369) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fix possible segfault with `distributed_group_by_no_merge` enabled (introduced in 20.3.7.46 by [#10131](https://github.com/ClickHouse/ClickHouse/issues/10131)). [#10399](https://github.com/ClickHouse/ClickHouse/pull/10399) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) * Fix wrong flattening of `Array(Tuple(...))` data types. This fixes [#10259](https://github.com/ClickHouse/ClickHouse/issues/10259) [#10390](https://github.com/ClickHouse/ClickHouse/pull/10390) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -2620,12 +2620,12 @@ No changes compared to v20.4.3.16-stable. * Fix the issue when a query with ARRAY JOIN, ORDER BY and LIMIT may return incomplete result. This fixes [#10226](https://github.com/ClickHouse/ClickHouse/issues/10226). Author: [Vadim Plakhtinskiy](https://github.com/VadimPlh). [#10427](https://github.com/ClickHouse/ClickHouse/pull/10427) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Check the number and type of arguments when creating BloomFilter index [#9623](https://github.com/ClickHouse/ClickHouse/issues/9623) [#10431](https://github.com/ClickHouse/ClickHouse/pull/10431) ([Winter Zhang](https://github.com/zhang2014)) -#### Performance Improvement {#performance-improvement-15} +#### Performance improvement {#performance-improvement-15} * Improved performance of queries with explicitly defined sets at right side of `IN` operator and tuples in the left side. This fixes performance regression in version 20.3. [#9740](https://github.com/ClickHouse/ClickHouse/pull/9740), [#10385](https://github.com/ClickHouse/ClickHouse/pull/10385) ([Anton Popov](https://github.com/CurtizJ)) ### ClickHouse release v20.3.7.46, 2020-04-17 {#clickhouse-release-v203746-2020-04-17} -#### Bug Fix {#bug-fix-44} +#### Bug fix {#bug-fix-44} * Fix `Logical error: CROSS JOIN has expressions` error for queries with comma and names joins mix. [#10311](https://github.com/ClickHouse/ClickHouse/pull/10311) ([Artem Zuikov](https://github.com/4ertus2)). * Fix queries with `max_bytes_before_external_group_by`. [#10302](https://github.com/ClickHouse/ClickHouse/pull/10302) ([Artem Zuikov](https://github.com/4ertus2)). @@ -2634,11 +2634,11 @@ No changes compared to v20.4.3.16-stable. ### ClickHouse release v20.3.6.40, 2020-04-16 {#clickhouse-release-v203640-2020-04-16} -#### New Feature {#new-feature-10} +#### New feature {#new-feature-10} * Added function `isConstant`. This function checks whether its argument is constant expression and returns 1 or 0. It is intended for development, debugging and demonstration purposes. [#10198](https://github.com/ClickHouse/ClickHouse/pull/10198) ([alexey-milovidov](https://github.com/alexey-milovidov)). -#### Bug Fix {#bug-fix-45} +#### Bug fix {#bug-fix-45} * Fix error `Pipeline stuck` with `max_rows_to_group_by` and `group_by_overflow_mode = 'break'`. [#10279](https://github.com/ClickHouse/ClickHouse/pull/10279) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix rare possible exception `Cannot drain connections: cancel first`. [#10239](https://github.com/ClickHouse/ClickHouse/pull/10239) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -2672,14 +2672,14 @@ No changes compared to v20.4.3.16-stable. * Fix error with qualified names in `distributed_product_mode='local'`. Fixes [#4756](https://github.com/ClickHouse/ClickHouse/issues/4756). [#9891](https://github.com/ClickHouse/ClickHouse/pull/9891) ([Artem Zuikov](https://github.com/4ertus2)). * Fix calculating grants for introspection functions from the setting 'allow_introspection_functions'. [#9840](https://github.com/ClickHouse/ClickHouse/pull/9840) ([Vitaly Baranov](https://github.com/vitlibar)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-19} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-19} * Fix integration test `test_settings_constraints`. [#9962](https://github.com/ClickHouse/ClickHouse/pull/9962) ([Vitaly Baranov](https://github.com/vitlibar)). * Removed dependency on `clock_getres`. [#9833](https://github.com/ClickHouse/ClickHouse/pull/9833) ([alexey-milovidov](https://github.com/alexey-milovidov)). ### ClickHouse release v20.3.5.21, 2020-03-27 {#clickhouse-release-v203521-2020-03-27} -#### Bug Fix {#bug-fix-46} +#### Bug fix {#bug-fix-46} * Fix 'Different expressions with the same alias' error when query has PREWHERE and WHERE on distributed table and `SET distributed_product_mode = 'local'`. [#9871](https://github.com/ClickHouse/ClickHouse/pull/9871) ([Artem Zuikov](https://github.com/4ertus2)). * Fix mutations excessive memory consumption for tables with a composite primary key. This fixes [#9850](https://github.com/ClickHouse/ClickHouse/issues/9850). [#9860](https://github.com/ClickHouse/ClickHouse/pull/9860) ([alesapin](https://github.com/alesapin)). @@ -2697,20 +2697,20 @@ No changes compared to v20.4.3.16-stable. ### ClickHouse release v20.3.4.10, 2020-03-20 {#clickhouse-release-v203410-2020-03-20} -#### Bug Fix {#bug-fix-47} +#### Bug fix {#bug-fix-47} * This release also contains all bug fixes from 20.1.8.41 * Fix missing `rows_before_limit_at_least` for queries over http (with processors pipeline). This fixes [#9730](https://github.com/ClickHouse/ClickHouse/issues/9730). [#9757](https://github.com/ClickHouse/ClickHouse/pull/9757) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) ### ClickHouse release v20.3.3.6, 2020-03-17 {#clickhouse-release-v20336-2020-03-17} -#### Bug Fix {#bug-fix-48} +#### Bug fix {#bug-fix-48} * This release also contains all bug fixes from 20.1.7.38 * Fix bug in a replication that does not allow replication to work if the user has executed mutations on the previous version. This fixes [#9645](https://github.com/ClickHouse/ClickHouse/issues/9645). [#9652](https://github.com/ClickHouse/ClickHouse/pull/9652) ([alesapin](https://github.com/alesapin)). It makes version 20.3 backward compatible again. * Add setting `use_compact_format_in_distributed_parts_names` which allows to write files for `INSERT` queries into `Distributed` table with more compact format. This fixes [#9647](https://github.com/ClickHouse/ClickHouse/issues/9647). [#9653](https://github.com/ClickHouse/ClickHouse/pull/9653) ([alesapin](https://github.com/alesapin)). It makes version 20.3 backward compatible again. ### ClickHouse release v20.3.2.1, 2020-03-12 {#clickhouse-release-v20321-2020-03-12} -#### Backward Incompatible Change {#backward-incompatible-change-9} +#### Backward incompatible change {#backward-incompatible-change-9} * Fixed the issue `file name too long` when sending data for `Distributed` tables for a large number of replicas. Fixed the issue that replica credentials were exposed in the server log. The format of directory name on disk was changed to `[shard{shard_index}[_replica{replica_index}]]`. [#8911](https://github.com/ClickHouse/ClickHouse/pull/8911) ([Mikhail Korotov](https://github.com/millb)) After you upgrade to the new version, you will not be able to downgrade without manual intervention, because old server version does not recognize the new directory format. If you want to downgrade, you have to manually rename the corresponding directories to the old format. This change is relevant only if you have used asynchronous `INSERT`s to `Distributed` tables. In the version 20.3.3 we will introduce a setting that will allow you to enable the new format gradually. * Changed the format of replication log entries for mutation commands. You have to wait for old mutations to process before installing the new version. @@ -2724,7 +2724,7 @@ No changes compared to v20.4.3.16-stable. * Require server to be restarted to apply the changes in logging configuration. This is a temporary workaround to avoid the bug where the server logs to a deleted log file (see [#8696](https://github.com/ClickHouse/ClickHouse/issues/8696)). [#8707](https://github.com/ClickHouse/ClickHouse/pull/8707) ([Alexander Kuzmenkov](https://github.com/akuzm)) * The setting `experimental_use_processors` is enabled by default. This setting enables usage of the new query pipeline. This is internal refactoring and we expect no visible changes. If you will see any issues, set it to back zero. [#8768](https://github.com/ClickHouse/ClickHouse/pull/8768) ([alexey-milovidov](https://github.com/alexey-milovidov)) -#### New Feature {#new-feature-11} +#### New feature {#new-feature-11} * Add `Avro` and `AvroConfluent` input/output formats [#8571](https://github.com/ClickHouse/ClickHouse/pull/8571) ([Andrew Onyshchuk](https://github.com/oandrew)) [#8957](https://github.com/ClickHouse/ClickHouse/pull/8957) ([Andrew Onyshchuk](https://github.com/oandrew)) [#8717](https://github.com/ClickHouse/ClickHouse/pull/8717) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Multi-threaded and non-blocking updates of expired keys in `cache` dictionaries (with optional permission to read old ones). [#8303](https://github.com/ClickHouse/ClickHouse/pull/8303) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)) * Add query `ALTER ... MATERIALIZE TTL`. It runs mutation that forces to remove expired data by TTL and recalculates meta-information about TTL in all parts. [#8775](https://github.com/ClickHouse/ClickHouse/pull/8775) ([Anton Popov](https://github.com/CurtizJ)) @@ -2754,11 +2754,11 @@ No changes compared to v20.4.3.16-stable. * Add function `moduloOrZero` [#9358](https://github.com/ClickHouse/ClickHouse/pull/9358) ([hcz](https://github.com/hczhcz)) * Added system tables `system.zeros` and `system.zeros_mt` as well as tale functions `zeros()` and `zeros_mt()`. Tables (and table functions) contain single column with name `zero` and type `UInt8`. This column contains zeros. It is needed for test purposes as the fastest method to generate many rows. This fixes [#6604](https://github.com/ClickHouse/ClickHouse/issues/6604) [#9593](https://github.com/ClickHouse/ClickHouse/pull/9593) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) -#### Experimental Feature {#experimental-feature-7} +#### Experimental feature {#experimental-feature-7} * Add new compact format of parts in `MergeTree`-family tables in which all columns are stored in one file. It helps to increase performance of small and frequent inserts. The old format (one file per column) is now called wide. Data storing format is controlled by settings `min_bytes_for_wide_part` and `min_rows_for_wide_part`. [#8290](https://github.com/ClickHouse/ClickHouse/pull/8290) ([Anton Popov](https://github.com/CurtizJ)) * Support for S3 storage for `Log`, `TinyLog` and `StripeLog` tables. [#8862](https://github.com/ClickHouse/ClickHouse/pull/8862) ([Pavel Kovalenko](https://github.com/Jokser)) -#### Bug Fix {#bug-fix-49} +#### Bug fix {#bug-fix-49} * Fixed inconsistent whitespaces in log messages. [#9322](https://github.com/ClickHouse/ClickHouse/pull/9322) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fix bug in which arrays of unnamed tuples were flattened as Nested structures on table creation. [#8866](https://github.com/ClickHouse/ClickHouse/pull/8866) ([achulkov2](https://github.com/achulkov2)) * Fixed the issue when "Too many open files" error may happen if there are too many files matching glob pattern in `File` table or `file` table function. Now files are opened lazily. This fixes [#8857](https://github.com/ClickHouse/ClickHouse/issues/8857) [#8861](https://github.com/ClickHouse/ClickHouse/pull/8861) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -2886,7 +2886,7 @@ No changes compared to v20.4.3.16-stable. * Add a system metric tracking the number of client connections using MySQL protocol ([#9013](https://github.com/ClickHouse/ClickHouse/issues/9013)). [#9015](https://github.com/ClickHouse/ClickHouse/pull/9015) ([Eugene Klimov](https://github.com/Slach)) * From now on, HTTP responses will have `X-ClickHouse-Timezone` header set to the same timezone value that `SELECT timezone()` would report. [#9493](https://github.com/ClickHouse/ClickHouse/pull/9493) ([Denis Glazachev](https://github.com/traceon)) -#### Performance Improvement {#performance-improvement-16} +#### Performance improvement {#performance-improvement-16} * Improve performance of analysing index with IN [#9261](https://github.com/ClickHouse/ClickHouse/pull/9261) ([Anton Popov](https://github.com/CurtizJ)) * Simpler and more efficient code in Logical Functions + code cleanups. A followup to [#8718](https://github.com/ClickHouse/ClickHouse/issues/8718) [#8728](https://github.com/ClickHouse/ClickHouse/pull/8728) ([Alexander Kazakov](https://github.com/Akazz)) * Overall performance improvement (in range of 5%..200% for affected queries) by ensuring even more strict aliasing with C++20 features. [#9304](https://github.com/ClickHouse/ClickHouse/pull/9304) ([Amos Bird](https://github.com/amosbird)) @@ -2898,7 +2898,7 @@ No changes compared to v20.4.3.16-stable. * Improve performance of `reinterpretAsFixedString` function. [#9342](https://github.com/ClickHouse/ClickHouse/pull/9342) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Do not send blocks to client for `Null` format in processors pipeline. [#8797](https://github.com/ClickHouse/ClickHouse/pull/8797) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) [#8767](https://github.com/ClickHouse/ClickHouse/pull/8767) ([Alexander Kuzmenkov](https://github.com/akuzm)) -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-20} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-20} * Exception handling now works correctly on Windows Subsystem for Linux. See https://github.com/ClickHouse-Extras/libunwind/pull/3 This fixes [#6480](https://github.com/ClickHouse/ClickHouse/issues/6480) [#9564](https://github.com/ClickHouse/ClickHouse/pull/9564) ([sobolevsv](https://github.com/sobolevsv)) * Replace `readline` with `replxx` for interactive line editing in `clickhouse-client` [#8416](https://github.com/ClickHouse/ClickHouse/pull/8416) ([Ivan](https://github.com/abyss7)) * Better build time and less template instantiations in FunctionsComparison. [#9324](https://github.com/ClickHouse/ClickHouse/pull/9324) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -2950,7 +2950,7 @@ No changes compared to v20.4.3.16-stable. ### ClickHouse release v20.1.16.120-stable 2020-60-26 {#clickhouse-release-v20116120-stable-2020-60-26} -#### Bug Fix {#bug-fix-50} +#### Bug fix {#bug-fix-50} * Fix rare crash caused by using `Nullable` column in prewhere condition. Continuation of [#11608](https://github.com/ClickHouse/ClickHouse/issues/11608). [#11869](https://github.com/ClickHouse/ClickHouse/pull/11869) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Don't allow arrayJoin inside higher order functions. It was leading to broken protocol synchronization. This closes [#3933](https://github.com/ClickHouse/ClickHouse/issues/3933). [#11846](https://github.com/ClickHouse/ClickHouse/pull/11846) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -2969,19 +2969,19 @@ No changes compared to v20.4.3.16-stable. ### ClickHouse release v20.1.15.109-stable 2020-06-19 {#clickhouse-release-v20115109-stable-2020-06-19} -#### Bug Fix {#bug-fix-51} +#### Bug fix {#bug-fix-51} * Fix excess lock for structure during alter. [#11790](https://github.com/ClickHouse/ClickHouse/pull/11790) ([alesapin](https://github.com/alesapin)). ### ClickHouse release v20.1.14.107-stable 2020-06-11 {#clickhouse-release-v20114107-stable-2020-06-11} -#### Bug Fix {#bug-fix-52} +#### Bug fix {#bug-fix-52} * Fix error `Size of offsets does not match size of column` for queries with `PREWHERE column in (subquery)` and `ARRAY JOIN`. [#11580](https://github.com/ClickHouse/ClickHouse/pull/11580) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). ### ClickHouse release v20.1.13.105-stable 2020-06-10 {#clickhouse-release-v20113105-stable-2020-06-10} -#### Bug Fix {#bug-fix-53} +#### Bug fix {#bug-fix-53} * Fix the error `Data compressed with different methods` that can happen if `min_bytes_to_use_direct_io` is enabled and PREWHERE is active and using SAMPLE or high number of threads. This fixes [#11539](https://github.com/ClickHouse/ClickHouse/issues/11539). [#11540](https://github.com/ClickHouse/ClickHouse/pull/11540) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix return compressed size for codecs. [#11448](https://github.com/ClickHouse/ClickHouse/pull/11448) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -3022,7 +3022,7 @@ No changes compared to v20.4.3.16-stable. ### ClickHouse release v20.1.12.86, 2020-05-26 {#clickhouse-release-v2011286-2020-05-26} -#### Bug Fix {#bug-fix-54} +#### Bug fix {#bug-fix-54} * Fixed incompatibility of two-level aggregation between versions 20.1 and earlier. This incompatibility happens when different versions of ClickHouse are used on initiator node and remote nodes and the size of GROUP BY result is large and aggregation is performed by a single String field. It leads to several unmerged rows for a single key in result. [#10952](https://github.com/ClickHouse/ClickHouse/pull/10952) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed data corruption for `LowCardinality(FixedString)` key column in `SummingMergeTree` which could have happened after merge. Fixes [#10489](https://github.com/ClickHouse/ClickHouse/issues/10489). [#10721](https://github.com/ClickHouse/ClickHouse/pull/10721) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -3044,13 +3044,13 @@ No changes compared to v20.4.3.16-stable. * Fixed several bugs when some data was inserted with quorum, then deleted somehow (DROP PARTITION, TTL) and this leaded to the stuck of INSERTs or false-positive exceptions in SELECTs. This fixes [#9946](https://github.com/ClickHouse/ClickHouse/issues/9946). [#10188](https://github.com/ClickHouse/ClickHouse/pull/10188) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Fixed incompatibility when versions prior to 18.12.17 are used on remote servers and newer is used on initiating server, and GROUP BY both fixed and non-fixed keys, and when two-level group by method is activated. [#3254](https://github.com/ClickHouse/ClickHouse/pull/3254) ([alexey-milovidov](https://github.com/alexey-milovidov)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-21} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-21} * Added CA certificates to clickhouse-server docker image. [#10476](https://github.com/ClickHouse/ClickHouse/pull/10476) ([filimonov](https://github.com/filimonov)). ### ClickHouse release v20.1.10.70, 2020-04-17 {#clickhouse-release-v2011070-2020-04-17} -#### Bug Fix {#bug-fix-55} +#### Bug fix {#bug-fix-55} * Fix rare possible exception `Cannot drain connections: cancel first`. [#10239](https://github.com/ClickHouse/ClickHouse/pull/10239) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fixed bug where ClickHouse would throw `'Unknown function lambda.'` error message when user tries to run `ALTER UPDATE/DELETE` on tables with `ENGINE = Replicated*`. Check for nondeterministic functions now handles lambda expressions correctly. [#10237](https://github.com/ClickHouse/ClickHouse/pull/10237) ([Alexander Kazakov](https://github.com/Akazz)). @@ -3074,13 +3074,13 @@ No changes compared to v20.4.3.16-stable. * Fix race condition between drop and optimize in `ReplicatedMergeTree`. [#9901](https://github.com/ClickHouse/ClickHouse/pull/9901) ([alesapin](https://github.com/alesapin)). * Fixed `DeleteOnDestroy` logic in `ATTACH PART` which could lead to automatic removal of attached part and added few tests. [#9410](https://github.com/ClickHouse/ClickHouse/pull/9410) ([Vladimir Chebotarev](https://github.com/excitoon)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-22} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-22} * Fix unit test `collapsing_sorted_stream`. [#9367](https://github.com/ClickHouse/ClickHouse/pull/9367) ([Deleted user](https://github.com/ghost)). ### ClickHouse release v20.1.9.54, 2020-03-28 {#clickhouse-release-v201954-2020-03-28} -#### Bug Fix {#bug-fix-56} +#### Bug fix {#bug-fix-56} * Fix `'Different expressions with the same alias'` error when query has `PREWHERE` and `WHERE` on distributed table and `SET distributed_product_mode = 'local'`. [#9871](https://github.com/ClickHouse/ClickHouse/pull/9871) ([Artem Zuikov](https://github.com/4ertus2)). * Fix mutations excessive memory consumption for tables with a composite primary key. This fixes [#9850](https://github.com/ClickHouse/ClickHouse/issues/9850). [#9860](https://github.com/ClickHouse/ClickHouse/pull/9860) ([alesapin](https://github.com/alesapin)). @@ -3093,20 +3093,20 @@ No changes compared to v20.4.3.16-stable. * Remove `ORDER BY` stage from mutations because we read from a single ordered part in a single thread. Also add check that the order of rows in mutation is ordered in sorting key order and this order is not violated. [#9886](https://github.com/ClickHouse/ClickHouse/pull/9886) ([alesapin](https://github.com/alesapin)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-23} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-23} * Clean up duplicated linker flags. Make sure the linker won't look up an unexpected symbol. [#9433](https://github.com/ClickHouse/ClickHouse/pull/9433) ([Amos Bird](https://github.com/amosbird)). ### ClickHouse release v20.1.8.41, 2020-03-20 {#clickhouse-release-v201841-2020-03-20} -#### Bug Fix {#bug-fix-57} +#### Bug fix {#bug-fix-57} * Fix possible permanent `Cannot schedule a task` error (due to unhandled exception in `ParallelAggregatingBlockInputStream::Handler::onFinish/onFinishThread`). This fixes [#6833](https://github.com/ClickHouse/ClickHouse/issues/6833). [#9154](https://github.com/ClickHouse/ClickHouse/pull/9154) ([Azat Khuzhin](https://github.com/azat)) * Fix excessive memory consumption in `ALTER` queries (mutations). This fixes [#9533](https://github.com/ClickHouse/ClickHouse/issues/9533) and [#9670](https://github.com/ClickHouse/ClickHouse/issues/9670). [#9754](https://github.com/ClickHouse/ClickHouse/pull/9754) ([alesapin](https://github.com/alesapin)) * Fix bug in backquoting in external dictionaries DDL. This fixes [#9619](https://github.com/ClickHouse/ClickHouse/issues/9619). [#9734](https://github.com/ClickHouse/ClickHouse/pull/9734) ([alesapin](https://github.com/alesapin)) ### ClickHouse release v20.1.7.38, 2020-03-18 {#clickhouse-release-v201738-2020-03-18} -#### Bug Fix {#bug-fix-58} +#### Bug fix {#bug-fix-58} * Fixed incorrect internal function names for `sumKahan` and `sumWithOverflow`. I lead to exception while using this functions in remote queries. [#9636](https://github.com/ClickHouse/ClickHouse/pull/9636) ([Azat Khuzhin](https://github.com/azat)). This issue was in all ClickHouse releases. * Allow `ALTER ON CLUSTER` of `Distributed` tables with internal replication. This fixes [#3268](https://github.com/ClickHouse/ClickHouse/issues/3268). [#9617](https://github.com/ClickHouse/ClickHouse/pull/9617) ([shinoi2](https://github.com/shinoi2)). This issue was in all ClickHouse releases. * Fix possible exceptions `Size of filter does not match size of column` and `Invalid number of rows in Chunk` in `MergeTreeRangeReader`. They could appear while executing `PREWHERE` in some cases. Fixes [#9132](https://github.com/ClickHouse/ClickHouse/issues/9132). [#9612](https://github.com/ClickHouse/ClickHouse/pull/9612) ([Anton Popov](https://github.com/CurtizJ)) @@ -3122,13 +3122,13 @@ No changes compared to v20.4.3.16-stable. * Fix not(has()) for the bloom_filter index of array types. [#9407](https://github.com/ClickHouse/ClickHouse/pull/9407) ([achimbab](https://github.com/achimbab)) * Fixed the behaviour of `match` and `extract` functions when haystack has zero bytes. The behaviour was wrong when haystack was constant. This fixes [#9160](https://github.com/ClickHouse/ClickHouse/issues/9160) [#9163](https://github.com/ClickHouse/ClickHouse/pull/9163) ([alexey-milovidov](https://github.com/alexey-milovidov)) [#9345](https://github.com/ClickHouse/ClickHouse/pull/9345) ([alexey-milovidov](https://github.com/alexey-milovidov)) -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-24} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-24} * Exception handling now works correctly on Windows Subsystem for Linux. See https://github.com/ClickHouse-Extras/libunwind/pull/3 This fixes [#6480](https://github.com/ClickHouse/ClickHouse/issues/6480) [#9564](https://github.com/ClickHouse/ClickHouse/pull/9564) ([sobolevsv](https://github.com/sobolevsv)) ### ClickHouse release v20.1.6.30, 2020-03-05 {#clickhouse-release-v201630-2020-03-05} -#### Bug Fix {#bug-fix-59} +#### Bug fix {#bug-fix-59} * Fix data incompatibility when compressed with `T64` codec. [#9039](https://github.com/ClickHouse/ClickHouse/pull/9039) [(abyss7)](https://github.com/abyss7) @@ -3181,13 +3181,13 @@ No changes compared to v20.4.3.16-stable. * Fix bug leading to server termination when trying to use / drop `Kafka` table created with wrong parameters. Fixes [#9494](https://github.com/ClickHouse/ClickHouse/issues/9494). Incorporates [#9507](https://github.com/ClickHouse/ClickHouse/issues/9507). [#9513](https://github.com/ClickHouse/ClickHouse/pull/9513) [(filimonov)](https://github.com/filimonov) -#### New Feature {#new-feature-12} +#### New feature {#new-feature-12} * Add `deduplicate_blocks_in_dependent_materialized_views` option to control the behaviour of idempotent inserts into tables with materialized views. This new feature was added to the bugfix release by a special request from Altinity. [#9070](https://github.com/ClickHouse/ClickHouse/pull/9070) [(urykhy)](https://github.com/urykhy) ### ClickHouse release v20.1.2.4, 2020-01-22 {#clickhouse-release-v20124-2020-01-22} -#### Backward Incompatible Change {#backward-incompatible-change-10} +#### Backward incompatible change {#backward-incompatible-change-10} * Make the setting `merge_tree_uniform_read_distribution` obsolete. The server still recognizes this setting but it has no effect. [#8308](https://github.com/ClickHouse/ClickHouse/pull/8308) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Changed return type of the function `greatCircleDistance` to `Float32` because now the result of calculation is `Float32`. [#7993](https://github.com/ClickHouse/ClickHouse/pull/7993) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Now it's expected that query parameters are represented in "escaped" format. For example, to pass string `ab` you have to write `a\tb` or `a\b` and respectively, `a%5Ctb` or `a%5C%09b` in URL. This is needed to add the possibility to pass NULL as `\N`. This fixes [#7488](https://github.com/ClickHouse/ClickHouse/issues/7488). [#8517](https://github.com/ClickHouse/ClickHouse/pull/8517) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -3196,7 +3196,7 @@ No changes compared to v20.4.3.16-stable. * Add new `ANY JOIN` logic for `StorageJoin` consistent with `JOIN` operation. To upgrade without changes in behaviour you need add `SETTINGS any_join_distinct_right_table_keys = 1` to Engine Join tables metadata or recreate these tables after upgrade. [#8400](https://github.com/ClickHouse/ClickHouse/pull/8400) ([Artem Zuikov](https://github.com/4ertus2)) * Require server to be restarted to apply the changes in logging configuration. This is a temporary workaround to avoid the bug where the server logs to a deleted log file (see [#8696](https://github.com/ClickHouse/ClickHouse/issues/8696)). [#8707](https://github.com/ClickHouse/ClickHouse/pull/8707) ([Alexander Kuzmenkov](https://github.com/akuzm)) -#### New Feature {#new-feature-13} +#### New feature {#new-feature-13} * Added information about part paths to `system.merges`. [#8043](https://github.com/ClickHouse/ClickHouse/pull/8043) ([Vladimir Chebotarev](https://github.com/excitoon)) * Add ability to execute `SYSTEM RELOAD DICTIONARY` query in `ON CLUSTER` mode. [#8288](https://github.com/ClickHouse/ClickHouse/pull/8288) ([Guillaume Tassery](https://github.com/YiuRULE)) * Add ability to execute `CREATE DICTIONARY` queries in `ON CLUSTER` mode. [#8163](https://github.com/ClickHouse/ClickHouse/pull/8163) ([alesapin](https://github.com/alesapin)) @@ -3229,7 +3229,7 @@ No changes compared to v20.4.3.16-stable. * Added support for brotli (`br`) compression in file-related storages and table functions. This fixes [#8156](https://github.com/ClickHouse/ClickHouse/issues/8156). [#8526](https://github.com/ClickHouse/ClickHouse/pull/8526) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Add `groupBit*` functions for the `SimpleAggregationFunction` type. [#8485](https://github.com/ClickHouse/ClickHouse/pull/8485) ([Guillaume Tassery](https://github.com/YiuRULE)) -#### Bug Fix {#bug-fix-60} +#### Bug fix {#bug-fix-60} * Fix rename of tables with `Distributed` engine. Fixes issue [#7868](https://github.com/ClickHouse/ClickHouse/issues/7868). [#8306](https://github.com/ClickHouse/ClickHouse/pull/8306) ([tavplubix](https://github.com/tavplubix)) * Now dictionaries support `EXPRESSION` for attributes in arbitrary string in non-ClickHouse SQL dialect. [#8098](https://github.com/ClickHouse/ClickHouse/pull/8098) ([alesapin](https://github.com/alesapin)) * Fix broken `INSERT SELECT FROM mysql(...)` query. This fixes [#8070](https://github.com/ClickHouse/ClickHouse/issues/8070) and [#7960](https://github.com/ClickHouse/ClickHouse/issues/7960). [#8234](https://github.com/ClickHouse/ClickHouse/pull/8234) ([tavplubix](https://github.com/tavplubix)) @@ -3378,7 +3378,7 @@ No changes compared to v20.4.3.16-stable. * The settings `mark_cache_min_lifetime` is now obsolete and does nothing. In previous versions, mark cache can grow in memory larger than `mark_cache_size` to accomodate data within `mark_cache_min_lifetime` seconds. That was leading to confusion and higher memory usage than expected, that is especially bad on memory constrained systems. If you will see performance degradation after installing this release, you should increase the `mark_cache_size`. [#8484](https://github.com/ClickHouse/ClickHouse/pull/8484) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Preparation to use `tid` everywhere. This is needed for [#7477](https://github.com/ClickHouse/ClickHouse/issues/7477). [#8276](https://github.com/ClickHouse/ClickHouse/pull/8276) ([alexey-milovidov](https://github.com/alexey-milovidov)) -#### Performance Improvement {#performance-improvement-17} +#### Performance improvement {#performance-improvement-17} * Performance optimizations in processors pipeline. [#7988](https://github.com/ClickHouse/ClickHouse/pull/7988) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) * Non-blocking updates of expired keys in cache dictionaries (with permission to read old ones). [#8303](https://github.com/ClickHouse/ClickHouse/pull/8303) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)) * Compile ClickHouse without `-fno-omit-frame-pointer` globally to spare one more register. [#8097](https://github.com/ClickHouse/ClickHouse/pull/8097) ([Amos Bird](https://github.com/amosbird)) @@ -3403,7 +3403,7 @@ No changes compared to v20.4.3.16-stable. * Parallel parsing data formats [#6553](https://github.com/ClickHouse/ClickHouse/pull/6553) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)) * Enable optimized parser of `Values` with expressions by default (`input_format_values_deduce_templates_of_expressions=1`). [#8231](https://github.com/ClickHouse/ClickHouse/pull/8231) ([tavplubix](https://github.com/tavplubix)) -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-25} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-25} * Build fixes for `ARM` and in minimal mode. [#8304](https://github.com/ClickHouse/ClickHouse/pull/8304) ([proller](https://github.com/proller)) * Add coverage file flush for `clickhouse-server` when std::atexit is not called. Also slightly improved logging in stateless tests with coverage. [#8267](https://github.com/ClickHouse/ClickHouse/pull/8267) ([alesapin](https://github.com/alesapin)) * Update LLVM library in contrib. Avoid using LLVM from OS packages. [#8258](https://github.com/ClickHouse/ClickHouse/pull/8258) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -3473,13 +3473,13 @@ No changes compared to v20.4.3.16-stable. * Update contrib/protobuf. [#8256](https://github.com/ClickHouse/ClickHouse/pull/8256) ([Matwey V. Kornilov](https://github.com/matwey)) * In preparation of switching to c++20 as a new year celebration. "May the C++ force be with ClickHouse." [#8447](https://github.com/ClickHouse/ClickHouse/pull/8447) ([Amos Bird](https://github.com/amosbird)) -#### Experimental Feature {#experimental-feature-8} +#### Experimental feature {#experimental-feature-8} * Added experimental setting `min_bytes_to_use_mmap_io`. It allows to read big files without copying data from kernel to userspace. The setting is disabled by default. Recommended threshold is about 64 MB, because mmap/munmap is slow. [#8520](https://github.com/ClickHouse/ClickHouse/pull/8520) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Reworked quotas as a part of access control system. Added new table `system.quotas`, new functions `currentQuota`, `currentQuotaKey`, new SQL syntax `CREATE QUOTA`, `ALTER QUOTA`, `DROP QUOTA`, `SHOW QUOTA`. [#7257](https://github.com/ClickHouse/ClickHouse/pull/7257) ([Vitaly Baranov](https://github.com/vitlibar)) * Allow skipping unknown settings with warnings instead of throwing exceptions. [#7653](https://github.com/ClickHouse/ClickHouse/pull/7653) ([Vitaly Baranov](https://github.com/vitlibar)) * Reworked row policies as a part of access control system. Added new table `system.row_policies`, new function `currentRowPolicies()`, new SQL syntax `CREATE POLICY`, `ALTER POLICY`, `DROP POLICY`, `SHOW CREATE POLICY`, `SHOW POLICIES`. [#7808](https://github.com/ClickHouse/ClickHouse/pull/7808) ([Vitaly Baranov](https://github.com/vitlibar)) -#### Security Fix {#security-fix} +#### Security fix {#security-fix} * Fixed the possibility of reading directories structure in tables with `File` table engine. This fixes [#8536](https://github.com/ClickHouse/ClickHouse/issues/8536). [#8537](https://github.com/ClickHouse/ClickHouse/pull/8537) ([alexey-milovidov](https://github.com/alexey-milovidov)) ## [Changelog for 2019](./2019.md) {#changelog-for-2019} diff --git a/docs/whats-new/changelog/2021.md b/docs/whats-new/changelog/2021.md index 326aa2b244c..30fab9d6721 100644 --- a/docs/whats-new/changelog/2021.md +++ b/docs/whats-new/changelog/2021.md @@ -10,14 +10,14 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor ### ClickHouse release v21.12, 2021-12-15 {#clickhouse-release-v2112-2021-12-15} -#### Backward Incompatible Change {#backward-incompatible-change} +#### Backward incompatible change {#backward-incompatible-change} * *A fix for a feature that previously had unwanted behaviour.* Do not allow direct select for Kafka/RabbitMQ/FileLog. Can be enabled by setting `stream_like_engine_allow_direct_select`. Direct select will be not allowed even if enabled by setting, in case there is an attached materialized view. For Kafka and RabbitMQ direct selectm if allowed, will not commit massages by default. To enable commits with direct select, user must use storage level setting `kafka{rabbitmq}_commit_on_select=1` (default `0`). [#31053](https://github.com/ClickHouse/ClickHouse/pull/31053) ([Kseniia Sumarokova](https://github.com/kssenii)). * *A slight change in behaviour of a new function.* Return unquoted string in JSON_VALUE. Closes [#27965](https://github.com/ClickHouse/ClickHouse/issues/27965). [#31008](https://github.com/ClickHouse/ClickHouse/pull/31008) ([Kseniia Sumarokova](https://github.com/kssenii)). * *Setting rename.* Add custom null representation support for TSV/CSV input formats. Fix deserialing Nullable(String) in TSV/CSV/JSONCompactStringsEachRow/JSONStringsEachRow input formats. Rename `output_format_csv_null_representation` and `output_format_tsv_null_representation` to `format_csv_null_representation` and `format_tsv_null_representation` accordingly. [#30497](https://github.com/ClickHouse/ClickHouse/pull/30497) ([Kruglov Pavel](https://github.com/Avogar)). * *Further deprecation of already unused code.* This is relevant only for users of ClickHouse versions older than 20.6. A "leader election" mechanism is removed from `ReplicatedMergeTree`, because multiple leaders are supported since 20.6. If you are upgrading from an older version and some replica with an old version is a leader, then server will fail to start after upgrade. Stop replicas with old version to make new version start. After that it will not be possible to downgrade to version older than 20.6. [#32140](https://github.com/ClickHouse/ClickHouse/pull/32140) ([tavplubix](https://github.com/tavplubix)). -#### New Feature {#new-feature} +#### New feature {#new-feature} * Implemented more of the ZooKeeper Four Letter Words commands in clickhouse-keeper: https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html#sc_zkCommands. [#28981](https://github.com/ClickHouse/ClickHouse/pull/28981) ([JackyWoo](https://github.com/JackyWoo)). Now `clickhouse-keeper` is feature complete. * Support for `Bool` data type. [#31072](https://github.com/ClickHouse/ClickHouse/pull/31072) ([kevin wan](https://github.com/MaxWk)). @@ -36,13 +36,13 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Add Map combinator for `Map` type. - Rename old `sum-, min-, max- Map` for mapped arrays to `sum-, min-, max- MappedArrays`. [#24539](https://github.com/ClickHouse/ClickHouse/pull/24539) ([Ildus Kurbangaliev](https://github.com/ildus)). * Make reading from HTTP retriable. Closes [#29696](https://github.com/ClickHouse/ClickHouse/issues/29696). [#29894](https://github.com/ClickHouse/ClickHouse/pull/29894) ([Kseniia Sumarokova](https://github.com/kssenii)). -#### Experimental Feature {#experimental-feature} +#### Experimental feature {#experimental-feature} * `WINDOW VIEW` to enable stream processing in ClickHouse. [#8331](https://github.com/ClickHouse/ClickHouse/pull/8331) ([vxider](https://github.com/Vxider)). * Drop support for using Ordinary databases with `MaterializedMySQL`. [#31292](https://github.com/ClickHouse/ClickHouse/pull/31292) ([Stig Bakken](https://github.com/stigsb)). * Implement the commands BACKUP and RESTORE for the Log family. This feature is under development. [#30688](https://github.com/ClickHouse/ClickHouse/pull/30688) ([Vitaly Baranov](https://github.com/vitlibar)). -#### Performance Improvement {#performance-improvement} +#### Performance improvement {#performance-improvement} * Reduce memory usage when reading with `s3` / `url` / `hdfs` formats `Parquet`, `ORC`, `Arrow` (controlled by setting `input_format_allow_seeks`, enabled by default). Also add setting `remote_read_min_bytes_for_seek` to control seeks. Closes [#10461](https://github.com/ClickHouse/ClickHouse/issues/10461). Closes [#16857](https://github.com/ClickHouse/ClickHouse/issues/16857). [#30936](https://github.com/ClickHouse/ClickHouse/pull/30936) ([Kseniia Sumarokova](https://github.com/kssenii)). * Add optimizations for constant conditions in JOIN ON, ref [#26928](https://github.com/ClickHouse/ClickHouse/issues/26928). [#27021](https://github.com/ClickHouse/ClickHouse/pull/27021) ([Vladimir C](https://github.com/vdimir)). @@ -117,7 +117,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Fix abort in debug server and `DB::Exception: std::out_of_range: basic_string` error in release server in case of bad hdfs url by adding additional check of hdfs url structure. [#31042](https://github.com/ClickHouse/ClickHouse/pull/31042) ([Kruglov Pavel](https://github.com/Avogar)). * Fix possible assert in `hdfs` table function/engine, add test. [#31036](https://github.com/ClickHouse/ClickHouse/pull/31036) ([Kruglov Pavel](https://github.com/Avogar)). -#### Bug Fixes {#bug-fixes} +#### Bug fixes {#bug-fixes} * Fix group by / order by / limit by aliases with positional arguments enabled. Closes [#31173](https://github.com/ClickHouse/ClickHouse/issues/31173). [#31741](https://github.com/ClickHouse/ClickHouse/pull/31741) ([Kseniia Sumarokova](https://github.com/kssenii)). * Fix usage of `Buffer` table engine with type `Map`. Fixes [#30546](https://github.com/ClickHouse/ClickHouse/issues/30546). [#31742](https://github.com/ClickHouse/ClickHouse/pull/31742) ([Anton Popov](https://github.com/CurtizJ)). @@ -172,7 +172,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Skip `max_partition_size_to_drop check` in case of ATTACH PARTITION ... FROM and MOVE PARTITION ... [#30995](https://github.com/ClickHouse/ClickHouse/pull/30995) ([Amr Alaa](https://github.com/amralaa-MSFT)). * Fix some corner cases with `INTERSECT` and `EXCEPT` operators. Closes [#30803](https://github.com/ClickHouse/ClickHouse/issues/30803). [#30965](https://github.com/ClickHouse/ClickHouse/pull/30965) ([Kseniia Sumarokova](https://github.com/kssenii)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement} * Fix incorrect filtering result on non-x86 builds. This closes [#31417](https://github.com/ClickHouse/ClickHouse/issues/31417). This closes [#31524](https://github.com/ClickHouse/ClickHouse/issues/31524). [#31574](https://github.com/ClickHouse/ClickHouse/pull/31574) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Make ClickHouse build fully reproducible (byte identical on different machines). This closes [#22113](https://github.com/ClickHouse/ClickHouse/issues/22113). [#31899](https://github.com/ClickHouse/ClickHouse/pull/31899) ([alexey-milovidov](https://github.com/alexey-milovidov)). Remove filesystem path to the build directory from binaries to enable reproducible builds. This needed for [#22113](https://github.com/ClickHouse/ClickHouse/issues/22113). [#31838](https://github.com/ClickHouse/ClickHouse/pull/31838) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -187,14 +187,14 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor ### ClickHouse release v21.11, 2021-11-09 {#clickhouse-release-v2111-2021-11-09} -#### Backward Incompatible Change {#backward-incompatible-change-1} +#### Backward incompatible change {#backward-incompatible-change-1} * Change order of json_path and json arguments in SQL/JSON functions (to be consistent with the standard). Closes [#30449](https://github.com/ClickHouse/ClickHouse/issues/30449). [#30474](https://github.com/ClickHouse/ClickHouse/pull/30474) ([Kseniia Sumarokova](https://github.com/kssenii)). * Remove `MergeTree` table setting `write_final_mark`. It will be always `true`. [#30455](https://github.com/ClickHouse/ClickHouse/pull/30455) ([Kseniia Sumarokova](https://github.com/kssenii)). No actions required, all tables are compatible with the new version. * Function `bayesAB` is removed. Please help to return this function back, refreshed. This closes [#26233](https://github.com/ClickHouse/ClickHouse/issues/26233). [#29934](https://github.com/ClickHouse/ClickHouse/pull/29934) ([alexey-milovidov](https://github.com/alexey-milovidov)). * This is relevant only if you already started using the experimental `clickhouse-keeper` support. Now ClickHouse Keeper snapshots compressed with `ZSTD` codec by default instead of custom ClickHouse LZ4 block compression. This behavior can be turned off with `compress_snapshots_with_zstd_format` coordination setting (must be equal on all quorum replicas). Backward incompatibility is quite rare and may happen only when new node will send snapshot (happens in case of recovery) to the old node which is unable to read snapshots in ZSTD format. [#29417](https://github.com/ClickHouse/ClickHouse/pull/29417) ([alesapin](https://github.com/alesapin)). -#### New Feature {#new-feature-1} +#### New feature {#new-feature-1} * New asynchronous INSERT mode allows to accumulate inserted data and store it in a single batch in background. On client it can be enabled by setting `async_insert` for `INSERT` queries with data inlined in query or in separate buffer (e.g. for `INSERT` queries via HTTP protocol). If `wait_for_async_insert` is true (by default) the client will wait until data will be flushed to table. On server-side it controlled by the settings `async_insert_threads`, `async_insert_max_data_size` and `async_insert_busy_timeout_ms`. Implements [#18282](https://github.com/ClickHouse/ClickHouse/issues/18282). [#27537](https://github.com/ClickHouse/ClickHouse/pull/27537) ([Anton Popov](https://github.com/CurtizJ)). [#20557](https://github.com/ClickHouse/ClickHouse/pull/20557) ([Ivan](https://github.com/abyss7)). Notes on performance: with asynchronous inserts you can do up to around 10 000 individual INSERT queries per second, so it is still recommended to insert in batches if you want to achieve performance up to millions inserted rows per second. * Add interactive mode for `clickhouse-local`. So, you can just run `clickhouse-local` to get a command line ClickHouse interface without connecting to a server and process data from files and external data sources. Also merge the code of `clickhouse-client` and `clickhouse-local` together. Closes [#7203](https://github.com/ClickHouse/ClickHouse/issues/7203). Closes [#25516](https://github.com/ClickHouse/ClickHouse/issues/25516). Closes [#22401](https://github.com/ClickHouse/ClickHouse/issues/22401). [#26231](https://github.com/ClickHouse/ClickHouse/pull/26231) ([Kseniia Sumarokova](https://github.com/kssenii)). @@ -235,7 +235,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Added function accurateCastOrDefault(x, T). Closes [#21330](https://github.com/ClickHouse/ClickHouse/issues/21330). Authors @taiyang-li. [#23028](https://github.com/ClickHouse/ClickHouse/pull/23028) ([Maksim Kita](https://github.com/kitaisreal)). * Add Function `toUUIDOrDefault`, `toUInt8/16/32/64/256OrDefault`, `toInt8/16/32/64/128/256OrDefault`, which enables user defining default value(not null) when string parsing is failed. [#21330](https://github.com/ClickHouse/ClickHouse/pull/21330) ([taiyang-li](https://github.com/taiyang-li)). -#### Performance Improvement {#performance-improvement-1} +#### Performance improvement {#performance-improvement-1} * Background merges can be preempted by each other and they are scheduled with appropriate priorities. Now long running merges won't prevent short merges to proceed. This is needed for a better scheduling and controlling of merges execution. It reduces the chances to get "too many parts" error. [#22381](https://github.com/ClickHouse/ClickHouse/issues/22381). [#25165](https://github.com/ClickHouse/ClickHouse/pull/25165) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). Added an ability to execute more merges and mutations than the number of threads in background pool. Merges and mutations will be executed step by step according to their sizes (lower is more prioritized). The ratio of the number of tasks to threads to execute is controlled by a setting `background_merges_mutations_concurrency_ratio`, 2 by default. [#29140](https://github.com/ClickHouse/ClickHouse/pull/29140) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Allow to use asynchronous reads for remote filesystems. Lower the number of seeks while reading from remote filesystems. It improves performance tremendously and makes the experimental `web` and `s3` disks to work faster than EBS under certain conditions. [#29205](https://github.com/ClickHouse/ClickHouse/pull/29205) ([Kseniia Sumarokova](https://github.com/kssenii)). In the meantime, the `web` disk type (static dataset hosted on a web server) is graduated from being experimental to be production ready. @@ -260,7 +260,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * If query has multiple quantile aggregate functions with the same arguments but different level parameter, they will be fused together and executed in one pass if the setting `optimize_syntax_fuse_functions` is enabled. [#26657](https://github.com/ClickHouse/ClickHouse/pull/26657) ([hexiaoting](https://github.com/hexiaoting)). * Now min-max aggregation over the first expression of primary key is optimized by projection. This is for [#329](https://github.com/ClickHouse/ClickHouse/issues/329). [#29918](https://github.com/ClickHouse/ClickHouse/pull/29918) ([Amos Bird](https://github.com/amosbird)). -#### Experimental Feature {#experimental-feature-1} +#### Experimental feature {#experimental-feature-1} * Add ability to change nodes configuration (in `.xml` file) for ClickHouse Keeper. [#30372](https://github.com/ClickHouse/ClickHouse/pull/30372) ([alesapin](https://github.com/alesapin)). * Add `sparkbar` aggregate function. This closes [#26175](https://github.com/ClickHouse/ClickHouse/issues/26175). [#27481](https://github.com/ClickHouse/ClickHouse/pull/27481) ([小路](https://github.com/nicelulu)). Note: there is one flaw in this function, the behaviour will be changed in future releases. @@ -346,7 +346,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Fix shutdown of `AccessControlManager` to fix flaky test. [#29951](https://github.com/ClickHouse/ClickHouse/pull/29951) ([Vitaly Baranov](https://github.com/vitlibar)). * Fix failed assertion in reading from `HDFS`. Update libhdfs3 library to be able to run in tests in debug. Closes [#29251](https://github.com/ClickHouse/ClickHouse/issues/29251). Closes [#27814](https://github.com/ClickHouse/ClickHouse/issues/27814). [#29276](https://github.com/ClickHouse/ClickHouse/pull/29276) ([Kseniia Sumarokova](https://github.com/kssenii)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-1} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-1} * Add support for FreeBSD builds for Aarch64 machines. [#29952](https://github.com/ClickHouse/ClickHouse/pull/29952) ([MikaelUrankar](https://github.com/MikaelUrankar)). * Recursive submodules are no longer needed for ClickHouse. [#30315](https://github.com/ClickHouse/ClickHouse/pull/30315) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -364,7 +364,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Add time dependency for clickhouse-server unit (systemd and sysvinit init). [#28891](https://github.com/ClickHouse/ClickHouse/pull/28891) ([Azat Khuzhin](https://github.com/azat)). * Reload stacktrace cache when symbol is reloaded. [#28137](https://github.com/ClickHouse/ClickHouse/pull/28137) ([Amos Bird](https://github.com/amosbird)). -#### Bug Fix {#bug-fix} +#### Bug fix {#bug-fix} * Functions for case-insensitive search in UTF-8 strings like `positionCaseInsensitiveUTF8` and `countSubstringsCaseInsensitiveUTF8` might find substrings that actually does not match in very rare cases, it's fixed. [#30663](https://github.com/ClickHouse/ClickHouse/pull/30663) ([tavplubix](https://github.com/tavplubix)). * Fix reading from empty file on encrypted disk. [#30494](https://github.com/ClickHouse/ClickHouse/pull/30494) ([Vitaly Baranov](https://github.com/vitlibar)). @@ -451,11 +451,11 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor ### ClickHouse release v21.10, 2021-10-16 {#clickhouse-release-v2110-2021-10-16} -#### Backward Incompatible Change {#backward-incompatible-change-2} +#### Backward incompatible change {#backward-incompatible-change-2} * Now the following MergeTree table-level settings: `replicated_max_parallel_sends`, `replicated_max_parallel_sends_for_table`, `replicated_max_parallel_fetches`, `replicated_max_parallel_fetches_for_table` do nothing. They never worked well and were replaced with `max_replicated_fetches_network_bandwidth`, `max_replicated_sends_network_bandwidth` and `background_fetches_pool_size`. [#28404](https://github.com/ClickHouse/ClickHouse/pull/28404) ([alesapin](https://github.com/alesapin)). -#### New Feature {#new-feature-2} +#### New feature {#new-feature-2} * Add feature for creating user-defined functions (UDF) as lambda expressions. Syntax `CREATE FUNCTION {function_name} as ({parameters}) -> {function core}`. Example `CREATE FUNCTION plus_one as (a) -> a + 1`. Authors @Realist007. [#27796](https://github.com/ClickHouse/ClickHouse/pull/27796) ([Maksim Kita](https://github.com/kitaisreal)) [#23978](https://github.com/ClickHouse/ClickHouse/pull/23978) ([Realist007](https://github.com/Realist007)). * Added `Executable` storage engine and `executable` table function. It enables data processing with external scripts in streaming fashion. [#28102](https://github.com/ClickHouse/ClickHouse/pull/28102) ([Maksim Kita](https://github.com/kitaisreal)) ([ruct](https://github.com/ruct)). @@ -471,12 +471,12 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Add function `SHA512`. [#27830](https://github.com/ClickHouse/ClickHouse/pull/27830) ([zhanglistar](https://github.com/zhanglistar)). * Add `log_queries_probability` setting that allows user to write to query_log only a sample of queries. Closes [#16609](https://github.com/ClickHouse/ClickHouse/issues/16609). [#27527](https://github.com/ClickHouse/ClickHouse/pull/27527) ([Nikolay Degterinsky](https://github.com/evillique)). -#### Experimental Feature {#experimental-feature-2} +#### Experimental feature {#experimental-feature-2} * `web` type of disks to store readonly tables on web server in form of static files. See [#23982](https://github.com/ClickHouse/ClickHouse/issues/23982). [#25251](https://github.com/ClickHouse/ClickHouse/pull/25251) ([Kseniia Sumarokova](https://github.com/kssenii)). This is mostly needed to faciliate testing of operation on shared storage and for easy importing of datasets. Not recommended to use before release 21.11. * Added new commands `BACKUP` and `RESTORE`. [#21945](https://github.com/ClickHouse/ClickHouse/pull/21945) ([Vitaly Baranov](https://github.com/vitlibar)). This is under development and not intended to be used in current version. -#### Performance Improvement {#performance-improvement-2} +#### Performance improvement {#performance-improvement-2} * Speed up `sumIf` and `countIf` aggregation functions. [#28272](https://github.com/ClickHouse/ClickHouse/pull/28272) ([Raúl Marín](https://github.com/Algunenano)). * Create virtual projection for `minmax` indices. Now, when `allow_experimental_projection_optimization` is enabled, queries will use minmax index instead of reading the data when possible. [#26286](https://github.com/ClickHouse/ClickHouse/pull/26286) ([Amos Bird](https://github.com/amosbird)). @@ -518,7 +518,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Improve `ALTER ... MATERIALIZE TTL` by recalculating metadata only without actual TTL action. [#27019](https://github.com/ClickHouse/ClickHouse/pull/27019) ([lthaooo](https://github.com/lthaooo)). * Allow reading the list of custom top level domains without a new line at EOF. [#28213](https://github.com/ClickHouse/ClickHouse/pull/28213) ([Azat Khuzhin](https://github.com/azat)). -#### Bug Fix {#bug-fix-1} +#### Bug fix {#bug-fix-1} * Fix cases, when reading compressed data from `carbon-clickhouse` fails with 'attempt to read after end of file'. Closes [#26149](https://github.com/ClickHouse/ClickHouse/issues/26149). [#28150](https://github.com/ClickHouse/ClickHouse/pull/28150) ([FArthur-cmd](https://github.com/FArthur-cmd)). * Fix checking access grants when executing `GRANT WITH REPLACE` statement with `ON CLUSTER` clause. This PR improves fix [#27001](https://github.com/ClickHouse/ClickHouse/pull/27701). [#27983](https://github.com/ClickHouse/ClickHouse/pull/27983) ([Vitaly Baranov](https://github.com/vitlibar)). @@ -542,7 +542,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Fix handling null value with type of `Nullable(String)` in function `JSONExtract`. This fixes [#27929](https://github.com/ClickHouse/ClickHouse/issues/27929) and [#27930](https://github.com/ClickHouse/ClickHouse/issues/27930). This was introduced in https://github.com/ClickHouse/ClickHouse/pull/25452 . [#27939](https://github.com/ClickHouse/ClickHouse/pull/27939) ([Amos Bird](https://github.com/amosbird)). * Multiple fixes for the new `clickhouse-keeper` tool. Fix a rare bug in `clickhouse-keeper` when the client can receive a watch response before request-response. [#28197](https://github.com/ClickHouse/ClickHouse/pull/28197) ([alesapin](https://github.com/alesapin)). Fix incorrect behavior in `clickhouse-keeper` when list watches (`getChildren`) triggered with `set` requests for children. [#28190](https://github.com/ClickHouse/ClickHouse/pull/28190) ([alesapin](https://github.com/alesapin)). Fix rare case when changes of `clickhouse-keeper` settings may lead to lost logs and server hung. [#28360](https://github.com/ClickHouse/ClickHouse/pull/28360) ([alesapin](https://github.com/alesapin)). Fix bug in `clickhouse-keeper` which can lead to endless logs when `rotate_logs_interval` decreased. [#28152](https://github.com/ClickHouse/ClickHouse/pull/28152) ([alesapin](https://github.com/alesapin)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-2} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-2} * Enable Thread Fuzzer in Stress Test. Thread Fuzzer is ClickHouse feature that allows to test more permutations of thread scheduling and discover more potential issues. This closes [#9813](https://github.com/ClickHouse/ClickHouse/issues/9813). This closes [#9814](https://github.com/ClickHouse/ClickHouse/issues/9814). This closes [#9515](https://github.com/ClickHouse/ClickHouse/issues/9515). This closes [#9516](https://github.com/ClickHouse/ClickHouse/issues/9516). [#27538](https://github.com/ClickHouse/ClickHouse/pull/27538) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Add new log level `test` for testing environments. It is even more verbose than the default `trace`. [#28559](https://github.com/ClickHouse/ClickHouse/pull/28559) ([alesapin](https://github.com/alesapin)). @@ -551,7 +551,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor ### ClickHouse release v21.9, 2021-09-09 {#clickhouse-release-v219-2021-09-09} -#### Backward Incompatible Change {#backward-incompatible-change-3} +#### Backward incompatible change {#backward-incompatible-change-3} * Do not output trailing zeros in text representation of `Decimal` types. Example: `1.23` will be printed instead of `1.230000` for decimal with scale 6. This closes [#15794](https://github.com/ClickHouse/ClickHouse/issues/15794). It may introduce slight incompatibility if your applications somehow relied on the trailing zeros. Serialization in output formats can be controlled with the setting `output_format_decimal_trailing_zeros`. Implementation of `toString` and casting to String is changed unconditionally. [#27680](https://github.com/ClickHouse/ClickHouse/pull/27680) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Do not allow to apply parametric aggregate function with `-Merge` combinator to aggregate function state if state was produced by aggregate function with different parameters. For example, state of `fooState(42)(x)` cannot be finalized with `fooMerge(s)` or `fooMerge(123)(s)`, parameters must be specified explicitly like `fooMerge(42)(s)` and must be equal. It does not affect some special aggregate functions like `quantile` and `sequence*` that use parameters for finalization only. [#26847](https://github.com/ClickHouse/ClickHouse/pull/26847) ([tavplubix](https://github.com/tavplubix)). @@ -561,7 +561,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Introduce syntax for here documents. Example `SELECT $doc$ VALUE $doc$`. [#26671](https://github.com/ClickHouse/ClickHouse/pull/26671) ([Maksim Kita](https://github.com/kitaisreal)). This change is backward incompatible if in query there are identifiers that contain `$` [#28768](https://github.com/ClickHouse/ClickHouse/issues/28768). * Now indices can handle Nullable types, including `isNull` and `isNotNull`. [#12433](https://github.com/ClickHouse/ClickHouse/pull/12433) and [#12455](https://github.com/ClickHouse/ClickHouse/pull/12455) ([Amos Bird](https://github.com/amosbird)) and [#27250](https://github.com/ClickHouse/ClickHouse/pull/27250) ([Azat Khuzhin](https://github.com/azat)). But this was done with on-disk format changes, and even though new server can read old data, old server cannot. Also, in case you have `MINMAX` data skipping indices, you may get `Data after mutation/merge is not byte-identical` error, since new index will have `.idx2` extension while before it was `.idx`. That said, that you should not delay updating all existing replicas, in this case, otherwise, if old replica (<21.9) will download data from new replica with 21.9+ it will not be able to apply index for downloaded part. -#### New Feature {#new-feature-3} +#### New feature {#new-feature-3} * Implementation of short circuit function evaluation, closes [#12587](https://github.com/ClickHouse/ClickHouse/issues/12587). Add settings `short_circuit_function_evaluation` to configure short circuit function evaluation. [#23367](https://github.com/ClickHouse/ClickHouse/pull/23367) ([Kruglov Pavel](https://github.com/Avogar)). * Add support for INTERSECT, EXCEPT, ANY, ALL operators. [#24757](https://github.com/ClickHouse/ClickHouse/pull/24757) ([Kirill Ershov](https://github.com/zdikov)). ([Kseniia Sumarokova](https://github.com/kssenii)). @@ -594,12 +594,12 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Allow setting default database for users. [#25268](https://github.com/ClickHouse/ClickHouse/issues/25268). [#25687](https://github.com/ClickHouse/ClickHouse/pull/25687) ([kevin wan](https://github.com/MaxWk)). * Add an optional parameter to `MongoDB` engine to accept connection string options and support SSL connection. Closes [#21189](https://github.com/ClickHouse/ClickHouse/issues/21189). Closes [#21041](https://github.com/ClickHouse/ClickHouse/issues/21041). [#22045](https://github.com/ClickHouse/ClickHouse/pull/22045) ([Omar Bazaraa](https://github.com/OmarBazaraa)). -#### Experimental Feature {#experimental-feature-3} +#### Experimental feature {#experimental-feature-3} * Added a compression codec `AES_128_GCM_SIV` which encrypts columns instead of compressing them. [#19896](https://github.com/ClickHouse/ClickHouse/pull/19896) ([PHO](https://github.com/depressed-pho)). Will be rewritten, do not use. * Rename `MaterializeMySQL` to `MaterializedMySQL`. [#26822](https://github.com/ClickHouse/ClickHouse/pull/26822) ([tavplubix](https://github.com/tavplubix)). -#### Performance Improvement {#performance-improvement-3} +#### Performance improvement {#performance-improvement-3} * Improve the performance of fast queries when `max_execution_time = 0` by reducing the number of `clock_gettime` system calls. [#27325](https://github.com/ClickHouse/ClickHouse/pull/27325) ([filimonov](https://github.com/filimonov)). * Specialize date time related comparison to achieve better performance. This fixes [#27083](https://github.com/ClickHouse/ClickHouse/issues/27083) . [#27122](https://github.com/ClickHouse/ClickHouse/pull/27122) ([Amos Bird](https://github.com/amosbird)). @@ -684,7 +684,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Support `LowCardinality` and `FixedString` keys/values for `Map` type. [#21543](https://github.com/ClickHouse/ClickHouse/pull/21543) ([hexiaoting](https://github.com/hexiaoting)). * Enable reloading of local disk config. [#19526](https://github.com/ClickHouse/ClickHouse/pull/19526) ([taiyang-li](https://github.com/taiyang-li)). -#### Bug Fix {#bug-fix-2} +#### Bug fix {#bug-fix-2} * Fix a couple of bugs that may cause replicas to diverge. [#27808](https://github.com/ClickHouse/ClickHouse/pull/27808) ([tavplubix](https://github.com/tavplubix)). * Fix a rare bug in `DROP PART` which can lead to the error `Unexpected merged part intersects drop range`. [#27807](https://github.com/ClickHouse/ClickHouse/pull/27807) ([alesapin](https://github.com/alesapin)). @@ -748,7 +748,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Fixed `Not found column ...` and `Missing column ...` errors when selecting from `MaterializeMySQL`. Fixes [#23708](https://github.com/ClickHouse/ClickHouse/issues/23708), [#24830](https://github.com/ClickHouse/ClickHouse/issues/24830), [#25794](https://github.com/ClickHouse/ClickHouse/issues/25794). [#25822](https://github.com/ClickHouse/ClickHouse/pull/25822) ([tavplubix](https://github.com/tavplubix)). * Fix `optimize_skip_unused_shards_rewrite_in` for non-UInt64 types (may select incorrect shards eventually or throw `Cannot infer type of an empty tuple` or `Function tuple requires at least one argument`). [#25798](https://github.com/ClickHouse/ClickHouse/pull/25798) ([Azat Khuzhin](https://github.com/azat)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-3} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-3} * Now we ran stateful and stateless tests in random timezones. Fixes [#12439](https://github.com/ClickHouse/ClickHouse/issues/12439). Reading String as DateTime and writing DateTime as String in Protobuf format now respect timezone. Reading UInt16 as DateTime in Arrow and Parquet formats now treat it as Date and then converts to DateTime with respect to DateTime's timezone, because Date is serialized in Arrow and Parquet as UInt16. GraphiteMergeTree now respect time zone for rounding of times. Fixes [#5098](https://github.com/ClickHouse/ClickHouse/issues/5098). Author: @alexey-milovidov. [#15408](https://github.com/ClickHouse/ClickHouse/pull/15408) ([alesapin](https://github.com/alesapin)). * `clickhouse-test` supports SQL tests with [Jinja2](https://jinja.palletsprojects.com/en/3.0.x/templates/#synopsis) templates. [#26579](https://github.com/ClickHouse/ClickHouse/pull/26579) ([Vladimir C](https://github.com/vdimir)). @@ -759,10 +759,10 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor ### ClickHouse release v21.8, 2021-08-12 {#clickhouse-release-v218-2021-08-12} -#### Upgrade Notes {#upgrade-notes} +#### Upgrade notes {#upgrade-notes} * New version is using `Map` data type for system logs tables (`system.query_log`, `system.query_thread_log`, `system.processes`, `system.opentelemetry_span_log`). These tables will be auto-created with new data types. Virtual columns are created to support old queries. Closes [#18698](https://github.com/ClickHouse/ClickHouse/issues/18698). [#23934](https://github.com/ClickHouse/ClickHouse/pull/23934), [#25773](https://github.com/ClickHouse/ClickHouse/pull/25773) ([hexiaoting](https://github.com/hexiaoting), [sundy-li](https://github.com/sundy-li), [Maksim Kita](https://github.com/kitaisreal)). If you want to *downgrade* from version 21.8 to older versions, you will need to cleanup system tables with logs manually. Look at `/var/lib/clickhouse/data/system/*_log`. -#### New Features {#new-features} +#### New features {#new-features} * Add support for a part of SQL/JSON standard. [#24148](https://github.com/ClickHouse/ClickHouse/pull/24148) ([l1tsolaiki](https://github.com/l1tsolaiki), [Kseniia Sumarokova](https://github.com/kssenii)). * Collect common system metrics (in `system.asynchronous_metrics` and `system.asynchronous_metric_log`) on CPU usage, disk usage, memory usage, IO, network, files, load average, CPU frequencies, thermal sensors, EDAC counters, system uptime; also added metrics about the scheduling jitter and the time spent collecting the metrics. It works similar to `atop` in ClickHouse and allows access to monitoring data even if you have no additional tools installed. Close [#9430](https://github.com/ClickHouse/ClickHouse/issues/9430). [#24416](https://github.com/ClickHouse/ClickHouse/pull/24416) ([alexey-milovidov](https://github.com/alexey-milovidov), [Yegor Levankov](https://github.com/elevankoff)). @@ -776,7 +776,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Add an ability to reset a custom setting to default and remove it from the table's metadata. It allows rolling back the change without knowing the system/config's default. Closes [#14449](https://github.com/ClickHouse/ClickHouse/issues/14449). [#17769](https://github.com/ClickHouse/ClickHouse/pull/17769) ([xjewer](https://github.com/xjewer)). * Render pipelines as graphs in Web UI if `EXPLAIN PIPELINE graph = 1` query is submitted. [#26067](https://github.com/ClickHouse/ClickHouse/pull/26067) ([alexey-milovidov](https://github.com/alexey-milovidov)). -#### Performance Improvements {#performance-improvements} +#### Performance improvements {#performance-improvements} * Compile aggregate functions. Use option `compile_aggregate_expressions` to enable it. [#24789](https://github.com/ClickHouse/ClickHouse/pull/24789) ([Maksim Kita](https://github.com/kitaisreal)). * Improve latency of short queries that require reading from tables with many columns. [#26371](https://github.com/ClickHouse/ClickHouse/pull/26371) ([Anton Popov](https://github.com/CurtizJ)). @@ -817,7 +817,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Web UI: if the value looks like a URL, automatically generate a link. [#25965](https://github.com/ClickHouse/ClickHouse/pull/25965) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Make `sudo service clickhouse-server start` to work on systems with `systemd` like Centos 8. Close [#14298](https://github.com/ClickHouse/ClickHouse/issues/14298). Close [#17799](https://github.com/ClickHouse/ClickHouse/issues/17799). [#25921](https://github.com/ClickHouse/ClickHouse/pull/25921) ([alexey-milovidov](https://github.com/alexey-milovidov)). -#### Bug Fixes {#bug-fixes-1} +#### Bug fixes {#bug-fixes-1} * Fix incorrect `SET ROLE` in some cases. [#26707](https://github.com/ClickHouse/ClickHouse/pull/26707) ([Vitaly Baranov](https://github.com/vitlibar)). * Fix potential `nullptr` dereference in window functions. Fix [#25276](https://github.com/ClickHouse/ClickHouse/issues/25276). [#26668](https://github.com/ClickHouse/ClickHouse/pull/26668) ([Alexander Kuzmenkov](https://github.com/akuzm)). @@ -860,12 +860,12 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor ### ClickHouse release v21.7, 2021-07-09 {#clickhouse-release-v217-2021-07-09} -#### Backward Incompatible Change {#backward-incompatible-change-4} +#### Backward incompatible change {#backward-incompatible-change-4} * Improved performance of queries with explicitly defined large sets. Added compatibility setting `legacy_column_name_of_tuple_literal`. It makes sense to set it to `true`, while doing rolling update of cluster from version lower than 21.7 to any higher version. Otherwise distributed queries with explicitly defined sets at `IN` clause may fail during update. [#25371](https://github.com/ClickHouse/ClickHouse/pull/25371) ([Anton Popov](https://github.com/CurtizJ)). * Forward/backward incompatible change of maximum buffer size in clickhouse-keeper (an experimental alternative to ZooKeeper). Better to do it now (before production), than later. [#25421](https://github.com/ClickHouse/ClickHouse/pull/25421) ([alesapin](https://github.com/alesapin)). -#### New Feature {#new-feature-4} +#### New feature {#new-feature-4} * Support configuration in YAML format as alternative to XML. This closes [#3607](https://github.com/ClickHouse/ClickHouse/issues/3607). [#21858](https://github.com/ClickHouse/ClickHouse/pull/21858) ([BoloniniD](https://github.com/BoloniniD)). * Provides a way to restore replicated table when the data is (possibly) present, but the ZooKeeper metadata is lost. Resolves [#13458](https://github.com/ClickHouse/ClickHouse/issues/13458). [#13652](https://github.com/ClickHouse/ClickHouse/pull/13652) ([Mike Kot](https://github.com/myrrc)). @@ -881,12 +881,12 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Add `quantileBFloat16` aggregate function as well as the corresponding `quantilesBFloat16` and `medianBFloat16`. It is very simple and fast quantile estimator with relative error not more than 0.390625%. This closes [#16641](https://github.com/ClickHouse/ClickHouse/issues/16641). [#23204](https://github.com/ClickHouse/ClickHouse/pull/23204) ([Ivan Novitskiy](https://github.com/RedClusive)). * Implement `sequenceNextNode()` function useful for `flow analysis`. [#19766](https://github.com/ClickHouse/ClickHouse/pull/19766) ([achimbab](https://github.com/achimbab)). -#### Experimental Feature {#experimental-feature-4} +#### Experimental feature {#experimental-feature-4} * Add support for virtual filesystem over HDFS. [#11058](https://github.com/ClickHouse/ClickHouse/pull/11058) ([overshov](https://github.com/overshov)) ([Kseniia Sumarokova](https://github.com/kssenii)). * Now clickhouse-keeper (an experimental alternative to ZooKeeper) supports ZooKeeper-like `digest` ACLs. [#24448](https://github.com/ClickHouse/ClickHouse/pull/24448) ([alesapin](https://github.com/alesapin)). -#### Performance Improvement {#performance-improvement-4} +#### Performance improvement {#performance-improvement-4} * Added optimization that transforms some functions to reading of subcolumns to reduce amount of read data. E.g., statement `col IS NULL` is transformed to reading of subcolumn `col.null`. Optimization can be enabled by setting `optimize_functions_to_subcolumns` which is currently off by default. [#24406](https://github.com/ClickHouse/ClickHouse/pull/24406) ([Anton Popov](https://github.com/CurtizJ)). * Rewrite more columns to possible alias expressions. This may enable better optimization, such as projections. [#24405](https://github.com/ClickHouse/ClickHouse/pull/24405) ([Amos Bird](https://github.com/amosbird)). @@ -942,7 +942,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Enable reading of subcolumns (e.g. components of Tuples) for distributed tables. [#24472](https://github.com/ClickHouse/ClickHouse/pull/24472) ([Anton Popov](https://github.com/CurtizJ)). * A feature for MySQL compatibility protocol: make `user` function to return correct output. Closes [#25697](https://github.com/ClickHouse/ClickHouse/pull/25697). [#25697](https://github.com/ClickHouse/ClickHouse/pull/25697) ([sundyli](https://github.com/sundy-li)). -#### Bug Fix {#bug-fix-3} +#### Bug fix {#bug-fix-3} * Improvement for backward compatibility. Use old modulo function version when used in partition key. Closes [#23508](https://github.com/ClickHouse/ClickHouse/issues/23508). [#24157](https://github.com/ClickHouse/ClickHouse/pull/24157) ([Kseniia Sumarokova](https://github.com/kssenii)). * Fix extremely rare bug on low-memory servers which can lead to the inability to perform merges without restart. Possibly fixes [#24603](https://github.com/ClickHouse/ClickHouse/issues/24603). [#24872](https://github.com/ClickHouse/ClickHouse/pull/24872) ([alesapin](https://github.com/alesapin)). @@ -1001,7 +1001,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Make column LowCardinality property in JOIN output the same as in the input, close [#23351](https://github.com/ClickHouse/ClickHouse/issues/23351), close [#20315](https://github.com/ClickHouse/ClickHouse/issues/20315). [#24061](https://github.com/ClickHouse/ClickHouse/pull/24061) ([Vladimir](https://github.com/vdimir)). * A fix for Kafka tables. Fix the bug in failover behavior when Engine = Kafka was not able to start consumption if the same consumer had an empty assignment previously. Closes [#21118](https://github.com/ClickHouse/ClickHouse/issues/21118). [#21267](https://github.com/ClickHouse/ClickHouse/pull/21267) ([filimonov](https://github.com/filimonov)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-4} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-4} * Add `darwin-aarch64` (Mac M1 / Apple Silicon) builds in CI [#25560](https://github.com/ClickHouse/ClickHouse/pull/25560) ([Ivan](https://github.com/abyss7)) and put the links to the docs and website ([alexey-milovidov](https://github.com/alexey-milovidov)). * Adds cross-platform embedding of binary resources into executables. It works on Illumos. [#25146](https://github.com/ClickHouse/ClickHouse/pull/25146) ([bnaecker](https://github.com/bnaecker)). @@ -1015,16 +1015,16 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor ### ClickHouse release 21.6, 2021-06-05 {#clickhouse-release-216-2021-06-05} -#### Backward Incompatible Change {#backward-incompatible-change-5} +#### Backward incompatible change {#backward-incompatible-change-5} * uniqState / uniqHLL12State / uniqCombinedState / uniqCombined64State produce incompatible states with `UUID` type. [#33607](https://github.com/ClickHouse/ClickHouse/issues/33607). -#### Upgrade Notes {#upgrade-notes-1} +#### Upgrade notes {#upgrade-notes-1} * `zstd` compression library is updated to v1.5.0. You may get messages about "checksum does not match" in replication. These messages are expected due to update of compression algorithm and you can ignore them. These messages are informational and do not indicate any kinds of undesired behaviour. * The setting `compile_expressions` is enabled by default. Although it has been heavily tested on variety of scenarios, if you find some undesired behaviour on your servers, you can try turning this setting off. * Values of `UUID` type cannot be compared with integer. For example, instead of writing `uuid != 0` type `uuid != '00000000-0000-0000-0000-000000000000'`. -#### New Feature {#new-feature-5} +#### New feature {#new-feature-5} * Add Postgres-like cast operator (`::`). E.g.: `[1, 2]::Array(UInt8)`, `0.1::Decimal(4, 4)`, `number::UInt16`. [#23871](https://github.com/ClickHouse/ClickHouse/pull/23871) ([Anton Popov](https://github.com/CurtizJ)). * Make big integers production ready. Add support for `UInt128` data type. Fix known issues with the `Decimal256` data type. Support big integers in dictionaries. Support `gcd`/`lcm` functions for big integers. Support big integers in array search and conditional functions. Support `LowCardinality(UUID)`. Support big integers in `generateRandom` table function and `clickhouse-obfuscator`. Fix error with returning `UUID` from scalar subqueries. This fixes [#7834](https://github.com/ClickHouse/ClickHouse/issues/7834). This fixes [#23936](https://github.com/ClickHouse/ClickHouse/issues/23936). This fixes [#4176](https://github.com/ClickHouse/ClickHouse/issues/4176). This fixes [#24018](https://github.com/ClickHouse/ClickHouse/issues/24018). Backward incompatible change: values of `UUID` type cannot be compared with integer. For example, instead of writing `uuid != 0` type `uuid != '00000000-0000-0000-0000-000000000000'`. [#23631](https://github.com/ClickHouse/ClickHouse/pull/23631) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -1046,11 +1046,11 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Added less secure IMDS credentials provider for S3 which works under docker correctly. [#21852](https://github.com/ClickHouse/ClickHouse/pull/21852) ([Vladimir Chebotarev](https://github.com/excitoon)). * Add back `indexHint` function. This is for [#21238](https://github.com/ClickHouse/ClickHouse/issues/21238). This reverts [#9542](https://github.com/ClickHouse/ClickHouse/pull/9542). This fixes [#9540](https://github.com/ClickHouse/ClickHouse/issues/9540). [#21304](https://github.com/ClickHouse/ClickHouse/pull/21304) ([Amos Bird](https://github.com/amosbird)). -#### Experimental Feature {#experimental-feature-5} +#### Experimental feature {#experimental-feature-5} * Add `PROJECTION` support for `MergeTree*` tables. [#20202](https://github.com/ClickHouse/ClickHouse/pull/20202) ([Amos Bird](https://github.com/amosbird)). -#### Performance Improvement {#performance-improvement-5} +#### Performance improvement {#performance-improvement-5} * Enable `compile_expressions` setting by default. When this setting enabled, compositions of simple functions and operators will be compiled to native code with LLVM at runtime. [#8482](https://github.com/ClickHouse/ClickHouse/pull/8482) ([Maksim Kita](https://github.com/kitaisreal), [alexey-milovidov](https://github.com/alexey-milovidov)). Note: if you feel in trouble, turn this option off. * Update `re2` library. Performance of regular expressions matching is improved. Also this PR adds compatibility with gcc-11. [#24196](https://github.com/ClickHouse/ClickHouse/pull/24196) ([Raúl Marín](https://github.com/Algunenano)). @@ -1097,7 +1097,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Fix usage of tuples in `CREATE .. AS SELECT` queries. [#24464](https://github.com/ClickHouse/ClickHouse/pull/24464) ([Anton Popov](https://github.com/CurtizJ)). * Support for `Parquet` format in `Kafka` tables. [#23412](https://github.com/ClickHouse/ClickHouse/pull/23412) ([Chao Ma](https://github.com/godliness)). -#### Bug Fix {#bug-fix-4} +#### Bug fix {#bug-fix-4} * Use old modulo function version when used in partition key and primary key. Closes [#23508](https://github.com/ClickHouse/ClickHouse/issues/23508). [#24157](https://github.com/ClickHouse/ClickHouse/pull/24157) ([Kseniia Sumarokova](https://github.com/kssenii)). It was a source of backward incompatibility in previous releases. * Fixed the behavior when query `SYSTEM RESTART REPLICA` or `SYSTEM SYNC REPLICA` is being processed infinitely. This was detected on server with extremely little amount of RAM. [#24457](https://github.com/ClickHouse/ClickHouse/pull/24457) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). @@ -1131,7 +1131,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Add type conversion when the setting `optimize_skip_unused_shards_rewrite_in` is enabled. This fixes MSan report. [#23219](https://github.com/ClickHouse/ClickHouse/pull/23219) ([Azat Khuzhin](https://github.com/azat)). * Add a missing check when updating nested subcolumns, close issue: [#22353](https://github.com/ClickHouse/ClickHouse/issues/22353). [#22503](https://github.com/ClickHouse/ClickHouse/pull/22503) ([hexiaoting](https://github.com/hexiaoting)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-5} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-5} * Support building on Illumos. [#24144](https://github.com/ClickHouse/ClickHouse/pull/24144). Adds support for building on Solaris-derived operating systems. [#23746](https://github.com/ClickHouse/ClickHouse/pull/23746) ([bnaecker](https://github.com/bnaecker)). * Add more benchmarks for hash tables, including the Swiss Table from Google (that appeared to be slower than ClickHouse hash map in our specific usage scenario). [#24111](https://github.com/ClickHouse/ClickHouse/pull/24111) ([Maksim Kita](https://github.com/kitaisreal)). @@ -1143,12 +1143,12 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor ## ClickHouse release 21.5, 2021-05-20 {#clickhouse-release-215-2021-05-20} -#### Backward Incompatible Change {#backward-incompatible-change-6} +#### Backward incompatible change {#backward-incompatible-change-6} * Change comparison of integers and floating point numbers when integer is not exactly representable in the floating point data type. In new version comparison will return false as the rounding error will occur. Example: `9223372036854775808.0 != 9223372036854775808`, because the number `9223372036854775808` is not representable as floating point number exactly (and `9223372036854775808.0` is rounded to `9223372036854776000.0`). But in previous version the comparison will return as the numbers are equal, because if the floating point number `9223372036854776000.0` get converted back to UInt64, it will yield `9223372036854775808`. For the reference, the Python programming language also treats these numbers as equal. But this behaviour was dependend on CPU model (different results on AMD64 and AArch64 for some out-of-range numbers), so we make the comparison more precise. It will treat int and float numbers equal only if int is represented in floating point type exactly. [#22595](https://github.com/ClickHouse/ClickHouse/pull/22595) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Remove support for `argMin` and `argMax` for single `Tuple` argument. The code was not memory-safe. The feature was added by mistake and it is confusing for people. These functions can be reintroduced under different names later. This fixes [#22384](https://github.com/ClickHouse/ClickHouse/issues/22384) and reverts [#17359](https://github.com/ClickHouse/ClickHouse/issues/17359). [#23393](https://github.com/ClickHouse/ClickHouse/pull/23393) ([alexey-milovidov](https://github.com/alexey-milovidov)). -#### New Feature {#new-feature-6} +#### New feature {#new-feature-6} * Added functions `dictGetChildren(dictionary, key)`, `dictGetDescendants(dictionary, key, level)`. Function `dictGetChildren` return all children as an array if indexes. It is a inverse transformation for `dictGetHierarchy`. Function `dictGetDescendants` return all descendants as if `dictGetChildren` was applied `level` times recursively. Zero `level` value is equivalent to infinity. Improved performance of `dictGetHierarchy`, `dictIsIn` functions. Closes [#14656](https://github.com/ClickHouse/ClickHouse/issues/14656). [#22096](https://github.com/ClickHouse/ClickHouse/pull/22096) ([Maksim Kita](https://github.com/kitaisreal)). * Added function `dictGetOrNull`. It works like `dictGet`, but return `Null` in case key was not found in dictionary. Closes [#22375](https://github.com/ClickHouse/ClickHouse/issues/22375). [#22413](https://github.com/ClickHouse/ClickHouse/pull/22413) ([Maksim Kita](https://github.com/kitaisreal)). @@ -1157,7 +1157,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Added `ALTER TABLE ... FETCH PART ...` query. It's similar to `FETCH PARTITION`, but fetches only one part. [#22706](https://github.com/ClickHouse/ClickHouse/pull/22706) ([turbo jason](https://github.com/songenjie)). * Added a setting `max_distributed_depth` that limits the depth of recursive queries to `Distributed` tables. Closes [#20229](https://github.com/ClickHouse/ClickHouse/issues/20229). [#21942](https://github.com/ClickHouse/ClickHouse/pull/21942) ([flynn](https://github.com/ucasFL)). -#### Performance Improvement {#performance-improvement-6} +#### Performance improvement {#performance-improvement-6} * Improved performance of `intDiv` by dynamic dispatch for AVX2. This closes [#22314](https://github.com/ClickHouse/ClickHouse/issues/22314). [#23000](https://github.com/ClickHouse/ClickHouse/pull/23000) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Improved performance of reading from `ArrowStream` input format for sources other then local file (e.g. URL). [#22673](https://github.com/ClickHouse/ClickHouse/pull/22673) ([nvartolomei](https://github.com/nvartolomei)). @@ -1209,7 +1209,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Fixed approximate total rows accounting for reverse reading from MergeTree. [#22726](https://github.com/ClickHouse/ClickHouse/pull/22726) ([Azat Khuzhin](https://github.com/azat)). * Fix the case when it was possible to configure dictionary with clickhouse source that was looking to itself that leads to infinite loop. Closes [#14314](https://github.com/ClickHouse/ClickHouse/issues/14314). [#22479](https://github.com/ClickHouse/ClickHouse/pull/22479) ([Maksim Kita](https://github.com/kitaisreal)). -#### Bug Fix {#bug-fix-5} +#### Bug fix {#bug-fix-5} * Multiple fixes for hedged requests. Fixed an error `Can't initialize pipeline with empty pipe` for queries with `GLOBAL IN/JOIN` when the setting `use_hedged_requests` is enabled. Fixes [#23431](https://github.com/ClickHouse/ClickHouse/issues/23431). [#23805](https://github.com/ClickHouse/ClickHouse/pull/23805) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). Fixed a race condition in hedged connections which leads to crash. This fixes [#22161](https://github.com/ClickHouse/ClickHouse/issues/22161). [#22443](https://github.com/ClickHouse/ClickHouse/pull/22443) ([Kruglov Pavel](https://github.com/Avogar)). Fix possible crash in case if `unknown packet` was received from remote query (with `async_socket_for_remote` enabled). Fixes [#21167](https://github.com/ClickHouse/ClickHouse/issues/21167). [#23309](https://github.com/ClickHouse/ClickHouse/pull/23309) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fixed the behavior when disabling `input_format_with_names_use_header ` setting discards all the input with CSVWithNames format. This fixes [#22406](https://github.com/ClickHouse/ClickHouse/issues/22406). [#23202](https://github.com/ClickHouse/ClickHouse/pull/23202) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). @@ -1261,7 +1261,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Fixed name clashes in pushdown optimization. It caused incorrect `WHERE` filtration after FULL JOIN. Close [#20497](https://github.com/ClickHouse/ClickHouse/issues/20497). [#20622](https://github.com/ClickHouse/ClickHouse/pull/20622) ([Vladimir](https://github.com/vdimir)). * Fixed very rare bug when quorum insert with `quorum_parallel=1` is not really "quorum" because of deduplication. [#18215](https://github.com/ClickHouse/ClickHouse/pull/18215) ([filimonov](https://github.com/filimonov) - reported, [alesapin](https://github.com/alesapin) - fixed). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-6} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-6} * Run stateless tests in parallel in CI. [#22300](https://github.com/ClickHouse/ClickHouse/pull/22300) ([alesapin](https://github.com/alesapin)). * Simplify debian packages. This fixes [#21698](https://github.com/ClickHouse/ClickHouse/issues/21698). [#22976](https://github.com/ClickHouse/ClickHouse/pull/22976) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -1283,7 +1283,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor ### ClickHouse release 21.4.1 2021-04-12 {#clickhouse-release-2141-2021-04-12} -#### Backward Incompatible Change {#backward-incompatible-change-7} +#### Backward incompatible change {#backward-incompatible-change-7} * The `toStartOfIntervalFunction` will align hour intervals to the midnight (in previous versions they were aligned to the start of unix epoch). For example, `toStartOfInterval(x, INTERVAL 11 HOUR)` will split every day into three intervals: `00:00:00..10:59:59`, `11:00:00..21:59:59` and `22:00:00..23:59:59`. This behaviour is more suited for practical needs. This closes [#9510](https://github.com/ClickHouse/ClickHouse/issues/9510). [#22060](https://github.com/ClickHouse/ClickHouse/pull/22060) ([alexey-milovidov](https://github.com/alexey-milovidov)). * `Age` and `Precision` in graphite rollup configs should increase from retention to retention. Now it's checked and the wrong config raises an exception. [#21496](https://github.com/ClickHouse/ClickHouse/pull/21496) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). @@ -1294,7 +1294,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * It's not possible to rollback to older ClickHouse version after executing `ALTER ... ATTACH` query in new version as the old servers would fail to pass the `ATTACH_PART` entry in the replicated log. * In this version, empty `` will block all access to remote hosts while in previous versions it did nothing. If you want to keep old behaviour and you have empty `remote_url_allow_hosts` element in configuration file, remove it. [#20058](https://github.com/ClickHouse/ClickHouse/pull/20058) ([Vladimir Chebotarev](https://github.com/excitoon)). -#### New Feature {#new-feature-7} +#### New feature {#new-feature-7} * Extended range of `DateTime64` to support dates from year 1925 to 2283. Improved support of `DateTime` around zero date (`1970-01-01`). [#9404](https://github.com/ClickHouse/ClickHouse/pull/9404) ([alexey-milovidov](https://github.com/alexey-milovidov), [Vasily Nemkov](https://github.com/Enmk)). Not every time and date functions are working for extended range of dates. * Added support of Kerberos authentication for preconfigured users and HTTP requests (GSS-SPNEGO). [#14995](https://github.com/ClickHouse/ClickHouse/pull/14995) ([Denis Glazachev](https://github.com/traceon)). @@ -1315,13 +1315,13 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Added new SQL command `ALTER TABLE 'table_name' UNFREEZE [PARTITION 'part_expr'] WITH NAME 'backup_name'`. This command is needed to properly remove 'freezed' partitions from all disks. [#21142](https://github.com/ClickHouse/ClickHouse/pull/21142) ([Pavel Kovalenko](https://github.com/Jokser)). * Supports implicit key type conversion for JOIN. [#19885](https://github.com/ClickHouse/ClickHouse/pull/19885) ([Vladimir](https://github.com/vdimir)). -#### Experimental Feature {#experimental-feature-6} +#### Experimental feature {#experimental-feature-6} * Support `RANGE OFFSET` frame (for window functions) for floating point types. Implement `lagInFrame`/`leadInFrame` window functions, which are analogous to `lag`/`lead`, but respect the window frame. They are identical when the frame is `between unbounded preceding and unbounded following`. This closes [#5485](https://github.com/ClickHouse/ClickHouse/issues/5485). [#21895](https://github.com/ClickHouse/ClickHouse/pull/21895) ([Alexander Kuzmenkov](https://github.com/akuzm)). * Zero-copy replication for `ReplicatedMergeTree` over S3 storage. [#16240](https://github.com/ClickHouse/ClickHouse/pull/16240) ([ianton-ru](https://github.com/ianton-ru)). * Added possibility to migrate existing S3 disk to the schema with backup-restore capabilities. [#22070](https://github.com/ClickHouse/ClickHouse/pull/22070) ([Pavel Kovalenko](https://github.com/Jokser)). -#### Performance Improvement {#performance-improvement-7} +#### Performance improvement {#performance-improvement-7} * Supported parallel formatting in `clickhouse-local` and everywhere else. [#21630](https://github.com/ClickHouse/ClickHouse/pull/21630) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Support parallel parsing for `CSVWithNames` and `TSVWithNames` formats. This closes [#21085](https://github.com/ClickHouse/ClickHouse/issues/21085). [#21149](https://github.com/ClickHouse/ClickHouse/pull/21149) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). @@ -1371,7 +1371,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Add option `--backslash` for `clickhouse-format`, which can add a backslash at the end of each line of the formatted query. [#21494](https://github.com/ClickHouse/ClickHouse/pull/21494) ([flynn](https://github.com/ucasFL)). * Now clickhouse will not throw `LOGICAL_ERROR` exception when we try to mutate the already covered part. Fixes [#22013](https://github.com/ClickHouse/ClickHouse/issues/22013). [#22291](https://github.com/ClickHouse/ClickHouse/pull/22291) ([alesapin](https://github.com/alesapin)). -#### Bug Fix {#bug-fix-6} +#### Bug fix {#bug-fix-6} * Remove socket from epoll before cancelling packet receiver in `HedgedConnections` to prevent possible race. Fixes [#22161](https://github.com/ClickHouse/ClickHouse/issues/22161). [#22443](https://github.com/ClickHouse/ClickHouse/pull/22443) ([Kruglov Pavel](https://github.com/Avogar)). * Add (missing) memory accounting in parallel parsing routines. In previous versions OOM was possible when the resultset contains very large blocks of data. This closes [#22008](https://github.com/ClickHouse/ClickHouse/issues/22008). [#22425](https://github.com/ClickHouse/ClickHouse/pull/22425) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -1415,7 +1415,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * `force_drop_table` flag didn't work for `MATERIALIZED VIEW`, it's fixed. Fixes [#18943](https://github.com/ClickHouse/ClickHouse/issues/18943). [#20626](https://github.com/ClickHouse/ClickHouse/pull/20626) ([tavplubix](https://github.com/tavplubix)). * Fix name clashes in `PredicateRewriteVisitor`. It caused incorrect `WHERE` filtration after full join. Close [#20497](https://github.com/ClickHouse/ClickHouse/issues/20497). [#20622](https://github.com/ClickHouse/ClickHouse/pull/20622) ([Vladimir](https://github.com/vdimir)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-7} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-7} * Add [Jepsen](https://github.com/jepsen-io/jepsen) tests for ClickHouse Keeper. [#21677](https://github.com/ClickHouse/ClickHouse/pull/21677) ([alesapin](https://github.com/alesapin)). * Run stateless tests in parallel in CI. Depends on [#22181](https://github.com/ClickHouse/ClickHouse/issues/22181). [#22300](https://github.com/ClickHouse/ClickHouse/pull/22300) ([alesapin](https://github.com/alesapin)). @@ -1430,17 +1430,17 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Fix macOS shared lib build. [#20184](https://github.com/ClickHouse/ClickHouse/pull/20184) ([nvartolomei](https://github.com/nvartolomei)). * Add `ctime` option to `zookeeper-dump-tree`. It allows to dump node creation time. [#21842](https://github.com/ClickHouse/ClickHouse/pull/21842) ([Ilya](https://github.com/HumanUser)). -## ClickHouse release 21.3 (LTS) {#clickhouse-release-213-lts} +## ClickHouse release 21.3 (lts) {#clickhouse-release-213-lts} ### ClickHouse release v21.3, 2021-03-12 {#clickhouse-release-v213-2021-03-12} -#### Backward Incompatible Change {#backward-incompatible-change-8} +#### Backward incompatible change {#backward-incompatible-change-8} * Now it's not allowed to create MergeTree tables in old syntax with table TTL because it's just ignored. Attach of old tables is still possible. [#20282](https://github.com/ClickHouse/ClickHouse/pull/20282) ([alesapin](https://github.com/alesapin)). * Now all case-insensitive function names will be rewritten to their canonical representations. This is needed for projection query routing (the upcoming feature). [#20174](https://github.com/ClickHouse/ClickHouse/pull/20174) ([Amos Bird](https://github.com/amosbird)). * Fix creation of `TTL` in cases, when its expression is a function and it is the same as `ORDER BY` key. Now it's allowed to set custom aggregation to primary key columns in `TTL` with `GROUP BY`. Backward incompatible: For primary key columns, which are not in `GROUP BY` and aren't set explicitly now is applied function `any` instead of `max`, when TTL is expired. Also if you use TTL with `WHERE` or `GROUP BY` you can see exceptions at merges, while making rolling update. [#15450](https://github.com/ClickHouse/ClickHouse/pull/15450) ([Anton Popov](https://github.com/CurtizJ)). -#### New Feature {#new-feature-8} +#### New feature {#new-feature-8} * Add file engine settings: `engine_file_empty_if_not_exists` and `engine_file_truncate_on_insert`. [#20620](https://github.com/ClickHouse/ClickHouse/pull/20620) ([M0r64n](https://github.com/M0r64n)). * Add aggregate function `deltaSum` for summing the differences between consecutive rows. [#20057](https://github.com/ClickHouse/ClickHouse/pull/20057) ([Russ Frank](https://github.com/rf)). @@ -1461,7 +1461,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Introduce experimental support for window functions, enabled with `allow_experimental_window_functions = 1`. This is a preliminary, alpha-quality implementation that is not suitable for production use and will change in backward-incompatible ways in future releases. Please see [the documentation](https://github.com/ClickHouse/ClickHouse/blob/master/docs/sql-reference/window-functions/index.md#experimental-window-functions) for the list of supported features. [#20337](https://github.com/ClickHouse/ClickHouse/pull/20337) ([Alexander Kuzmenkov](https://github.com/akuzm)). * Add the ability to backup/restore metadata files for DiskS3. [#18377](https://github.com/ClickHouse/ClickHouse/pull/18377) ([Pavel Kovalenko](https://github.com/Jokser)). -#### Performance Improvement {#performance-improvement-8} +#### Performance improvement {#performance-improvement-8} * Hedged requests for remote queries. When setting `use_hedged_requests` enabled (off by default), allow to establish many connections with different replicas for query. New connection is enabled in case existent connection(s) with replica(s) were not established within `hedged_connection_timeout` or no data was received within `receive_data_timeout`. Query uses the first connection which send non empty progress packet (or data packet, if `allow_changing_replica_until_first_data_packet`); other connections are cancelled. Queries with `max_parallel_replicas > 1` are supported. [#19291](https://github.com/ClickHouse/ClickHouse/pull/19291) ([Kruglov Pavel](https://github.com/Avogar)). This allows to significantly reduce tail latencies on very large clusters. * Added support for `PREWHERE` (and enable the corresponding optimization) when tables have row-level security expressions specified. [#19576](https://github.com/ClickHouse/ClickHouse/pull/19576) ([Denis Glazachev](https://github.com/traceon)). @@ -1517,7 +1517,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Fix some rare cases when write errors can be ignored in destructors. [#19451](https://github.com/ClickHouse/ClickHouse/pull/19451) ([Azat Khuzhin](https://github.com/azat)). * Print inline frames in stack traces for fatal errors. [#19317](https://github.com/ClickHouse/ClickHouse/pull/19317) ([Ivan](https://github.com/abyss7)). -#### Bug Fix {#bug-fix-7} +#### Bug fix {#bug-fix-7} * Fix redundant reconnects to ZooKeeper and the possibility of two active sessions for a single clickhouse server. Both problems introduced in #14678. [#21264](https://github.com/ClickHouse/ClickHouse/pull/21264) ([alesapin](https://github.com/alesapin)). * Fix error `Bad cast from type ... to DB::ColumnLowCardinality` while inserting into table with `LowCardinality` column from `Values` format. Fixes #21140 [#21357](https://github.com/ClickHouse/ClickHouse/pull/21357) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -1576,7 +1576,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Fix a bug that moving pieces to destination table may failed in case of launching multiple clickhouse-copiers. [#19743](https://github.com/ClickHouse/ClickHouse/pull/19743) ([madianjun](https://github.com/mdianjun)). * Background thread which executes `ON CLUSTER` queries might hang waiting for dropped replicated table to do something. It's fixed. [#19684](https://github.com/ClickHouse/ClickHouse/pull/19684) ([yiguolei](https://github.com/yiguolei)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-8} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-8} * Allow to build ClickHouse with AVX-2 enabled globally. It gives slight performance benefits on modern CPUs. Not recommended for production and will not be supported as official build for now. [#20180](https://github.com/ClickHouse/ClickHouse/pull/20180) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix some of the issues found by Coverity. See [#19964](https://github.com/ClickHouse/ClickHouse/issues/19964). [#20010](https://github.com/ClickHouse/ClickHouse/pull/20010) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -1589,7 +1589,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor ### ClickHouse release v21.2.2.8-stable, 2021-02-07 {#clickhouse-release-v21228-stable-2021-02-07} -#### Backward Incompatible Change {#backward-incompatible-change-9} +#### Backward incompatible change {#backward-incompatible-change-9} * Bitwise functions (`bitAnd`, `bitOr`, etc) are forbidden for floating point arguments. Now you have to do explicit cast to integer. [#19853](https://github.com/ClickHouse/ClickHouse/pull/19853) ([Azat Khuzhin](https://github.com/azat)). * Forbid `lcm`/`gcd` for floats. [#19532](https://github.com/ClickHouse/ClickHouse/pull/19532) ([Azat Khuzhin](https://github.com/azat)). @@ -1597,7 +1597,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Disallow floating point column as partition key, see [#18421](https://github.com/ClickHouse/ClickHouse/issues/18421#event-4147046255). [#18464](https://github.com/ClickHouse/ClickHouse/pull/18464) ([hexiaoting](https://github.com/hexiaoting)). * Excessive parenthesis in type definitions no longer supported, example: `Array((UInt8))`. -#### New Feature {#new-feature-9} +#### New feature {#new-feature-9} * Added `PostgreSQL` table engine (both select/insert, with support for multidimensional arrays), also as table function. Added `PostgreSQL` dictionary source. Added `PostgreSQL` database engine. [#18554](https://github.com/ClickHouse/ClickHouse/pull/18554) ([Kseniia Sumarokova](https://github.com/kssenii)). * Data type `Nested` now supports arbitrary levels of nesting. Introduced subcolumns of complex types, such as `size0` in `Array`, `null` in `Nullable`, names of `Tuple` elements, which can be read without reading of whole column. [#17310](https://github.com/ClickHouse/ClickHouse/pull/17310) ([Anton Popov](https://github.com/CurtizJ)). @@ -1617,7 +1617,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Support `EXISTS VIEW` syntax. [#18552](https://github.com/ClickHouse/ClickHouse/pull/18552) ([Du Chuan](https://github.com/spongedu)). * Add `SELECT ALL` syntax. closes [#18706](https://github.com/ClickHouse/ClickHouse/issues/18706). [#18723](https://github.com/ClickHouse/ClickHouse/pull/18723) ([flynn](https://github.com/ucasFL)). -#### Performance Improvement {#performance-improvement-9} +#### Performance improvement {#performance-improvement-9} * Faster parts removal by lowering the number of `stat` syscalls. This returns the optimization that existed while ago. More safe interface of `IDisk`. This closes [#19065](https://github.com/ClickHouse/ClickHouse/issues/19065). [#19086](https://github.com/ClickHouse/ClickHouse/pull/19086) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Aliases declared in `WITH` statement are properly used in index analysis. Queries like `WITH column AS alias SELECT ... WHERE alias = ...` may use index now. [#18896](https://github.com/ClickHouse/ClickHouse/pull/18896) ([Amos Bird](https://github.com/amosbird)). @@ -1670,7 +1670,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Added support for `WITH ... [AND] [PERIODIC] REFRESH [interval_in_sec]` clause when creating `LIVE VIEW` tables. [#14822](https://github.com/ClickHouse/ClickHouse/pull/14822) ([vzakaznikov](https://github.com/vzakaznikov)). * Restrict `MODIFY TTL` queries for `MergeTree` tables created in old syntax. Previously the query succeeded, but actually it had no effect. [#19064](https://github.com/ClickHouse/ClickHouse/pull/19064) ([Anton Popov](https://github.com/CurtizJ)). -#### Bug Fix {#bug-fix-8} +#### Bug fix {#bug-fix-8} * Fix index analysis of binary functions with constant argument which leads to wrong query results. This fixes [#18364](https://github.com/ClickHouse/ClickHouse/issues/18364). [#18373](https://github.com/ClickHouse/ClickHouse/pull/18373) ([Amos Bird](https://github.com/amosbird)). * Fix starting the server with tables having default expressions containing dictGet(). Allow getting return type of dictGet() without loading dictionary. [#19805](https://github.com/ClickHouse/ClickHouse/pull/19805) ([Vitaly Baranov](https://github.com/vitlibar)). @@ -1711,7 +1711,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Fix data type convert issue for MySQL engine. [#18124](https://github.com/ClickHouse/ClickHouse/pull/18124) ([bo zeng](https://github.com/mis98zb)). * Fix clickhouse-client abort exception while executing only `select`. [#19790](https://github.com/ClickHouse/ClickHouse/pull/19790) ([taiyang-li](https://github.com/taiyang-li)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-9} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-9} * Run [SQLancer](https://twitter.com/RiggerManuel/status/1352345625480884228) (logical SQL fuzzer) in CI. [#19006](https://github.com/ClickHouse/ClickHouse/pull/19006) ([Ilya Yatsishin](https://github.com/qoega)). * Query Fuzzer will fuzz newly added tests more extensively. This closes [#18916](https://github.com/ClickHouse/ClickHouse/issues/18916). [#19185](https://github.com/ClickHouse/ClickHouse/pull/19185) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -1732,7 +1732,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor ### ClickHouse release v21.1.3.32-stable, 2021-02-03 {#clickhouse-release-v211332-stable-2021-02-03} -#### Bug Fix {#bug-fix-9} +#### Bug fix {#bug-fix-9} * BloomFilter index crash fix. Fixes [#19757](https://github.com/ClickHouse/ClickHouse/issues/19757). [#19884](https://github.com/ClickHouse/ClickHouse/pull/19884) ([Maksim Kita](https://github.com/kitaisreal)). * Fix crash when pushing down predicates to union distinct subquery. This fixes [#19855](https://github.com/ClickHouse/ClickHouse/issues/19855). [#19861](https://github.com/ClickHouse/ClickHouse/pull/19861) ([Amos Bird](https://github.com/amosbird)). @@ -1760,7 +1760,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor ### ClickHouse release v21.1.2.15-stable 2021-01-18 {#clickhouse-release-v211215-stable-2021-01-18} -#### Backward Incompatible Change {#backward-incompatible-change-10} +#### Backward incompatible change {#backward-incompatible-change-10} * The setting `input_format_null_as_default` is enabled by default. [#17525](https://github.com/ClickHouse/ClickHouse/pull/17525) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Check settings constraints for profile settings from config. Server will fail to start if users.xml contain settings that do not meet corresponding constraints. [#18486](https://github.com/ClickHouse/ClickHouse/pull/18486) ([tavplubix](https://github.com/tavplubix)). @@ -1774,7 +1774,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * SSL Certificates with incorrect "key usage" are rejected. In previous versions they are used to work. See [#19262](https://github.com/ClickHouse/ClickHouse/issues/19262). * `incl` references to substitutions file (`/etc/metrika.xml`) were removed from the default config (``, ``, ``, ``, ``). If you were using substitutions file and were relying on those implicit references, you should put them back manually and explicitly by adding corresponding sections with `incl="..."` attributes before the update. See [#18740](https://github.com/ClickHouse/ClickHouse/pull/18740) ([alexey-milovidov](https://github.com/alexey-milovidov)). -#### New Feature {#new-feature-10} +#### New feature {#new-feature-10} * Implement gRPC protocol in ClickHouse. [#15111](https://github.com/ClickHouse/ClickHouse/pull/15111) ([Vitaly Baranov](https://github.com/vitlibar)). * Allow to use multiple zookeeper clusters. [#17070](https://github.com/ClickHouse/ClickHouse/pull/17070) ([fastio](https://github.com/fastio)). @@ -1817,13 +1817,13 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Added `query` parameter for `clickhouse-benchmark`. [#17832](https://github.com/ClickHouse/ClickHouse/pull/17832) ([Maksim Kita](https://github.com/kitaisreal)). * `EXPLAIN AST` now support queries other then `SELECT`. [#18136](https://github.com/ClickHouse/ClickHouse/pull/18136) ([taiyang-li](https://github.com/taiyang-li)). -#### Experimental Feature {#experimental-feature-8} +#### Experimental feature {#experimental-feature-8} * Added functions for calculation of minHash and simHash of text n-grams and shingles. They are intended for semi-duplicate search. Also functions `bitHammingDistance` and `tupleHammingDistance` are added. [#7649](https://github.com/ClickHouse/ClickHouse/pull/7649) ([flynn](https://github.com/ucasFL)). * Add new data type `Map`. See [#1841](https://github.com/ClickHouse/ClickHouse/issues/1841). First version for Map only supports `String` type of key and value. [#15806](https://github.com/ClickHouse/ClickHouse/pull/15806) ([hexiaoting](https://github.com/hexiaoting)). * Implement alternative SQL parser based on ANTLR4 runtime and generated from EBNF grammar. [#11298](https://github.com/ClickHouse/ClickHouse/pull/11298) ([Ivan](https://github.com/abyss7)). -#### Performance Improvement {#performance-improvement-10} +#### Performance improvement {#performance-improvement-10} * New IP Dictionary implementation with lower memory consumption, improved performance for some cases, and fixed bugs. [#16804](https://github.com/ClickHouse/ClickHouse/pull/16804) ([vdimir](https://github.com/vdimir)). * Parallel formatting for data export. [#11617](https://github.com/ClickHouse/ClickHouse/pull/11617) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). @@ -1909,7 +1909,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Fix never worked `fsync_part_directory`/`fsync_after_insert`/`in_memory_parts_insert_sync` (experimental feature). [#18845](https://github.com/ClickHouse/ClickHouse/pull/18845) ([Azat Khuzhin](https://github.com/azat)). * Allow using `Atomic` engine for nested database of `MaterializeMySQL` engine. [#14849](https://github.com/ClickHouse/ClickHouse/pull/14849) ([tavplubix](https://github.com/tavplubix)). -#### Bug Fix {#bug-fix-10} +#### Bug fix {#bug-fix-10} * Fix the issue when server can stop accepting connections in very rare cases. [#17542](https://github.com/ClickHouse/ClickHouse/pull/17542) (Amos Bird, [alexey-milovidov](https://github.com/alexey-milovidov)). * Fix index analysis of binary functions with constant argument which leads to wrong query results. This fixes [#18364](https://github.com/ClickHouse/ClickHouse/issues/18364). [#18373](https://github.com/ClickHouse/ClickHouse/pull/18373) ([Amos Bird](https://github.com/amosbird)). @@ -1999,7 +1999,7 @@ keywords: ['ClickHouse 2021', 'changelog 2021', 'release notes', 'version histor * Throw error when `REPLACE` column transformer operates on non existing column. [#16183](https://github.com/ClickHouse/ClickHouse/pull/16183) ([hexiaoting](https://github.com/hexiaoting)). * Throw exception in case of not equi-join ON expression in RIGH|FULL JOIN. [#15162](https://github.com/ClickHouse/ClickHouse/pull/15162) ([Artem Zuikov](https://github.com/4ertus2)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-10} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-10} * Add simple integrity check for ClickHouse binary. It allows to detect corruption due to faulty hardware (bit rot on storage media or bit flips in RAM). [#18811](https://github.com/ClickHouse/ClickHouse/pull/18811) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Change `OpenSSL` to `BoringSSL`. It allows to avoid issues with sanitizers. This fixes [#12490](https://github.com/ClickHouse/ClickHouse/issues/12490). This fixes [#17502](https://github.com/ClickHouse/ClickHouse/issues/17502). This fixes [#12952](https://github.com/ClickHouse/ClickHouse/issues/12952). [#18129](https://github.com/ClickHouse/ClickHouse/pull/18129) ([alexey-milovidov](https://github.com/alexey-milovidov)). diff --git a/docs/whats-new/changelog/2022.md b/docs/whats-new/changelog/2022.md index 36aac985174..826ebe92bd1 100644 --- a/docs/whats-new/changelog/2022.md +++ b/docs/whats-new/changelog/2022.md @@ -8,7 +8,7 @@ keywords: ['ClickHouse 2022', 'changelog 2022', 'release notes', 'version histor doc_type: 'changelog' --- -### ClickHouse release 22.12, 2022-12-15 {#a-id2212a-clickhouse-release-2212-2022-12-15} +### ClickHouse release 22.12, 2022-12-15 {#a-id2212a-clickhouse-release-2212-2022-12-15} :::warning @@ -18,10 +18,10 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl ::: -#### Upgrade Notes {#upgrade-notes} +#### Upgrade notes {#upgrade-notes} * Fixed backward incompatibility in (de)serialization of states of `min`, `max`, `any*`, `argMin`, `argMax` aggregate functions with `String` argument. The incompatibility affects 22.9, 22.10 and 22.11 branches (fixed since 22.9.6, 22.10.4 and 22.11.2 correspondingly). Some minor releases of 22.3, 22.7 and 22.8 branches are also affected: 22.3.13...22.3.14 (fixed since 22.3.15), 22.8.6...22.8.9 (fixed since 22.8.10), 22.7.6 and newer (will not be fixed in 22.7, we recommend upgrading from 22.7.* to 22.8.10 or newer). This release note does not concern users that have never used affected versions. Incompatible versions append an extra `'\0'` to strings when reading states of the aggregate functions mentioned above. For example, if an older version saved state of `anyState('foobar')` to `state_column` then the incompatible version will print `'foobar\0'` on `anyMerge(state_column)`. Also incompatible versions write states of the aggregate functions without trailing `'\0'`. Newer versions (that have the fix) can correctly read data written by all versions including incompatible versions, except one corner case. If an incompatible version saved a state with a string that actually ends with null character, then newer version will trim trailing `'\0'` when reading state of affected aggregate function. For example, if an incompatible version saved state of `anyState('abrac\0dabra\0')` to `state_column` then newer versions will print `'abrac\0dabra'` on `anyMerge(state_column)`. The issue also affects distributed queries when an incompatible version works in a cluster together with older or newer versions. [#43038](https://github.com/ClickHouse/ClickHouse/pull/43038) ([Alexander Tokmakov](https://github.com/tavplubix), [Raúl Marín](https://github.com/Algunenano)). Note: all the official ClickHouse builds already include the patches. This is not necessarily true for unofficial third-party builds that should be avoided. -#### New Feature {#new-feature} +#### New feature {#new-feature} * Add `BSONEachRow` input/output format. In this format, ClickHouse formats/parses each row as a separate BSON document and each column is formatted/parsed as a single BSON field with the column name as the key. [#42033](https://github.com/ClickHouse/ClickHouse/pull/42033) ([mark-polokhov](https://github.com/mark-polokhov)). * Add `grace_hash` JOIN algorithm, it can be enabled with `SET join_algorithm = 'grace_hash'`. [#38191](https://github.com/ClickHouse/ClickHouse/pull/38191) ([BigRedEye](https://github.com/BigRedEye), [Vladimir C](https://github.com/vdimir)). * Allow configuring password complexity rules and checks for creating and changing users. [#43719](https://github.com/ClickHouse/ClickHouse/pull/43719) ([Nikolay Degterinsky](https://github.com/evillique)). @@ -40,12 +40,12 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Added support to define constraints for merge tree settings. For example you can forbid overriding the `storage_policy` by users. [#43903](https://github.com/ClickHouse/ClickHouse/pull/43903) ([Sergei Trifonov](https://github.com/serxa)). * Add a new setting `input_format_json_read_objects_as_strings` that allows the parsing of nested JSON objects into Strings in all JSON input formats. This setting is disabled by default. [#44052](https://github.com/ClickHouse/ClickHouse/pull/44052) ([Kruglov Pavel](https://github.com/Avogar)). -#### Experimental Feature {#experimental-feature} +#### Experimental feature {#experimental-feature} * Support deduplication for asynchronous inserts. Before this change, async inserts did not support deduplication, because multiple small inserts coexisted in one inserted batch. Closes [#38075](https://github.com/ClickHouse/ClickHouse/issues/38075). [#43304](https://github.com/ClickHouse/ClickHouse/pull/43304) ([Han Fei](https://github.com/hanfei1991)). * Add support for cosine distance for the experimental Annoy (vector similarity search) index. [#42778](https://github.com/ClickHouse/ClickHouse/pull/42778) ([Filatenkov Artur](https://github.com/FArthur-cmd)). * Add `CREATE / ALTER / DROP NAMED COLLECTION` queries. [#43252](https://github.com/ClickHouse/ClickHouse/pull/43252) ([Kseniia Sumarokova](https://github.com/kssenii)). This feature is under development and the queries are not effective as of version 22.12. This changelog entry is added only to avoid confusion. Restrict default access to named collections to the user defined in config. This requires that `show_named_collections = 1` is set to be able to see them. [#43325](https://github.com/ClickHouse/ClickHouse/pull/43325) ([Kseniia Sumarokova](https://github.com/kssenii)). The `system.named_collections` table is introduced [#43147](https://github.com/ClickHouse/ClickHouse/pull/43147) ([Kseniia Sumarokova](https://github.com/kssenii)). -#### Performance Improvement {#performance-improvement} +#### Performance improvement {#performance-improvement} * Add settings `max_streams_for_merge_tree_reading` and `allow_asynchronous_read_from_io_pool_for_merge_tree`. Setting `max_streams_for_merge_tree_reading` limits the number of reading streams for MergeTree tables. Setting `allow_asynchronous_read_from_io_pool_for_merge_tree` enables a background I/O pool to read from `MergeTree` tables. This may increase performance for I/O bound queries if used together with `max_streams_to_max_threads_ratio` or `max_streams_for_merge_tree_reading`. [#43260](https://github.com/ClickHouse/ClickHouse/pull/43260) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). This improves performance up to 100 times in case of high latency storage, low number of CPU and high number of data parts. * Settings `merge_tree_min_rows_for_concurrent_read_for_remote_filesystem/merge_tree_min_bytes_for_concurrent_read_for_remote_filesystem` did not respect adaptive granularity. Fat rows did not decrease the number of read rows (as it was done for `merge_tree_min_rows_for_concurrent_read/merge_tree_min_bytes_for_concurrent_read`, which could lead to high memory usage when using remote filesystems. [#43965](https://github.com/ClickHouse/ClickHouse/pull/43965) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Optimized the number of list requests to ZooKeeper or ClickHouse Keeper when selecting a part to merge. Previously it could produce thousands of requests in some cases. Fixes [#43647](https://github.com/ClickHouse/ClickHouse/issues/43647). [#43675](https://github.com/ClickHouse/ClickHouse/pull/43675) ([Alexander Tokmakov](https://github.com/tavplubix)). @@ -81,13 +81,13 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * When ClickHouse requests a remote HTTP server, and it returns an error, the numeric HTTP code was not displayed correctly in the exception message. Closes [#43919](https://github.com/ClickHouse/ClickHouse/issues/43919). [#43920](https://github.com/ClickHouse/ClickHouse/pull/43920) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Correctly report errors in queries even when multiple JOINs optimization is taking place. [#43583](https://github.com/ClickHouse/ClickHouse/pull/43583) ([Salvatore](https://github.com/tbsal)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement} * Systemd integration now correctly notifies systemd that the service is really started and is ready to serve requests. [#43400](https://github.com/ClickHouse/ClickHouse/pull/43400) ([Коренберг Марк](https://github.com/socketpair)). * Added the option to build ClickHouse with OpenSSL using the [OpenSSL FIPS Module](https://www.openssl.org/docs/man3.0/man7/fips_module.html). This build type has not been tested to validate security and is not supported. [#43991](https://github.com/ClickHouse/ClickHouse/pull/43991) ([Boris Kuschel](https://github.com/bkuschel)). * Upgrade to the new `DeflateQpl` compression codec which has been implemented in a previous PR (details: https://github.com/ClickHouse/ClickHouse/pull/39494). This patch improves codec on below aspects: 1. QPL v0.2.0 to QPL v0.3.0 [Intel® Query Processing Library (QPL)](https://github.com/intel/qpl) 2. Improve CMake file for fixing QPL build issues for QPL v0.3.0. 3. Link the QPL library with libaccel-config at build time instead of runtime loading on QPL v0.2.0 (dlopen) 4. Fixed log print issue in CompressionCodecDeflateQpl.cpp. [#44024](https://github.com/ClickHouse/ClickHouse/pull/44024) ([jasperzhu](https://github.com/jinjunzh)). -#### Bug Fix (user-visible misbehavior in official stable or prestable release) {#bug-fix-user-visible-misbehavior-in-official-stable-or-prestable-release} +#### Bug fix (user-visible misbehavior in official stable or prestable release) {#bug-fix-user-visible-misbehavior-in-official-stable-or-prestable-release} * Fixed bug which could lead to deadlock while using asynchronous inserts. [#43233](https://github.com/ClickHouse/ClickHouse/pull/43233) ([Anton Popov](https://github.com/CurtizJ)). * Fix some incorrect logic in AST level optimization `optimize_normalize_count_variants`. [#43873](https://github.com/ClickHouse/ClickHouse/pull/43873) ([Duc Canh Le](https://github.com/canhld94)). @@ -129,12 +129,12 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Fixed exception when a subquery contains HAVING but doesn't contain an actual aggregation. [#44051](https://github.com/ClickHouse/ClickHouse/pull/44051) ([Nikita Taranov](https://github.com/nickitat)). * Fix race in s3 multipart upload. This race could cause the error `Part number must be an integer between 1 and 10000, inclusive. (S3_ERROR)` while restoring from a backup. [#44065](https://github.com/ClickHouse/ClickHouse/pull/44065) ([Vitaly Baranov](https://github.com/vitlibar)). -### ClickHouse release 22.11, 2022-11-17 {#a-id2211a-clickhouse-release-2211-2022-11-17} +### ClickHouse release 22.11, 2022-11-17 {#a-id2211a-clickhouse-release-2211-2022-11-17} -#### Backward Incompatible Change {#backward-incompatible-change} +#### Backward incompatible change {#backward-incompatible-change} * `JSONExtract` family of functions will now attempt to coerce to the requested type. [#41502](https://github.com/ClickHouse/ClickHouse/pull/41502) ([Márcio Martins](https://github.com/marcioapm)). -#### New Feature {#new-feature-1} +#### New feature {#new-feature-1} * Adds support for retries during INSERTs into ReplicatedMergeTree when a session with ClickHouse Keeper is lost. Apart from fault tolerance, it aims to provide better user experience, - avoid returning a user an error during insert if keeper is restarted (for example, due to upgrade). [#42607](https://github.com/ClickHouse/ClickHouse/pull/42607) ([Igor Nikonov](https://github.com/devcrafter)). * Add `Hudi` and `DeltaLake` table engines, read-only, only for tables on S3. [#41054](https://github.com/ClickHouse/ClickHouse/pull/41054) ([Daniil Rubin](https://github.com/rubin-do), [Kseniia Sumarokova](https://github.com/kssenii)). * Add table function `hudi` and `deltaLake`. [#43080](https://github.com/ClickHouse/ClickHouse/pull/43080) ([flynn](https://github.com/ucasfl)). @@ -151,14 +151,14 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Add `min_age_to_force_merge_on_partition_only` setting to optimize old parts for the entire partition only. [#42659](https://github.com/ClickHouse/ClickHouse/pull/42659) ([Antonio Andelic](https://github.com/antonio2368)). * Add generic implementation for arbitrary structured named collections, access type and `system.named_collections`. [#43147](https://github.com/ClickHouse/ClickHouse/pull/43147) ([Kseniia Sumarokova](https://github.com/kssenii)). -#### Performance Improvement {#performance-improvement-1} +#### Performance improvement {#performance-improvement-1} * `match` function can use the index if it's a condition on string prefix. This closes [#37333](https://github.com/ClickHouse/ClickHouse/issues/37333). [#42458](https://github.com/ClickHouse/ClickHouse/pull/42458) ([clarkcaoliu](https://github.com/Clark0)). * Speed up AND and OR operators when they are sequenced. [#42214](https://github.com/ClickHouse/ClickHouse/pull/42214) ([Zhiguo Zhou](https://github.com/ZhiguoZh)). * Support parallel parsing for `LineAsString` input format. This improves performance just slightly. This closes [#42502](https://github.com/ClickHouse/ClickHouse/issues/42502). [#42780](https://github.com/ClickHouse/ClickHouse/pull/42780) ([Kruglov Pavel](https://github.com/Avogar)). * ClickHouse Keeper performance improvement: improve commit performance for cases when many different nodes have uncommitted states. This should help with cases when a follower node can't sync fast enough. [#42926](https://github.com/ClickHouse/ClickHouse/pull/42926) ([Antonio Andelic](https://github.com/antonio2368)). * A condition like `NOT LIKE 'prefix%'` can use the primary index. [#42209](https://github.com/ClickHouse/ClickHouse/pull/42209) ([Duc Canh Le](https://github.com/canhld94)). -#### Experimental Feature {#experimental-feature-1} +#### Experimental feature {#experimental-feature-1} * Support type `Object` inside other types, e.g. `Array(JSON)`. [#36969](https://github.com/ClickHouse/ClickHouse/pull/36969) ([Anton Popov](https://github.com/CurtizJ)). * Ignore MySQL binlog SAVEPOINT event for MaterializedMySQL. [#42931](https://github.com/ClickHouse/ClickHouse/pull/42931) ([zzsmdfj](https://github.com/zzsmdfj)). Handle (ignore) SAVEPOINT queries in MaterializedMySQL. [#43086](https://github.com/ClickHouse/ClickHouse/pull/43086) ([Stig Bakken](https://github.com/stigsb)). @@ -188,7 +188,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Add some functions for compatibility with PowerBI, when it connects using MySQL protocol [#42612](https://github.com/ClickHouse/ClickHouse/pull/42612) ([Filatenkov Artur](https://github.com/FArthur-cmd)). * Better usability for Dashboard on changes [#42872](https://github.com/ClickHouse/ClickHouse/pull/42872) ([Vladimir C](https://github.com/vdimir)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-1} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-1} * Run SQLancer for each pull request and commit to master. [SQLancer](https://github.com/sqlancer/sqlancer) is an OpenSource fuzzer that focuses on automatic detection of logical bugs. [#42397](https://github.com/ClickHouse/ClickHouse/pull/42397) ([Ilya Yatsishin](https://github.com/qoega)). * Update to latest zlib-ng. [#42463](https://github.com/ClickHouse/ClickHouse/pull/42463) ([Boris Kuschel](https://github.com/bkuschel)). * Add support for testing ClickHouse server with Jepsen. By the way, we already have support for testing ClickHouse Keeper with Jepsen. This pull request extends it to Replicated tables. [#42619](https://github.com/ClickHouse/ClickHouse/pull/42619) ([Antonio Andelic](https://github.com/antonio2368)). @@ -196,7 +196,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Before the fix, the user-defined config was preserved by RPM in `$file.rpmsave`. The PR fixes it and won't replace the user's files from packages. [#42936](https://github.com/ClickHouse/ClickHouse/pull/42936) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). * Remove some libraries from Ubuntu Docker image. [#42622](https://github.com/ClickHouse/ClickHouse/pull/42622) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### Bug Fix (user-visible misbehavior in official stable or prestable release) {#bug-fix-user-visible-misbehavior-in-official-stable-or-prestable-release-1} +#### Bug fix (user-visible misbehavior in official stable or prestable release) {#bug-fix-user-visible-misbehavior-in-official-stable-or-prestable-release-1} * Updated normaliser to clone the alias ast. Resolves [#42452](https://github.com/ClickHouse/ClickHouse/issues/42452) Implementation: * Updated QueryNormalizer to clone alias ast, when its replaced. Previously just assigning the same leads to exception in LogicalExpressinsOptimizer as it would be the same parent being inserted again. * This bug is not seen with new analyser (allow_experimental_analyzer), so no changes for it. I added a test for the same. [#42827](https://github.com/ClickHouse/ClickHouse/pull/42827) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni)). * Fix race for backup of tables in `Lazy` databases. [#43104](https://github.com/ClickHouse/ClickHouse/pull/43104) ([Vitaly Baranov](https://github.com/vitlibar)). @@ -231,14 +231,14 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * The `indent` field produced by the `git-import` tool was miscalculated. See https://clickhouse.com/docs/getting-started/example-datasets/github/. [#43191](https://github.com/ClickHouse/ClickHouse/pull/43191) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Fixed unexpected behaviour of `Interval` types with subquery and casting. [#43193](https://github.com/ClickHouse/ClickHouse/pull/43193) ([jh0x](https://github.com/jh0x)). -### ClickHouse release 22.10, 2022-10-25 {#a-id2210a-clickhouse-release-2210-2022-10-25} +### ClickHouse release 22.10, 2022-10-25 {#a-id2210a-clickhouse-release-2210-2022-10-25} -#### Backward Incompatible Change {#backward-incompatible-change-1} +#### Backward incompatible change {#backward-incompatible-change-1} * Rename cache commands: `show caches` -> `show filesystem caches`, `describe cache` -> `describe filesystem cache`. [#41508](https://github.com/ClickHouse/ClickHouse/pull/41508) ([Kseniia Sumarokova](https://github.com/kssenii)). * Remove support for the `WITH TIMEOUT` section for `LIVE VIEW`. This closes [#40557](https://github.com/ClickHouse/ClickHouse/issues/40557). [#42173](https://github.com/ClickHouse/ClickHouse/pull/42173) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Remove support for the `{database}` macro from the client's prompt. It was displayed incorrectly if the database was unspecified and it was not updated on `USE` statements. This closes [#25891](https://github.com/ClickHouse/ClickHouse/issues/25891). [#42508](https://github.com/ClickHouse/ClickHouse/pull/42508) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### New Feature {#new-feature-2} +#### New feature {#new-feature-2} * Composable protocol configuration is added. Now different protocols can be set up with different listen hosts. Protocol wrappers such as PROXYv1 can be set up over any other protocols (TCP, TCP secure, MySQL, Postgres). [#41198](https://github.com/ClickHouse/ClickHouse/pull/41198) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). * Add `S3` as a new type of the destination of backups. Support BACKUP to S3 with as-is path/data structure. [#42333](https://github.com/ClickHouse/ClickHouse/pull/42333) ([Vitaly Baranov](https://github.com/vitlibar)), [#42232](https://github.com/ClickHouse/ClickHouse/pull/42232) ([Azat Khuzhin](https://github.com/azat)). * Added functions (`randUniform`, `randNormal`, `randLogNormal`, `randExponential`, `randChiSquared`, `randStudentT`, `randFisherF`, `randBernoulli`, `randBinomial`, `randNegativeBinomial`, `randPoisson`) to generate random values according to the specified distributions. This closes [#21834](https://github.com/ClickHouse/ClickHouse/issues/21834). [#42411](https://github.com/ClickHouse/ClickHouse/pull/42411) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). @@ -254,11 +254,11 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Adds Morton Coding (ZCurve) encode/decode functions. [#41753](https://github.com/ClickHouse/ClickHouse/pull/41753) ([Constantine Peresypkin](https://github.com/pkit)). * Add support for `SET setting_name = DEFAULT`. [#42187](https://github.com/ClickHouse/ClickHouse/pull/42187) ([Filatenkov Artur](https://github.com/FArthur-cmd)). -#### Experimental Feature {#experimental-feature-2} +#### Experimental feature {#experimental-feature-2} * Added new infrastructure for query analysis and planning under the `allow_experimental_analyzer` setting. [#31796](https://github.com/ClickHouse/ClickHouse/pull/31796) ([Maksim Kita](https://github.com/kitaisreal)). * Initial implementation of Kusto Query Language. Please don't use it. [#37961](https://github.com/ClickHouse/ClickHouse/pull/37961) ([Yong Wang](https://github.com/kashwy)). -#### Performance Improvement {#performance-improvement-2} +#### Performance improvement {#performance-improvement-2} * Relax the "Too many parts" threshold. This closes [#6551](https://github.com/ClickHouse/ClickHouse/issues/6551). Now ClickHouse will allow more parts in a partition if the average part size is large enough (at least 10 GiB). This allows to have up to petabytes of data in a single partition of a single table on a single server, which is possible using disk shelves or object storage. [#42002](https://github.com/ClickHouse/ClickHouse/pull/42002) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Implement operator precedence element parser to make the required stack size smaller. [#34892](https://github.com/ClickHouse/ClickHouse/pull/34892) ([Nikolay Degterinsky](https://github.com/evillique)). * DISTINCT in order optimization leverage sorting properties of data streams. This improvement will enable reading in order for DISTINCT if applicable (before it was necessary to provide ORDER BY for columns in DISTINCT). [#41014](https://github.com/ClickHouse/ClickHouse/pull/41014) ([Igor Nikonov](https://github.com/devcrafter)). @@ -311,7 +311,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Added new infrastructure for query analysis and planning under `allow_experimental_analyzer` setting. [#31796](https://github.com/ClickHouse/ClickHouse/pull/31796) ([Maksim Kita](https://github.com/kitaisreal)). * Improve the time to recover lost keeper connections. [#42541](https://github.com/ClickHouse/ClickHouse/pull/42541) ([Raúl Marín](https://github.com/Algunenano)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-2} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-2} * Add fuzzer for table definitions [#40096](https://github.com/ClickHouse/ClickHouse/pull/40096) ([Anton Popov](https://github.com/CurtizJ)). This represents the biggest advancement for ClickHouse testing in this year so far. * Beta version of the ClickHouse Cloud service is released: [https://console.clickhouse.cloud/](https://console.clickhouse.cloud/). It provides the easiest way to use ClickHouse (even slightly easier than the single-command installation). * Added support of WHERE clause generation to AST Fuzzer and possibility to add or remove ORDER BY and WHERE clause. [#38519](https://github.com/ClickHouse/ClickHouse/pull/38519) ([Ilya Yatsishin](https://github.com/qoega)). @@ -321,7 +321,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Update cctz to `2022e` to support the new timezone changes. Palestine transitions are now Saturdays at 02:00. Simplify three Ukraine zones into one. Jordan and Syria switch from +02/+03 with DST to year-round +03. (https://data.iana.org/time-zones/tzdb/NEWS). This closes [#42252](https://github.com/ClickHouse/ClickHouse/issues/42252). [#42327](https://github.com/ClickHouse/ClickHouse/pull/42327) ([Alexey Milovidov](https://github.com/alexey-milovidov)). [#42273](https://github.com/ClickHouse/ClickHouse/pull/42273) ([Dom Del Nano](https://github.com/ddelnano)). * Add Rust code support into ClickHouse with BLAKE3 hash-function library as an example. [#33435](https://github.com/ClickHouse/ClickHouse/pull/33435) ([BoloniniD](https://github.com/BoloniniD)). -#### Bug Fix (user-visible misbehavior in official stable or prestable release) {#bug-fix-user-visible-misbehavior-in-official-stable-or-prestable-release-2} +#### Bug fix (user-visible misbehavior in official stable or prestable release) {#bug-fix-user-visible-misbehavior-in-official-stable-or-prestable-release-2} * Choose correct aggregation method for `LowCardinality` with big integer types. [#42342](https://github.com/ClickHouse/ClickHouse/pull/42342) ([Duc Canh Le](https://github.com/canhld94)). * Several fixes for `web` disk. [#41652](https://github.com/ClickHouse/ClickHouse/pull/41652) ([Kseniia Sumarokova](https://github.com/kssenii)). @@ -364,9 +364,9 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Arithmetic operator with Date or DateTime and 128 or 256-bit integer was referencing uninitialized memory. [#42453](https://github.com/ClickHouse/ClickHouse/issues/42453). [#42573](https://github.com/ClickHouse/ClickHouse/pull/42573) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Fix unexpected table loading error when partition key contains alias function names during server upgrade. [#36379](https://github.com/ClickHouse/ClickHouse/pull/36379) ([Amos Bird](https://github.com/amosbird)). -### ClickHouse release 22.9, 2022-09-22 {#a-id229a-clickhouse-release-229-2022-09-22} +### ClickHouse release 22.9, 2022-09-22 {#a-id229a-clickhouse-release-229-2022-09-22} -#### Backward Incompatible Change {#backward-incompatible-change-2} +#### Backward incompatible change {#backward-incompatible-change-2} * Upgrade from 20.3 and older to 22.9 and newer should be done through an intermediate version if there are any `ReplicatedMergeTree` tables, otherwise server with the new version will not start. [#40641](https://github.com/ClickHouse/ClickHouse/pull/40641) ([Alexander Tokmakov](https://github.com/tavplubix)). * Remove the functions `accurate_Cast` and `accurate_CastOrNull` (they are different to `accurateCast` and `accurateCastOrNull` by underscore in the name and they are not affected by the value of `cast_keep_nullable` setting). These functions were undocumented, untested, unused, and unneeded. They appeared to be alive due to code generalization. [#40682](https://github.com/ClickHouse/ClickHouse/pull/40682) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Add a test to ensure that every new table function will be documented. See [#40649](https://github.com/ClickHouse/ClickHouse/issues/40649). Rename table function `MeiliSearch` to `meilisearch`. [#40709](https://github.com/ClickHouse/ClickHouse/pull/40709) ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -376,7 +376,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl HTTP. [#40897](https://github.com/ClickHouse/ClickHouse/pull/40897) ([Robert Schulze](https://github.com/rschu1ze)). * Make interpretation of YAML configs to be more conventional. [#41044](https://github.com/ClickHouse/ClickHouse/pull/41044) ([Vitaly Baranov](https://github.com/vitlibar)). -#### New Feature {#new-feature-3} +#### New feature {#new-feature-3} * Support `insert_quorum = 'auto'` to use majority number. [#39970](https://github.com/ClickHouse/ClickHouse/pull/39970) ([Sachin](https://github.com/SachinSetiya)). * Add embedded dashboards to ClickHouse server. This is a demo project about how to achieve 90% results with 1% effort using ClickHouse features. [#40461](https://github.com/ClickHouse/ClickHouse/pull/40461) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Added new settings constraint writability kind `changeable_in_readonly`. [#40631](https://github.com/ClickHouse/ClickHouse/pull/40631) ([Sergei Trifonov](https://github.com/serxa)). @@ -387,13 +387,13 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Implement set operations on Apache Datasketches. [#39919](https://github.com/ClickHouse/ClickHouse/pull/39919) ([Fangyuan Deng](https://github.com/pzhdfy)). Note: there is no point of using Apache Datasketches, they are inferiour than ClickHouse and only make sense for integration with other systems. * Allow recording errors to specified file while reading text formats (`CSV`, `TSV`). [#40516](https://github.com/ClickHouse/ClickHouse/pull/40516) ([zjial](https://github.com/zjial)). -#### Experimental Feature {#experimental-feature-3} +#### Experimental feature {#experimental-feature-3} * Add ANN (approximate nearest neighbor) index based on `Annoy`. [#40818](https://github.com/ClickHouse/ClickHouse/pull/40818) ([Filatenkov Artur](https://github.com/FArthur-cmd)). [#37215](https://github.com/ClickHouse/ClickHouse/pull/37215) ([VVMak](https://github.com/VVMak)). * Add new storage engine `KeeperMap`, that uses ClickHouse Keeper or ZooKeeper as a key-value store. [#39976](https://github.com/ClickHouse/ClickHouse/pull/39976) ([Antonio Andelic](https://github.com/antonio2368)). This storage engine is intended to store a small amount of metadata. * Improvement for in-memory data parts: remove completely processed WAL files. [#40592](https://github.com/ClickHouse/ClickHouse/pull/40592) ([Azat Khuzhin](https://github.com/azat)). -#### Performance Improvement {#performance-improvement-3} +#### Performance improvement {#performance-improvement-3} * Implement compression of marks and primary key. Close [#34437](https://github.com/ClickHouse/ClickHouse/issues/34437). [#37693](https://github.com/ClickHouse/ClickHouse/pull/37693) ([zhongyuankai](https://github.com/zhongyuankai)). * Allow to load marks with threadpool in advance. Regulated by setting `load_marks_asynchronously` (default: 0). [#40821](https://github.com/ClickHouse/ClickHouse/pull/40821) ([Kseniia Sumarokova](https://github.com/kssenii)). * Virtual filesystem over s3 will use random object names split into multiple path prefixes for better performance on AWS. [#40968](https://github.com/ClickHouse/ClickHouse/pull/40968) ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -463,7 +463,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Add a setting `allow_suspicious_fixed_string_types` to prevent users from creating columns of type FixedString with size > 256. [#41495](https://github.com/ClickHouse/ClickHouse/pull/41495) ([Duc Canh Le](https://github.com/canhld94)). * Add `has_lightweight_delete` to system.parts. [#41564](https://github.com/ClickHouse/ClickHouse/pull/41564) ([Kseniia Sumarokova](https://github.com/kssenii)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-3} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-3} * Enforce documentation for every setting. [#40644](https://github.com/ClickHouse/ClickHouse/pull/40644) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Enforce documentation for every current metric. [#40645](https://github.com/ClickHouse/ClickHouse/pull/40645) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Enforce documentation for every profile event counter. Write the documentation where it was missing. [#40646](https://github.com/ClickHouse/ClickHouse/pull/40646) ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -488,7 +488,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Trying stress on top of S3 [#36837](https://github.com/ClickHouse/ClickHouse/pull/36837) ([alesapin](https://github.com/alesapin)). * Enable `concurrency-mt-unsafe` in `clang-tidy` [#40224](https://github.com/ClickHouse/ClickHouse/pull/40224) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### Bug Fix {#bug-fix} +#### Bug fix {#bug-fix} * Fix potential dataloss due to [a bug in AWS SDK](https://github.com/aws/aws-sdk-cpp/issues/658). Bug can be triggered only when clickhouse is used over S3. [#40506](https://github.com/ClickHouse/ClickHouse/pull/40506) ([alesapin](https://github.com/alesapin)). This bug has been open for 5 years in AWS SDK and is closed after our report. * Malicious data in Native format might cause a crash. [#41441](https://github.com/ClickHouse/ClickHouse/pull/41441) ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -573,15 +573,15 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Fix read bytes/rows in X-ClickHouse-Summary with materialized views. [#41586](https://github.com/ClickHouse/ClickHouse/pull/41586) ([Raúl Marín](https://github.com/Algunenano)). * Fix possible `pipeline stuck` exception for queries with `OFFSET`. The error was found with `enable_optimize_predicate_expression = 0` and always false condition in `WHERE`. Fixes [#41383](https://github.com/ClickHouse/ClickHouse/issues/41383). [#41588](https://github.com/ClickHouse/ClickHouse/pull/41588) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). -### ClickHouse release 22.8, 2022-08-18 {#a-id228a-clickhouse-release-228-2022-08-18} +### ClickHouse release 22.8, 2022-08-18 {#a-id228a-clickhouse-release-228-2022-08-18} -#### Backward Incompatible Change {#backward-incompatible-change-3} +#### Backward incompatible change {#backward-incompatible-change-3} * Extended range of `Date32` and `DateTime64` to support dates from the year 1900 to 2299. In previous versions, the supported interval was only from the year 1925 to 2283. The implementation is using the proleptic Gregorian calendar (which is conformant with [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601):2004 (clause 3.2.1 The Gregorian calendar)) instead of accounting for historical transitions from the Julian to the Gregorian calendar. This change affects implementation-specific behavior for out-of-range arguments. E.g. if in previous versions the value of `1899-01-01` was clamped to `1925-01-01`, in the new version it will be clamped to `1900-01-01`. It changes the behavior of rounding with `toStartOfInterval` if you pass `INTERVAL 3 QUARTER` up to one quarter because the intervals are counted from an implementation-specific point of time. Closes [#28216](https://github.com/ClickHouse/ClickHouse/issues/28216), improves [#38393](https://github.com/ClickHouse/ClickHouse/issues/38393). [#39425](https://github.com/ClickHouse/ClickHouse/pull/39425) ([Roman Vasin](https://github.com/rvasin)). * Now, all relevant dictionary sources respect `remote_url_allow_hosts` setting. It was already done for HTTP, Cassandra, Redis. Added ClickHouse, MongoDB, MySQL, PostgreSQL. Host is checked only for dictionaries created from DDL. [#39184](https://github.com/ClickHouse/ClickHouse/pull/39184) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Prebuilt ClickHouse x86 binaries now require support for AVX instructions, i.e. a CPU not older than Intel Sandy Bridge / AMD Bulldozer, both released in 2011. [#39000](https://github.com/ClickHouse/ClickHouse/pull/39000) ([Robert Schulze](https://github.com/rschu1ze)). * Make the remote filesystem cache composable, allow not to evict certain files (regarding idx, mrk, ..), delete old cache version. Now it is possible to configure cache over Azure blob storage disk, over Local disk, over StaticWeb disk, etc. This PR is marked backward incompatible because cache configuration changes and in order for cache to work need to update the config file. Old cache will still be used with new configuration. The server will startup fine with the old cache configuration. Closes https://github.com/ClickHouse/ClickHouse/issues/36140. Closes https://github.com/ClickHouse/ClickHouse/issues/37889. ([Kseniia Sumarokova](https://github.com/kssenii)). [#36171](https://github.com/ClickHouse/ClickHouse/pull/36171)) -#### New Feature {#new-feature-4} +#### New feature {#new-feature-4} * Support SQL standard DELETE FROM syntax on merge tree tables and lightweight delete implementation for merge tree families. [#37893](https://github.com/ClickHouse/ClickHouse/pull/37893) ([Jianmei Zhang](https://github.com/zhangjmruc)) ([Alexander Gololobov](https://github.com/davenger)). Note: this new feature does not make ClickHouse an HTAP DBMS. * Query parameters can be set in interactive mode as `SET param_abc = 'def'` and transferred via the native protocol as settings. [#39906](https://github.com/ClickHouse/ClickHouse/pull/39906) ([Nikita Taranov](https://github.com/nickitat)). * Quota key can be set in the native protocol ([Yakov Olkhovsky](https://github.com/ClickHouse/ClickHouse/pull/39874)). @@ -595,7 +595,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Support `SELECT ... INTO OUTFILE '...' AND STDOUT`. [#37490](https://github.com/ClickHouse/ClickHouse/issues/37490). [#39054](https://github.com/ClickHouse/ClickHouse/pull/39054) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni)). * Add formats `PrettyMonoBlock`, `PrettyNoEscapesMonoBlock`, `PrettyCompactNoEscapes`, `PrettyCompactNoEscapesMonoBlock`, `PrettySpaceNoEscapes`, `PrettySpaceMonoBlock`, `PrettySpaceNoEscapesMonoBlock`. [#39646](https://github.com/ClickHouse/ClickHouse/pull/39646) ([Kruglov Pavel](https://github.com/Avogar)). -#### Performance Improvement {#performance-improvement-4} +#### Performance improvement {#performance-improvement-4} * Improved memory usage during memory efficient merging of aggregation results. [#39429](https://github.com/ClickHouse/ClickHouse/pull/39429) ([Nikita Taranov](https://github.com/nickitat)). * Added concurrency control logic to limit total number of concurrent threads created by queries. [#37558](https://github.com/ClickHouse/ClickHouse/pull/37558) ([Sergei Trifonov](https://github.com/serxa)). Add `concurrent_threads_soft_limit parameter` to increase performance in case of high QPS by means of limiting total number of threads for all queries. [#37285](https://github.com/ClickHouse/ClickHouse/pull/37285) ([Roman Vasin](https://github.com/rvasin)). * Add `SLRU` cache policy for uncompressed cache and marks cache. ([Kseniia Sumarokova](https://github.com/kssenii)). [#34651](https://github.com/ClickHouse/ClickHouse/pull/34651) ([alexX512](https://github.com/alexX512)). Decoupling local cache function and cache algorithm [#38048](https://github.com/ClickHouse/ClickHouse/pull/38048) ([Han Shukai](https://github.com/KinderRiven)). @@ -644,7 +644,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Support hadoop secure RPC transfer (hadoop.rpc.protection=privacy and hadoop.rpc.protection=integrity). [#39411](https://github.com/ClickHouse/ClickHouse/pull/39411) ([michael1589](https://github.com/michael1589)). * Avoid continuously growing memory consumption of pattern cache when using functions multi(Fuzzy)Match(Any|AllIndices|AnyIndex)(). [#40264](https://github.com/ClickHouse/ClickHouse/pull/40264) ([Robert Schulze](https://github.com/rschu1ze)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-4} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-4} * [ClickFiddle](https://fiddle.clickhouse.com/): A new tool for testing ClickHouse versions in read/write mode (**Igor Baliuk**). * ClickHouse binary is made self-extracting [#35775](https://github.com/ClickHouse/ClickHouse/pull/35775) ([Yakov Olkhovskiy, Arthur Filatenkov](https://github.com/yakov-olkhovskiy)). * Update tzdata to 2022b to support the new timezone changes. See https://github.com/google/cctz/pull/226. Chile's 2022 DST start is delayed from September 4 to September 11. Iran plans to stop observing DST permanently, after it falls back on 2022-09-21. There are corrections of the historical time zone of Asia/Tehran in the year 1977: Iran adopted standard time in 1935, not 1946. In 1977 it observed DST from 03-21 23:00 to 10-20 24:00; its 1978 transitions were on 03-24 and 08-05, not 03-20 and 10-20; and its spring 1979 transition was on 05-27, not 03-21 (https://data.iana.org/time-zones/tzdb/NEWS). ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -658,7 +658,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Simplified function registration macro interface (`FUNCTION_REGISTER*`) to eliminate the step to add and call an extern function in the registerFunctions.cpp, it also makes incremental builds of a new function faster. [#38615](https://github.com/ClickHouse/ClickHouse/pull/38615) ([Li Yin](https://github.com/liyinsg)). * Docker: Now entrypoint.sh in docker image creates and executes chown for all folders it found in config for multidisk setup [#17717](https://github.com/ClickHouse/ClickHouse/issues/17717). [#39121](https://github.com/ClickHouse/ClickHouse/pull/39121) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). -#### Bug Fix {#bug-fix-1} +#### Bug fix {#bug-fix-1} * Fix possible segfault in `CapnProto` input format. This bug was found and send through ClickHouse bug-bounty [program](https://github.com/ClickHouse/ClickHouse/issues/38986) by *kiojj*. [#40241](https://github.com/ClickHouse/ClickHouse/pull/40241) ([Kruglov Pavel](https://github.com/Avogar)). * Fix a very rare case of incorrect behavior of array subscript operator. This closes [#28720](https://github.com/ClickHouse/ClickHouse/issues/28720). [#40185](https://github.com/ClickHouse/ClickHouse/pull/40185) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Fix insufficient argument check for encryption functions (found by query fuzzer). This closes [#39987](https://github.com/ClickHouse/ClickHouse/issues/39987). [#40194](https://github.com/ClickHouse/ClickHouse/pull/40194) ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -709,15 +709,15 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * A fix for reverse DNS resolution. [#40134](https://github.com/ClickHouse/ClickHouse/pull/40134) ([Arthur Passos](https://github.com/arthurpassos)). * Fix unexpected result `arrayDifference` of `Array(UInt32). [#40211](https://github.com/ClickHouse/ClickHouse/pull/40211) ([Duc Canh Le](https://github.com/canhld94)). -### ClickHouse release 22.7, 2022-07-21 {#a-id227a-clickhouse-release-227-2022-07-21} +### ClickHouse release 22.7, 2022-07-21 {#a-id227a-clickhouse-release-227-2022-07-21} -#### Upgrade Notes {#upgrade-notes-1} +#### Upgrade notes {#upgrade-notes-1} * Enable setting `enable_positional_arguments` by default. It allows queries like `SELECT ... ORDER BY 1, 2` where 1, 2 are the references to the select clause. If you need to return the old behavior, disable this setting. [#38204](https://github.com/ClickHouse/ClickHouse/pull/38204) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Disable `format_csv_allow_single_quotes` by default. See [#37096](https://github.com/ClickHouse/ClickHouse/issues/37096). ([Kruglov Pavel](https://github.com/Avogar)). * `Ordinary` database engine and old storage definition syntax for `*MergeTree` tables are deprecated. By default it's not possible to create new databases with `Ordinary` engine. If `system` database has `Ordinary` engine it will be automatically converted to `Atomic` on server startup. There are settings to keep old behavior (`allow_deprecated_database_ordinary` and `allow_deprecated_syntax_for_merge_tree`), but these settings may be removed in future releases. [#38335](https://github.com/ClickHouse/ClickHouse/pull/38335) ([Alexander Tokmakov](https://github.com/tavplubix)). * Force rewriting comma join to inner by default (set default value `cross_to_inner_join_rewrite = 2`). To have old behavior set `cross_to_inner_join_rewrite = 1`. [#39326](https://github.com/ClickHouse/ClickHouse/pull/39326) ([Vladimir C](https://github.com/vdimir)). If you will face any incompatibilities, you can turn this setting back. -#### New Feature {#new-feature-5} +#### New feature {#new-feature-5} * Support expressions with window functions. Closes [#19857](https://github.com/ClickHouse/ClickHouse/issues/19857). [#37848](https://github.com/ClickHouse/ClickHouse/pull/37848) ([Dmitry Novik](https://github.com/novikd)). * Add new `direct` join algorithm for `EmbeddedRocksDB` tables, see [#33582](https://github.com/ClickHouse/ClickHouse/issues/33582). [#35363](https://github.com/ClickHouse/ClickHouse/pull/35363) ([Vladimir C](https://github.com/vdimir)). * Added full sorting merge join algorithm. [#35796](https://github.com/ClickHouse/ClickHouse/pull/35796) ([Vladimir C](https://github.com/vdimir)). @@ -744,10 +744,10 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Add ability to pass HTTP headers to the `url` table function / storage via SQL. Closes [#37897](https://github.com/ClickHouse/ClickHouse/issues/37897). [#38176](https://github.com/ClickHouse/ClickHouse/pull/38176) ([Kseniia Sumarokova](https://github.com/kssenii)). * Add `clickhouse-diagnostics` binary to the packages. [#38647](https://github.com/ClickHouse/ClickHouse/pull/38647) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). -#### Experimental Feature {#experimental-feature-4} +#### Experimental feature {#experimental-feature-4} * Adds new setting `implicit_transaction` to run standalone queries inside a transaction. It handles both creation and closing (via COMMIT if the query succeeded or ROLLBACK if it didn't) of the transaction automatically. [#38344](https://github.com/ClickHouse/ClickHouse/pull/38344) ([Raúl Marín](https://github.com/Algunenano)). -#### Performance Improvement {#performance-improvement-5} +#### Performance improvement {#performance-improvement-5} * Distinct optimization for sorted columns. Use specialized distinct transformation in case input stream is sorted by column(s) in distinct. Optimization can be applied to pre-distinct, final distinct, or both. Initial implementation by @dimarub2000. [#37803](https://github.com/ClickHouse/ClickHouse/pull/37803) ([Igor Nikonov](https://github.com/devcrafter)). * Improve performance of `ORDER BY`, `MergeTree` merges, window functions using batch version of `BinaryHeap`. [#38022](https://github.com/ClickHouse/ClickHouse/pull/38022) ([Maksim Kita](https://github.com/kitaisreal)). * More parallel execution for queries with `FINAL` [#36396](https://github.com/ClickHouse/ClickHouse/pull/36396) ([Nikita Taranov](https://github.com/nickitat)). @@ -813,7 +813,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Functions `multiMatch[Fuzzy](AllIndices/Any/AnyIndex)` - don't throw a logical error if the needle argument is empty. [#39012](https://github.com/ClickHouse/ClickHouse/pull/39012) ([Robert Schulze](https://github.com/rschu1ze)). * Allow to declare `RabbitMQ` queue without default arguments `x-max-length` and `x-overflow`. [#39259](https://github.com/ClickHouse/ClickHouse/pull/39259) ([rnbondarenko](https://github.com/rnbondarenko)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-5} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-5} * Apply Clang Thread Safety Analysis (TSA) annotations to ClickHouse. [#38068](https://github.com/ClickHouse/ClickHouse/pull/38068) ([Robert Schulze](https://github.com/rschu1ze)). * Adapt universal installation script for FreeBSD. [#39302](https://github.com/ClickHouse/ClickHouse/pull/39302) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Preparation for building on `s390x` platform. [#39193](https://github.com/ClickHouse/ClickHouse/pull/39193) ([Harry Lee](https://github.com/HarryLeeIBM)). @@ -822,7 +822,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * System table "system.licenses" is now correctly populated on Mac (Darwin). [#38294](https://github.com/ClickHouse/ClickHouse/pull/38294) ([Robert Schulze](https://github.com/rschu1ze)). * Change `all|noarch` packages to architecture-dependent - Fix some documentation for it - Push aarch64|arm64 packages to artifactory and release assets - Fixes [#36443](https://github.com/ClickHouse/ClickHouse/issues/36443). [#38580](https://github.com/ClickHouse/ClickHouse/pull/38580) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). -#### Bug Fix (user-visible misbehavior in official stable or prestable release) {#bug-fix-user-visible-misbehavior-in-official-stable-or-prestable-release-3} +#### Bug fix (user-visible misbehavior in official stable or prestable release) {#bug-fix-user-visible-misbehavior-in-official-stable-or-prestable-release-3} * Fix rounding for `Decimal128/Decimal256` with more than 19-digits long scale. [#38027](https://github.com/ClickHouse/ClickHouse/pull/38027) ([Igor Nikonov](https://github.com/devcrafter)). * Fixed crash caused by data race in storage `Hive` (integration table engine). [#38887](https://github.com/ClickHouse/ClickHouse/pull/38887) ([lgbo](https://github.com/lgbo-ustc)). * Fix crash when executing GRANT ALL ON *.* with ON CLUSTER. It was broken in https://github.com/ClickHouse/ClickHouse/pull/35767. This closes [#38618](https://github.com/ClickHouse/ClickHouse/issues/38618). [#38674](https://github.com/ClickHouse/ClickHouse/pull/38674) ([Vitaly Baranov](https://github.com/vitlibar)). @@ -876,9 +876,9 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Fix `parallel_view_processing=1` with `optimize_trivial_insert_select=1`. Fix `max_insert_threads` while pushing to views. [#38731](https://github.com/ClickHouse/ClickHouse/pull/38731) ([Azat Khuzhin](https://github.com/azat)). * Fix use-after-free for aggregate functions with `Map` combinator that leads to incorrect result. [#38748](https://github.com/ClickHouse/ClickHouse/pull/38748) ([Azat Khuzhin](https://github.com/azat)). -### ClickHouse release 22.6, 2022-06-16 {#a-id226a-clickhouse-release-226-2022-06-16} +### ClickHouse release 22.6, 2022-06-16 {#a-id226a-clickhouse-release-226-2022-06-16} -#### Backward Incompatible Change {#backward-incompatible-change-4} +#### Backward incompatible change {#backward-incompatible-change-4} * Remove support for octal number literals in SQL. In previous versions they were parsed as Float64. [#37765](https://github.com/ClickHouse/ClickHouse/pull/37765) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). * Changes how settings using `seconds` as type are parsed to support floating point values (for example: `max_execution_time=0.5`). Infinity or NaN values will throw an exception. [#37187](https://github.com/ClickHouse/ClickHouse/pull/37187) ([Raúl Marín](https://github.com/Algunenano)). * Changed format of binary serialization of columns of experimental type `Object`. New format is more convenient to implement by third-party clients. [#37482](https://github.com/ClickHouse/ClickHouse/pull/37482) ([Anton Popov](https://github.com/CurtizJ)). @@ -886,7 +886,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * LIKE patterns with trailing escape symbol ('\\') are now disallowed (as mandated by the SQL standard). [#37764](https://github.com/ClickHouse/ClickHouse/pull/37764) ([Robert Schulze](https://github.com/rschu1ze)). * If you run different ClickHouse versions on a cluster with AArch64 CPU or mix AArch64 and amd64 on a cluster, and use distributed queries with GROUP BY multiple keys of fixed-size type that fit in 256 bits but don't fit in 64 bits, and the size of the result is huge, the data will not be fully aggregated in the result of these queries during upgrade. Workaround: upgrade with downtime instead of a rolling upgrade. -#### New Feature {#new-feature-6} +#### New feature {#new-feature-6} * Add `GROUPING` function. It allows to disambiguate the records in the queries with `ROLLUP`, `CUBE` or `GROUPING SETS`. Closes [#19426](https://github.com/ClickHouse/ClickHouse/issues/19426). [#37163](https://github.com/ClickHouse/ClickHouse/pull/37163) ([Dmitry Novik](https://github.com/novikd)). * A new codec [FPC](https://userweb.cs.txstate.edu/~burtscher/papers/dcc07a.pdf) algorithm for floating point data compression. [#37553](https://github.com/ClickHouse/ClickHouse/pull/37553) ([Mikhail Guzov](https://github.com/koloshmet)). * Add new columnar JSON formats: `JSONColumns`, `JSONCompactColumns`, `JSONColumnsWithMetadata`. Closes [#36338](https://github.com/ClickHouse/ClickHouse/issues/36338) Closes [#34509](https://github.com/ClickHouse/ClickHouse/issues/34509). [#36975](https://github.com/ClickHouse/ClickHouse/pull/36975) ([Kruglov Pavel](https://github.com/Avogar)). @@ -907,12 +907,12 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Implemented changing the comment for `ReplicatedMergeTree` tables. [#37416](https://github.com/ClickHouse/ClickHouse/pull/37416) ([Vasily Nemkov](https://github.com/Enmk)). * Added `SYSTEM UNFREEZE` query that deletes the whole backup regardless if the corresponding table is deleted or not. [#36424](https://github.com/ClickHouse/ClickHouse/pull/36424) ([Vadim Volodin](https://github.com/PolyProgrammist)). -#### Experimental Feature {#experimental-feature-5} +#### Experimental feature {#experimental-feature-5} * Enables `POPULATE` for `WINDOW VIEW`. [#36945](https://github.com/ClickHouse/ClickHouse/pull/36945) ([vxider](https://github.com/Vxider)). * `ALTER TABLE ... MODIFY QUERY` support for `WINDOW VIEW`. [#37188](https://github.com/ClickHouse/ClickHouse/pull/37188) ([vxider](https://github.com/Vxider)). * This PR changes the behavior of the `ENGINE` syntax in `WINDOW VIEW`, to make it like in `MATERIALIZED VIEW`. [#37214](https://github.com/ClickHouse/ClickHouse/pull/37214) ([vxider](https://github.com/Vxider)). -#### Performance Improvement {#performance-improvement-6} +#### Performance improvement {#performance-improvement-6} * Added numerous optimizations for ARM NEON [#38093](https://github.com/ClickHouse/ClickHouse/pull/38093)([Daniel Kutenin](https://github.com/danlark1)), ([Alexandra Pilipyuk](https://github.com/chalice19)) Note: if you run different ClickHouse versions on a cluster with ARM CPU and use distributed queries with GROUP BY multiple keys of fixed-size type that fit in 256 bits but don't fit in 64 bits, the result of the aggregation query will be wrong during upgrade. Workaround: upgrade with downtime instead of a rolling upgrade. * Improve performance and memory usage for select of subset of columns for formats Native, Protobuf, CapnProto, JSONEachRow, TSKV, all formats with suffixes WithNames/WithNamesAndTypes. Previously while selecting only subset of columns from files in these formats all columns were read and stored in memory. Now only required columns are read. This PR enables setting `input_format_skip_unknown_fields` by default, because otherwise in case of select of subset of columns exception will be thrown. [#37192](https://github.com/ClickHouse/ClickHouse/pull/37192) ([Kruglov Pavel](https://github.com/Avogar)). * Now more filters can be pushed down for join. [#37472](https://github.com/ClickHouse/ClickHouse/pull/37472) ([Amos Bird](https://github.com/amosbird)). @@ -983,13 +983,13 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Now if setting `always_fetch_merged_part` is enabled for ReplicatedMergeTree merges will try to find parts on other replicas rarely with smaller load for [Zoo]Keeper. [#37995](https://github.com/ClickHouse/ClickHouse/pull/37995) ([alesapin](https://github.com/alesapin)). * Add implicit grants with grant option too. For example `GRANT CREATE TABLE ON test.* TO A WITH GRANT OPTION` now allows `A` to execute `GRANT CREATE VIEW ON test.* TO B`. [#38017](https://github.com/ClickHouse/ClickHouse/pull/38017) ([Vitaly Baranov](https://github.com/vitlibar)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-6} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-6} * Use `clang-14` and LLVM infrastructure version 14 for builds. This closes [#34681](https://github.com/ClickHouse/ClickHouse/issues/34681). [#34754](https://github.com/ClickHouse/ClickHouse/pull/34754) ([Alexey Milovidov](https://github.com/alexey-milovidov)). Note: `clang-14` has [a bug](https://github.com/google/sanitizers/issues/1540) in ThreadSanitizer that makes our CI work worse. * Allow to drop privileges at startup. This simplifies Docker images. Closes [#36293](https://github.com/ClickHouse/ClickHouse/issues/36293). [#36341](https://github.com/ClickHouse/ClickHouse/pull/36341) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Add docs spellcheck to CI. [#37790](https://github.com/ClickHouse/ClickHouse/pull/37790) ([Vladimir C](https://github.com/vdimir)). * Fix overly aggressive stripping which removed the embedded hash required for checking the consistency of the executable. [#37993](https://github.com/ClickHouse/ClickHouse/pull/37993) ([Robert Schulze](https://github.com/rschu1ze)). -#### Bug Fix {#bug-fix-2} +#### Bug fix {#bug-fix-2} * Fix `SELECT ... INTERSECT` and `EXCEPT SELECT` statements with constant string types. [#37738](https://github.com/ClickHouse/ClickHouse/pull/37738) ([Antonio Andelic](https://github.com/antonio2368)). * Fix `GROUP BY` `AggregateFunction` (i.e. you `GROUP BY` by the column that has `AggregateFunction` type). [#37093](https://github.com/ClickHouse/ClickHouse/pull/37093) ([Azat Khuzhin](https://github.com/azat)). @@ -1041,9 +1041,9 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Fix possible heap-use-after-free error when reading system.projection_parts and system.projection_parts_columns . This fixes [#37184](https://github.com/ClickHouse/ClickHouse/issues/37184). [#37185](https://github.com/ClickHouse/ClickHouse/pull/37185) ([Amos Bird](https://github.com/amosbird)). * Fixed `DateTime64` fractional seconds behavior prior to Unix epoch. [#37697](https://github.com/ClickHouse/ClickHouse/pull/37697) ([Andrey Zvonov](https://github.com/zvonand)). [#37039](https://github.com/ClickHouse/ClickHouse/pull/37039) ([李扬](https://github.com/taiyang-li)). -### ClickHouse release 22.5, 2022-05-19 {#a-id225a-clickhouse-release-225-2022-05-19} +### ClickHouse release 22.5, 2022-05-19 {#a-id225a-clickhouse-release-225-2022-05-19} -#### Upgrade Notes {#upgrade-notes-2} +#### Upgrade notes {#upgrade-notes-2} * Now, background merges, mutations and `OPTIMIZE` will not increment `SelectedRows` and `SelectedBytes` metrics. They (still) will increment `MergedRows` and `MergedUncompressedBytes` as it was before. This only affects the metric values, and makes them better. This change does not introduce any incompatibility, but you may wonder about the changes of metrics, so we put in this category. [#37040](https://github.com/ClickHouse/ClickHouse/pull/37040) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Updated the BoringSSL module to the official FIPS compliant version. This makes ClickHouse FIPS compliant. [#35914](https://github.com/ClickHouse/ClickHouse/pull/35914) ([Meena-Renganathan](https://github.com/Meena-Renganathan)). The ciphers `aes-192-cfb128` and `aes-256-cfb128` were removed, because they are not included in the FIPS certified version of BoringSSL. @@ -1051,7 +1051,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Disable `log_query_threads` setting by default. It controls the logging of statistics about every thread participating in query execution. After supporting asynchronous reads, the total number of distinct thread ids became too large, and logging into the `query_thread_log` has become too heavy. [#37077](https://github.com/ClickHouse/ClickHouse/pull/37077) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Remove function `groupArraySorted` which has a bug. [#36822](https://github.com/ClickHouse/ClickHouse/pull/36822) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### New Feature {#new-feature-7} +#### New feature {#new-feature-7} * Enable memory overcommit by default. [#35921](https://github.com/ClickHouse/ClickHouse/pull/35921) ([Dmitry Novik](https://github.com/novikd)). * Add support of GROUPING SETS in GROUP BY clause. This implementation supports a parallel processing of grouping sets. [#33631](https://github.com/ClickHouse/ClickHouse/pull/33631) ([Dmitry Novik](https://github.com/novikd)). @@ -1063,7 +1063,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Show the `total_rows` and `total_bytes` fields in `system.tables` for temporary tables. [#36401](https://github.com/ClickHouse/ClickHouse/issues/36401). [#36439](https://github.com/ClickHouse/ClickHouse/pull/36439) ([xiedeyantu](https://github.com/xiedeyantu)). * Allow to override `parts_to_delay_insert` and `parts_to_throw_insert` with query-level settings. If they are defined, they will override table-level settings. [#36371](https://github.com/ClickHouse/ClickHouse/pull/36371) ([Memo](https://github.com/Joeywzr)). -#### Experimental Feature {#experimental-feature-6} +#### Experimental feature {#experimental-feature-6} * Implemented L1, L2, Linf, Cosine distance functions for arrays and L1, L2, Linf norm functions for arrays. [#37033](https://github.com/ClickHouse/ClickHouse/pull/37033) ([qieqieplus](https://github.com/qieqieplus)). Caveat: the functions will be renamed. @@ -1075,7 +1075,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Improvement for semistructured data: Allow to cast columns of type `Object(...)` to `Object(Nullable(...))`. [#36564](https://github.com/ClickHouse/ClickHouse/pull/36564) ([awakeljw](https://github.com/awakeljw)). * Improvement for parallel replicas: We create a local interpreter if we want to execute query on localhost replica. But for when executing query on multiple replicas we rely on the fact that a connection exists so replicas can talk to coordinator. It is now improved and localhost replica can talk to coordinator directly in the same process. [#36281](https://github.com/ClickHouse/ClickHouse/pull/36281) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). -#### Performance Improvement {#performance-improvement-7} +#### Performance improvement {#performance-improvement-7} * Improve performance of `avg`, `sum` aggregate functions if used without GROUP BY expression. [#37257](https://github.com/ClickHouse/ClickHouse/pull/37257) ([Maksim Kita](https://github.com/kitaisreal)). * Improve performance of unary arithmetic functions (`bitCount`, `bitNot`, `abs`, `intExp2`, `intExp10`, `negate`, `roundAge`, `roundDuration`, `roundToExp2`, `sign`) using dynamic dispatch. [#37289](https://github.com/ClickHouse/ClickHouse/pull/37289) ([Maksim Kita](https://github.com/kitaisreal)). @@ -1134,7 +1134,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Improve JSON report of clickhouse-benchmark. [#36473](https://github.com/ClickHouse/ClickHouse/pull/36473) ([Tian Xinhui](https://github.com/xinhuitian)). * Server might refuse to start if it cannot resolve hostname of external ClickHouse dictionary. It's fixed. Fixes [#36451](https://github.com/ClickHouse/ClickHouse/issues/36451). [#36463](https://github.com/ClickHouse/ClickHouse/pull/36463) ([tavplubix](https://github.com/tavplubix)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-7} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-7} * Now `clickhouse-keeper` for the `x86_64` architecture is statically linked with [musl](https://musl.libc.org/) and doesn't depend on any system libraries. [#31833](https://github.com/ClickHouse/ClickHouse/pull/31833) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * ClickHouse builds for `PowerPC64LE` architecture are now available in universal installation script `curl https://clickhouse.com/ | sh` and by direct link `https://builds.clickhouse.com/master/powerpc64le/clickhouse`. [#37095](https://github.com/ClickHouse/ClickHouse/pull/37095) ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -1143,7 +1143,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Fail performance comparison on errors in the report. [#34797](https://github.com/ClickHouse/ClickHouse/pull/34797) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). * Add ZSTD support for Arrow. This fixes [#35283](https://github.com/ClickHouse/ClickHouse/issues/35283). [#35486](https://github.com/ClickHouse/ClickHouse/pull/35486) ([Sean Lafferty](https://github.com/seanlaff)). -#### Bug Fix {#bug-fix-3} +#### Bug fix {#bug-fix-3} * Extracts Version ID if present from the URI and adds a request to the AWS HTTP URI. Closes [#31221](https://github.com/ClickHouse/ClickHouse/issues/31221). - [x] Extract `Version ID` from URI if present and reassemble without it. - [x] Configure `AWS HTTP URI` object with request. - [x] Unit Tests: [`gtest_s3_uri`](https://github.com/ClickHouse/ClickHouse/blob/2340a6c6849ebc05a8efbf97ba8de3ff9dc0eff4/src/IO/tests/gtest_s3_uri.cpp) - [x] Drop instrumentation commit. [#34571](https://github.com/ClickHouse/ClickHouse/pull/34571) ([Saad Ur Rahman](https://github.com/surahman)). * Fix system.opentelemetry_span_log attribute.values alias to values instead of keys. [#37275](https://github.com/ClickHouse/ClickHouse/pull/37275) ([Aleksandr Razumov](https://github.com/ernado)). @@ -1202,14 +1202,14 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Fix ALTER DROP COLUMN of nested column with compact parts (i.e. `ALTER TABLE x DROP COLUMN n`, when there is column `n.d`). [#35797](https://github.com/ClickHouse/ClickHouse/pull/35797) ([Azat Khuzhin](https://github.com/azat)). * Fix substring function range error length when `offset` and `length` is negative constant and `s` is not constant. [#33861](https://github.com/ClickHouse/ClickHouse/pull/33861) ([RogerYK](https://github.com/RogerYK)). -### ClickHouse release 22.4, 2022-04-19 {#a-id224a-clickhouse-release-224-2022-04-19} +### ClickHouse release 22.4, 2022-04-19 {#a-id224a-clickhouse-release-224-2022-04-19} -#### Backward Incompatible Change {#backward-incompatible-change-5} +#### Backward incompatible change {#backward-incompatible-change-5} * Do not allow SETTINGS after FORMAT for INSERT queries (there is compatibility setting `allow_settings_after_format_in_insert` to accept such queries, but it is turned OFF by default). [#35883](https://github.com/ClickHouse/ClickHouse/pull/35883) ([Azat Khuzhin](https://github.com/azat)). * Function `yandexConsistentHash` (consistent hashing algorithm by Konstantin "kostik" Oblakov) is renamed to `kostikConsistentHash`. The old name is left as an alias for compatibility. Although this change is backward compatible, we may remove the alias in subsequent releases, that's why it's recommended to update the usages of this function in your apps. [#35553](https://github.com/ClickHouse/ClickHouse/pull/35553) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### New Feature {#new-feature-8} +#### New feature {#new-feature-8} * Added INTERPOLATE extension to the ORDER BY ... WITH FILL. Closes [#34903](https://github.com/ClickHouse/ClickHouse/issues/34903). [#35349](https://github.com/ClickHouse/ClickHouse/pull/35349) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). * Profiling on Processors level (under `log_processors_profiles` setting, ClickHouse will write time that processor spent during execution/waiting for data to `system.processors_profile_log` table). [#34355](https://github.com/ClickHouse/ClickHouse/pull/34355) ([Azat Khuzhin](https://github.com/azat)). @@ -1228,14 +1228,14 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Added sanity checks on server startup (available memory and disk space, max thread count, etc). [#34566](https://github.com/ClickHouse/ClickHouse/pull/34566) ([Sergei Trifonov](https://github.com/serxa)). * INTERVAL improvement - can be used with `[MILLI|MICRO|NANO]SECOND`. Added `toStartOf[Milli|Micro|Nano]second()` functions. Added `[add|subtract][Milli|Micro|Nano]seconds()`. [#34353](https://github.com/ClickHouse/ClickHouse/pull/34353) ([Andrey Zvonov](https://github.com/zvonand)). -#### Experimental Feature {#experimental-feature-7} +#### Experimental feature {#experimental-feature-7} * Added support for transactions for simple `MergeTree` tables. This feature is highly experimental and not recommended for production. Part of [#22086](https://github.com/ClickHouse/ClickHouse/issues/22086). [#24258](https://github.com/ClickHouse/ClickHouse/pull/24258) ([tavplubix](https://github.com/tavplubix)). * Support schema inference for type `Object` in format `JSONEachRow`. Allow to convert columns of type `Map` to columns of type `Object`. [#35629](https://github.com/ClickHouse/ClickHouse/pull/35629) ([Anton Popov](https://github.com/CurtizJ)). * Allow to write remote FS cache on all write operations. Add `system.remote_filesystem_cache` table. Add `drop remote filesystem cache` query. Add introspection for s3 metadata with `system.remote_data_paths` table. Closes [#34021](https://github.com/ClickHouse/ClickHouse/issues/34021). Add cache option for merges by adding mode `read_from_filesystem_cache_if_exists_otherwise_bypass_cache` (turned on by default for merges and can also be turned on by query setting with the same name). Rename cache related settings (`remote_fs_enable_cache -> enable_filesystem_cache`, etc). [#35475](https://github.com/ClickHouse/ClickHouse/pull/35475) ([Kseniia Sumarokova](https://github.com/kssenii)). * An option to store parts metadata in RocksDB. Speed up parts loading process of MergeTree to accelerate starting up of clickhouse-server. With this improvement, clickhouse-server was able to decrease starting up time from 75 minutes to 20 seconds, with 700k mergetree parts. [#32928](https://github.com/ClickHouse/ClickHouse/pull/32928) ([李扬](https://github.com/taiyang-li)). -#### Performance Improvement {#performance-improvement-8} +#### Performance improvement {#performance-improvement-8} * A new query plan optimization. Evaluate functions after `ORDER BY` when possible. As an example, for a query `SELECT sipHash64(number) FROM numbers(1e8) ORDER BY number LIMIT 5`, function `sipHash64` would be evaluated after `ORDER BY` and `LIMIT`, which gives ~20x speed up. [#35623](https://github.com/ClickHouse/ClickHouse/pull/35623) ([Nikita Taranov](https://github.com/nickitat)). * Sizes of hash tables used during aggregation now collected and used in later queries to avoid hash tables resizes. [#33439](https://github.com/ClickHouse/ClickHouse/pull/33439) ([Nikita Taranov](https://github.com/nickitat)). @@ -1290,7 +1290,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Proper support of setting `max_rows_to_read` in case of reading in order of sorting key and specified limit. Previously the exception `Limit for rows or bytes to read exceeded` could be thrown even if query actually requires to read less amount of rows. [#33230](https://github.com/ClickHouse/ClickHouse/pull/33230) ([Anton Popov](https://github.com/CurtizJ)). * Respect only quota & period from cgroups, ignore shares (which are not really limit the number of the cores which can be used). [#35815](https://github.com/ClickHouse/ClickHouse/pull/35815) ([filimonov](https://github.com/filimonov)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-8} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-8} * Add next batch of randomization settings in functional tests. [#35047](https://github.com/ClickHouse/ClickHouse/pull/35047) ([Kruglov Pavel](https://github.com/Avogar)). * Add backward compatibility check in stress test. Closes [#25088](https://github.com/ClickHouse/ClickHouse/issues/25088). [#27928](https://github.com/ClickHouse/ClickHouse/pull/27928) ([Kruglov Pavel](https://github.com/Avogar)). @@ -1306,7 +1306,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Switch to libcxx / libcxxabi from LLVM 14. [#34906](https://github.com/ClickHouse/ClickHouse/pull/34906) ([Raúl Marín](https://github.com/Algunenano)). * Update unixodbc to mitigate CVE-2018-7485. Note: this CVE is not relevant for ClickHouse as it implements its own isolation layer for ODBC. [#35943](https://github.com/ClickHouse/ClickHouse/pull/35943) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). -#### Bug Fix {#bug-fix-4} +#### Bug fix {#bug-fix-4} * Added settings `input_format_ipv4_default_on_conversion_error`, `input_format_ipv6_default_on_conversion_error` to allow insert of invalid ip address values as default into tables. Closes [#35726](https://github.com/ClickHouse/ClickHouse/issues/35726). [#35733](https://github.com/ClickHouse/ClickHouse/pull/35733) ([Maksim Kita](https://github.com/kitaisreal)). * Avoid erasing columns from a block if it doesn't exist while reading data from Hive. [#35393](https://github.com/ClickHouse/ClickHouse/pull/35393) ([lgbo](https://github.com/lgbo-ustc)). @@ -1353,15 +1353,15 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Fix mutations in tables with enabled sparse columns. [#35284](https://github.com/ClickHouse/ClickHouse/pull/35284) ([Anton Popov](https://github.com/CurtizJ)). * Do not delay final part writing by default (fixes possible `Memory limit exceeded` during `INSERT` by adding `max_insert_delayed_streams_for_parallel_write` with default to 1000 for writes to s3 and disabled as before otherwise). [#34780](https://github.com/ClickHouse/ClickHouse/pull/34780) ([Azat Khuzhin](https://github.com/azat)). -### ClickHouse release v22.3-lts, 2022-03-17 {#a-id223a-clickhouse-release-v223-lts-2022-03-17} +### ClickHouse release v22.3-lts, 2022-03-17 {#a-id223a-clickhouse-release-v223-lts-2022-03-17} -#### Backward Incompatible Change {#backward-incompatible-change-6} +#### Backward incompatible change {#backward-incompatible-change-6} * Make `arrayCompact` function behave as other higher-order functions: perform compaction not of lambda function results but on the original array. If you're using nontrivial lambda functions in arrayCompact you may restore old behaviour by wrapping `arrayCompact` arguments into `arrayMap`. Closes [#34010](https://github.com/ClickHouse/ClickHouse/issues/34010) [#18535](https://github.com/ClickHouse/ClickHouse/issues/18535) [#14778](https://github.com/ClickHouse/ClickHouse/issues/14778). [#34795](https://github.com/ClickHouse/ClickHouse/pull/34795) ([Alexandre Snarskii](https://github.com/snar)). * Change implementation specific behavior on overflow of function `toDatetime`. It will be saturated to the nearest min/max supported instant of datetime instead of wraparound. This change is highlighted as "backward incompatible" because someone may unintentionally rely on the old behavior. [#32898](https://github.com/ClickHouse/ClickHouse/pull/32898) ([HaiBo Li](https://github.com/marising)). * Make function `cast(value, 'IPv4')`, `cast(value, 'IPv6')` behave same as `toIPv4`, `toIPv6` functions. Changed behavior of incorrect IP address passed into functions `toIPv4`,` toIPv6`, now if invalid IP address passes into this functions exception will be raised, before this function return default value. Added functions `IPv4StringToNumOrDefault`, `IPv4StringToNumOrNull`, `IPv6StringToNumOrDefault`, `IPv6StringOrNull` `toIPv4OrDefault`, `toIPv4OrNull`, `toIPv6OrDefault`, `toIPv6OrNull`. Functions `IPv4StringToNumOrDefault `, `toIPv4OrDefault `, `toIPv6OrDefault ` should be used if previous logic relied on `IPv4StringToNum`, `toIPv4`, `toIPv6` returning default value for invalid address. Added setting `cast_ipv4_ipv6_default_on_conversion_error`, if this setting enabled, then IP address conversion functions will behave as before. Closes [#22825](https://github.com/ClickHouse/ClickHouse/issues/22825). Closes [#5799](https://github.com/ClickHouse/ClickHouse/issues/5799). Closes [#35156](https://github.com/ClickHouse/ClickHouse/issues/35156). [#35240](https://github.com/ClickHouse/ClickHouse/pull/35240) ([Maksim Kita](https://github.com/kitaisreal)). -#### New Feature {#new-feature-9} +#### New feature {#new-feature-9} * Support for caching data locally for remote filesystems. It can be enabled for `s3` disks. Closes [#28961](https://github.com/ClickHouse/ClickHouse/issues/28961). [#33717](https://github.com/ClickHouse/ClickHouse/pull/33717) ([Kseniia Sumarokova](https://github.com/kssenii)). In the meantime, we enabled the test suite on s3 filesystem and no more known issues exist, so it is started to be production ready. * Add new table function `hive`. It can be used as follows `hive('', '', '', '', '')` for example `SELECT * FROM hive('thrift://hivetest:9083', 'test', 'demo', 'id Nullable(String), score Nullable(Int32), day Nullable(String)', 'day')`. [#34946](https://github.com/ClickHouse/ClickHouse/pull/34946) ([lgbo](https://github.com/lgbo-ustc)). @@ -1373,12 +1373,12 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Add three functions for Map data type: 1. `mapReplace(map1, map2)` - replaces values for keys in map1 with the values of the corresponding keys in map2; adds keys from map2 that don't exist in map1. 2. `mapFilter` 3. `mapMap`. mapFilter and mapMap are higher order functions, accepting two arguments, the first argument is a lambda function with k, v pair as arguments, the second argument is a column of type Map. [#33698](https://github.com/ClickHouse/ClickHouse/pull/33698) ([hexiaoting](https://github.com/hexiaoting)). * Allow getting default user and password for clickhouse-client from the `CLICKHOUSE_USER` and `CLICKHOUSE_PASSWORD` environment variables. Close [#34538](https://github.com/ClickHouse/ClickHouse/issues/34538). [#34947](https://github.com/ClickHouse/ClickHouse/pull/34947) ([DR](https://github.com/freedomDR)). -#### Experimental Feature {#experimental-feature-8} +#### Experimental feature {#experimental-feature-8} * New data type `Object()`, which supports storing of semi-structured data (for now JSON only). Data is written to such types as string. Then all paths are extracted according to format of semi-structured data and written as separate columns in most optimal types, that can store all their values. Those columns can be queried by names that match paths in source data. E.g `data.key1.key2` or with cast operator `data.key1.key2::Int64`. * Add `database_replicated_allow_only_replicated_engine` setting. When enabled, it only allowed to only create `Replicated` tables or tables with stateless engines in `Replicated` databases. [#35214](https://github.com/ClickHouse/ClickHouse/pull/35214) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). Note that `Replicated` database is still an experimental feature. -#### Performance Improvement {#performance-improvement-9} +#### Performance improvement {#performance-improvement-9} * Improve performance of insertion into `MergeTree` tables by optimizing sorting. Up to 2x improvement is observed on realistic benchmarks. [#34750](https://github.com/ClickHouse/ClickHouse/pull/34750) ([Maksim Kita](https://github.com/kitaisreal)). * Columns pruning when reading Parquet, ORC and Arrow files from URL and S3. Closes [#34163](https://github.com/ClickHouse/ClickHouse/issues/34163). [#34849](https://github.com/ClickHouse/ClickHouse/pull/34849) ([Kseniia Sumarokova](https://github.com/kssenii)). @@ -1426,7 +1426,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Improve OpenTelemetry span logs for INSERT operation on distributed table. [#34480](https://github.com/ClickHouse/ClickHouse/pull/34480) ([Frank Chen](https://github.com/FrankChen021)). * Make the znode `ctime` and `mtime` consistent between servers in ClickHouse Keeper. [#33441](https://github.com/ClickHouse/ClickHouse/pull/33441) ([小路](https://github.com/nicelulu)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-9} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-9} * Package repository is migrated to JFrog Artifactory (**Mikhail f. Shiryaev**). * Randomize some settings in functional tests, so more possible combinations of settings will be tested. This is yet another fuzzing method to ensure better test coverage. This closes [#32268](https://github.com/ClickHouse/ClickHouse/issues/32268). [#34092](https://github.com/ClickHouse/ClickHouse/pull/34092) ([Kruglov Pavel](https://github.com/Avogar)). @@ -1437,7 +1437,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Limit DWARF version for debug info by 4 max, because our internal stack symbolizer cannot parse DWARF version 5. This makes sense if you compile ClickHouse with clang-15. [#34777](https://github.com/ClickHouse/ClickHouse/pull/34777) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Remove `clickhouse-test` debian package as unneeded complication. CI use tests from repository and standalone testing via deb package is no longer supported. [#34606](https://github.com/ClickHouse/ClickHouse/pull/34606) ([Ilya Yatsishin](https://github.com/qoega)). -#### Bug Fix (user-visible misbehaviour in official stable or prestable release) {#bug-fix-user-visible-misbehaviour-in-official-stable-or-prestable-release} +#### Bug fix (user-visible misbehaviour in official stable or prestable release) {#bug-fix-user-visible-misbehaviour-in-official-stable-or-prestable-release} * A fix for HDFS integration: When the inner buffer size is too small, NEED_MORE_INPUT in `HadoopSnappyDecoder` will run multi times (>=3) for one compressed block. This makes the input data be copied into the wrong place in `HadoopSnappyDecoder::buffer`. [#35116](https://github.com/ClickHouse/ClickHouse/pull/35116) ([lgbo](https://github.com/lgbo-ustc)). * Ignore obsolete grants in ATTACH GRANT statements. This PR fixes [#34815](https://github.com/ClickHouse/ClickHouse/issues/34815). [#34855](https://github.com/ClickHouse/ClickHouse/pull/34855) ([Vitaly Baranov](https://github.com/vitlibar)). @@ -1480,13 +1480,13 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Fix incorrect result of trivial count query when part movement feature is used [#34089](https://github.com/ClickHouse/ClickHouse/issues/34089). [#34385](https://github.com/ClickHouse/ClickHouse/pull/34385) ([nvartolomei](https://github.com/nvartolomei)). * Fix inconsistency of `max_query_size` limitation in distributed subqueries. [#34078](https://github.com/ClickHouse/ClickHouse/pull/34078) ([Chao Ma](https://github.com/godliness)). -### ClickHouse release v22.2, 2022-02-17 {#a-id222a-clickhouse-release-v222-2022-02-17} +### ClickHouse release v22.2, 2022-02-17 {#a-id222a-clickhouse-release-v222-2022-02-17} -#### Upgrade Notes {#upgrade-notes-3} +#### Upgrade notes {#upgrade-notes-3} * Applying data skipping indexes for queries with FINAL may produce incorrect result. In this release we disabled data skipping indexes by default for queries with FINAL (a new setting `use_skip_indexes_if_final` is introduced and disabled by default). [#34243](https://github.com/ClickHouse/ClickHouse/pull/34243) ([Azat Khuzhin](https://github.com/azat)). -#### New Feature {#new-feature-10} +#### New feature {#new-feature-10} * Projections are production ready. Set `allow_experimental_projection_optimization` by default and deprecate this setting. [#34456](https://github.com/ClickHouse/ClickHouse/pull/34456) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * An option to create a new files on insert for `File`/`S3`/`HDFS` engines. Allow to overwrite a file in `HDFS`. Throw an exception in attempt to overwrite a file in `S3` by default. Throw an exception in attempt to append data to file in formats that have a suffix (and thus don't support appends, like `Parquet`, `ORC`). Closes [#31640](https://github.com/ClickHouse/ClickHouse/issues/31640) Closes [#31622](https://github.com/ClickHouse/ClickHouse/issues/31622) Closes [#23862](https://github.com/ClickHouse/ClickHouse/issues/23862) Closes [#15022](https://github.com/ClickHouse/ClickHouse/issues/15022) Closes [#16674](https://github.com/ClickHouse/ClickHouse/issues/16674). [#33302](https://github.com/ClickHouse/ClickHouse/pull/33302) ([Kruglov Pavel](https://github.com/Avogar)). @@ -1509,7 +1509,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Add function `addressToLineWithInlines`. Close [#26211](https://github.com/ClickHouse/ClickHouse/issues/26211). [#33467](https://github.com/ClickHouse/ClickHouse/pull/33467) ([SuperDJY](https://github.com/cmsxbc)). * Added `#!` and `# ` as a recognised start of a single line comment. Closes [#34138](https://github.com/ClickHouse/ClickHouse/issues/34138). [#34230](https://github.com/ClickHouse/ClickHouse/pull/34230) ([Aaron Katz](https://github.com/aaronstephenkatz)). -#### Experimental Feature {#experimental-feature-9} +#### Experimental feature {#experimental-feature-9} * Functions for text classification: language and charset detection. See [#23271](https://github.com/ClickHouse/ClickHouse/issues/23271). [#33314](https://github.com/ClickHouse/ClickHouse/pull/33314) ([Nikolay Degterinsky](https://github.com/evillique)). * Add memory overcommit to `MemoryTracker`. Added `guaranteed` settings for memory limits which represent soft memory limits. In case when hard memory limit is reached, `MemoryTracker` tries to cancel the most overcommited query. New setting `memory_usage_overcommit_max_wait_microseconds` specifies how long queries may wait another query to stop. Closes [#28375](https://github.com/ClickHouse/ClickHouse/issues/28375). [#31182](https://github.com/ClickHouse/ClickHouse/pull/31182) ([Dmitry Novik](https://github.com/novikd)). @@ -1517,7 +1517,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Support `SET`, `YEAR`, `TIME` and `GEOMETRY` data types in `MaterializedMySQL` (experimental feature). Fixes [#18091](https://github.com/ClickHouse/ClickHouse/issues/18091), [#21536](https://github.com/ClickHouse/ClickHouse/issues/21536), [#26361](https://github.com/ClickHouse/ClickHouse/issues/26361). [#33429](https://github.com/ClickHouse/ClickHouse/pull/33429) ([zzsmdfj](https://github.com/zzsmdfj)). * Fix various issues when projection is enabled by default. Each issue is described in separate commit. This is for [#33678](https://github.com/ClickHouse/ClickHouse/issues/33678) . This fixes [#34273](https://github.com/ClickHouse/ClickHouse/issues/34273). [#34305](https://github.com/ClickHouse/ClickHouse/pull/34305) ([Amos Bird](https://github.com/amosbird)). -#### Performance Improvement {#performance-improvement-10} +#### Performance improvement {#performance-improvement-10} * Support `optimize_read_in_order` if prefix of sorting key is already sorted. E.g. if we have sorting key `ORDER BY (a, b)` in table and query with `WHERE a = const ORDER BY b` clauses, now it will be applied reading in order of sorting key instead of full sort. [#32748](https://github.com/ClickHouse/ClickHouse/pull/32748) ([Anton Popov](https://github.com/CurtizJ)). * Improve performance of partitioned insert into table functions `URL`, `S3`, `File`, `HDFS`. Closes [#34348](https://github.com/ClickHouse/ClickHouse/issues/34348). [#34510](https://github.com/ClickHouse/ClickHouse/pull/34510) ([Maksim Kita](https://github.com/kitaisreal)). @@ -1588,7 +1588,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Fix parsing tables structure from ZooKeeper: now metadata from ZooKeeper compared with local metadata in canonical form. It helps when canonical function names can change between ClickHouse versions. [#33933](https://github.com/ClickHouse/ClickHouse/pull/33933) ([sunny](https://github.com/sunny19930321)). * Properly escape some characters for interaction with LDAP. [#33401](https://github.com/ClickHouse/ClickHouse/pull/33401) ([IlyaTsoi](https://github.com/IlyaTsoi)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-10} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-10} * Remove unbundled build support. [#33690](https://github.com/ClickHouse/ClickHouse/pull/33690) ([Azat Khuzhin](https://github.com/azat)). * Ensure that tests don't depend on the result of non-stable sorting of equal elements. Added equal items ranges randomization in debug after sort to prevent issues when we rely on equal items sort order. [#34393](https://github.com/ClickHouse/ClickHouse/pull/34393) ([Maksim Kita](https://github.com/kitaisreal)). @@ -1596,7 +1596,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Remove `clickhouse-test` debian package because it's obsolete. [#33948](https://github.com/ClickHouse/ClickHouse/pull/33948) ([Ilya Yatsishin](https://github.com/qoega)). * Multiple improvements for build system to remove the possibility of occasionally using packages from the OS and to enforce hermetic builds. [#33695](https://github.com/ClickHouse/ClickHouse/pull/33695) ([Amos Bird](https://github.com/amosbird)). -#### Bug Fix (user-visible misbehaviour in official stable or prestable release) {#bug-fix-user-visible-misbehaviour-in-official-stable-or-prestable-release-1} +#### Bug fix (user-visible misbehaviour in official stable or prestable release) {#bug-fix-user-visible-misbehaviour-in-official-stable-or-prestable-release-1} * Fixed the assertion in case of using `allow_experimental_parallel_reading_from_replicas` with `max_parallel_replicas` equals to 1. This fixes [#34525](https://github.com/ClickHouse/ClickHouse/issues/34525). [#34613](https://github.com/ClickHouse/ClickHouse/pull/34613) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Fix rare bug while reading of empty arrays, which could lead to `Data compressed with different methods` error. It can reproduce if you have mostly empty arrays, but not always. And reading is performed in backward direction with ORDER BY ... DESC. This error is extremely unlikely to happen. [#34327](https://github.com/ClickHouse/ClickHouse/pull/34327) ([Anton Popov](https://github.com/CurtizJ)). @@ -1655,14 +1655,14 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Fix issue [#18206](https://github.com/ClickHouse/ClickHouse/issues/18206). [#33977](https://github.com/ClickHouse/ClickHouse/pull/33977) ([Vitaly Baranov](https://github.com/vitlibar)). * This PR allows using multiple LDAP storages in the same list of user directories. It worked earlier but was broken because LDAP tests are disabled (they are part of the testflows tests). [#33574](https://github.com/ClickHouse/ClickHouse/pull/33574) ([Vitaly Baranov](https://github.com/vitlibar)). -### ClickHouse release v22.1, 2022-01-18 {#a-id221a-clickhouse-release-v221-2022-01-18} +### ClickHouse release v22.1, 2022-01-18 {#a-id221a-clickhouse-release-v221-2022-01-18} -#### Upgrade Notes {#upgrade-notes-4} +#### Upgrade notes {#upgrade-notes-4} * The functions `left` and `right` were previously implemented in parser and now full-featured. Distributed queries with `left` or `right` functions without aliases may throw exception if cluster contains different versions of clickhouse-server. If you are upgrading your cluster and encounter this error, you should finish upgrading your cluster to ensure all nodes have the same version. Also you can add aliases (`AS something`) to the columns in your queries to avoid this issue. [#33407](https://github.com/ClickHouse/ClickHouse/pull/33407) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Resource usage by scalar subqueries is fully accounted since this version. With this change, rows read in scalar subqueries are now reported in the query_log. If the scalar subquery is cached (repeated or called for several rows) the rows read are only counted once. This change allows KILLing queries and reporting progress while they are executing scalar subqueries. [#32271](https://github.com/ClickHouse/ClickHouse/pull/32271) ([Raúl Marín](https://github.com/Algunenano)). -#### New Feature {#new-feature-11} +#### New feature {#new-feature-11} * Implement data schema inference for input formats. Allow to skip structure (or write just `auto`) in table functions `file`, `url`, `s3`, `hdfs` and in parameters of `clickhouse-local` . Allow to skip structure in create query for table engines `File`, `HDFS`, `S3`, `URL`, `Merge`, `Buffer`, `Distributed` and `ReplicatedMergeTree` (if we add new replicas). [#32455](https://github.com/ClickHouse/ClickHouse/pull/32455) ([Kruglov Pavel](https://github.com/Avogar)). * Detect format by file extension in `file`/`hdfs`/`s3`/`url` table functions and `HDFS`/`S3`/`URL` table engines and also for `SELECT INTO OUTFILE` and `INSERT FROM INFILE` [#33565](https://github.com/ClickHouse/ClickHouse/pull/33565) ([Kruglov Pavel](https://github.com/Avogar)). Close [#30918](https://github.com/ClickHouse/ClickHouse/issues/30918). [#33443](https://github.com/ClickHouse/ClickHouse/pull/33443) ([OnePiece](https://github.com/zhongyuankai)). @@ -1681,7 +1681,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Add function `decodeURLFormComponent` slightly different to `decodeURLComponent`. Close [#10298](https://github.com/ClickHouse/ClickHouse/issues/10298). [#33451](https://github.com/ClickHouse/ClickHouse/pull/33451) ([SuperDJY](https://github.com/cmsxbc)). * Allow to split `GraphiteMergeTree` rollup rules for plain/tagged metrics (optional rule_type field). [#33494](https://github.com/ClickHouse/ClickHouse/pull/33494) ([Michail Safronov](https://github.com/msaf1980)). -#### Performance Improvement {#performance-improvement-11} +#### Performance improvement {#performance-improvement-11} * Support moving conditions to `PREWHERE` (setting `optimize_move_to_prewhere`) for tables of `Merge` engine if its all underlying tables supports `PREWHERE`. [#33300](https://github.com/ClickHouse/ClickHouse/pull/33300) ([Anton Popov](https://github.com/CurtizJ)). * More efficient handling of globs for URL storage. Now you can easily query million URLs in parallel with retries. Closes [#32866](https://github.com/ClickHouse/ClickHouse/issues/32866). [#32907](https://github.com/ClickHouse/ClickHouse/pull/32907) ([Kseniia Sumarokova](https://github.com/kssenii)). @@ -1696,7 +1696,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Optimize selecting of MergeTree parts that can be moved between volumes. [#33225](https://github.com/ClickHouse/ClickHouse/pull/33225) ([OnePiece](https://github.com/zhongyuankai)). * Fix `sparse_hashed` dict performance with sequential keys (wrong hash function). [#32536](https://github.com/ClickHouse/ClickHouse/pull/32536) ([Azat Khuzhin](https://github.com/azat)). -#### Experimental Feature {#experimental-feature-10} +#### Experimental feature {#experimental-feature-10} * Parallel reading from multiple replicas within a shard during distributed query without using sample key. To enable this, set `allow_experimental_parallel_reading_from_replicas = 1` and `max_parallel_replicas` to any number. This closes [#26748](https://github.com/ClickHouse/ClickHouse/issues/26748). [#29279](https://github.com/ClickHouse/ClickHouse/pull/29279) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Implemented sparse serialization. It can reduce usage of disk space and improve performance of some queries for columns, which contain a lot of default (zero) values. It can be enabled by setting `ratio_for_sparse_serialization`. Sparse serialization will be chosen dynamically for column, if it has ratio of number of default values to number of all values above that threshold. Serialization (default or sparse) will be fixed for every column in part, but may varies between parts. [#22535](https://github.com/ClickHouse/ClickHouse/pull/22535) ([Anton Popov](https://github.com/CurtizJ)). @@ -1754,7 +1754,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Updating `modification_time` for data part in `system.parts` after part movement [#32964](https://github.com/ClickHouse/ClickHouse/issues/32964). [#32965](https://github.com/ClickHouse/ClickHouse/pull/32965) ([save-my-heart](https://github.com/save-my-heart)). * Potential issue, cannot be exploited: integer overflow may happen in array resize. [#33024](https://github.com/ClickHouse/ClickHouse/pull/33024) ([varadarajkumar](https://github.com/varadarajkumar)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-11} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-11} * Add packages, functional tests and Docker builds for AArch64 (ARM) version of ClickHouse. [#32911](https://github.com/ClickHouse/ClickHouse/pull/32911) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). [#32415](https://github.com/ClickHouse/ClickHouse/pull/32415) * Prepare ClickHouse to be built with musl-libc. It is not enabled by default. [#33134](https://github.com/ClickHouse/ClickHouse/pull/33134) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -1768,7 +1768,7 @@ Refer to this issue on GitHub for more details: https://github.com/ClickHouse/Cl * Inject git information into clickhouse binary file. So we can get source code revision easily from clickhouse binary file. [#33124](https://github.com/ClickHouse/ClickHouse/pull/33124) ([taiyang-li](https://github.com/taiyang-li)). * Remove obsolete code from ConfigProcessor. Yandex specific code is not used anymore. The code contained one minor defect. This defect was reported by [Mallik Hassan](https://github.com/SadiHassan) in [#33032](https://github.com/ClickHouse/ClickHouse/issues/33032). This closes [#33032](https://github.com/ClickHouse/ClickHouse/issues/33032). [#33026](https://github.com/ClickHouse/ClickHouse/pull/33026) ([alexey-milovidov](https://github.com/alexey-milovidov)). -#### Bug Fix (user-visible misbehavior in official stable or prestable release) {#bug-fix-user-visible-misbehavior-in-official-stable-or-prestable-release-4} +#### Bug fix (user-visible misbehavior in official stable or prestable release) {#bug-fix-user-visible-misbehavior-in-official-stable-or-prestable-release-4} * Several fixes for format parsing. This is relevant if `clickhouse-server` is open for write access to adversary. Specifically crafted input data for `Native` format may lead to reading uninitialized memory or crash. This is relevant if `clickhouse-server` is open for write access to adversary. [#33050](https://github.com/ClickHouse/ClickHouse/pull/33050) ([Heena Bansal](https://github.com/HeenaBansal2009)). Fixed Apache Avro Union type index out of boundary issue in Apache Avro binary format. [#33022](https://github.com/ClickHouse/ClickHouse/pull/33022) ([Harry Lee](https://github.com/HarryLeeIBM)). Fix null pointer dereference in `LowCardinality` data when deserializing `LowCardinality` data in the Native format. [#33021](https://github.com/ClickHouse/ClickHouse/pull/33021) ([Harry Lee](https://github.com/HarryLeeIBM)). * ClickHouse Keeper handler will correctly remove operation when response sent. [#32988](https://github.com/ClickHouse/ClickHouse/pull/32988) ([JackyWoo](https://github.com/JackyWoo)). diff --git a/docs/whats-new/changelog/2023.md b/docs/whats-new/changelog/2023.md index 642ed40dfbb..26c1a93e01b 100644 --- a/docs/whats-new/changelog/2023.md +++ b/docs/whats-new/changelog/2023.md @@ -8,7 +8,7 @@ keywords: ['ClickHouse 2023', 'changelog 2023', 'release notes', 'version histor doc_type: 'changelog' --- -### Table of Contents {#table-of-contents} +### Table of contents {#table-of-contents} **[ClickHouse release v23.12, 2023-12-28](#2312)**
**[ClickHouse release v23.11, 2023-12-06](#2311)**
**[ClickHouse release v23.10, 2023-11-02](#2310)**
@@ -23,13 +23,13 @@ doc_type: 'changelog' **[ClickHouse release v23.1, 2023-01-25](#231)**
**[Changelog for 2022](/whats-new/changelog/2022/)**
-### ClickHouse release 23.12, 2023-12-28 {#2312} +### ClickHouse release 23.12, 2023-12-28 {#2312} -#### Backward Incompatible Change {#backward-incompatible-change} +#### Backward incompatible change {#backward-incompatible-change} * Fix check for non-deterministic functions in TTL expressions. Previously, you could create a TTL expression with non-deterministic functions in some cases, which could lead to undefined behavior later. This fixes [#37250](https://github.com/ClickHouse/ClickHouse/issues/37250). Disallow TTL expressions that don't depend on any columns of a table by default. It can be allowed back by `SET allow_suspicious_ttl_expressions = 1` or `SET compatibility = '23.11'`. Closes [#37286](https://github.com/ClickHouse/ClickHouse/issues/37286). [#51858](https://github.com/ClickHouse/ClickHouse/pull/51858) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * The MergeTree setting `clean_deleted_rows` is deprecated, it has no effect anymore. The `CLEANUP` keyword for the `OPTIMIZE` is not allowed by default (it can be unlocked with the `allow_experimental_replacing_merge_with_cleanup` setting). [#58267](https://github.com/ClickHouse/ClickHouse/pull/58267) ([Alexander Tokmakov](https://github.com/tavplubix)). This fixes [#57930](https://github.com/ClickHouse/ClickHouse/issues/57930). This closes [#54988](https://github.com/ClickHouse/ClickHouse/issues/54988). This closes [#54570](https://github.com/ClickHouse/ClickHouse/issues/54570). This closes [#50346](https://github.com/ClickHouse/ClickHouse/issues/50346). This closes [#47579](https://github.com/ClickHouse/ClickHouse/issues/47579). The feature has to be removed because it is not good. We have to remove it as quickly as possible, because there is no other option. [#57932](https://github.com/ClickHouse/ClickHouse/pull/57932) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### New Feature {#new-feature} +#### New feature {#new-feature} * Implement Refreshable Materialized Views, requested in [#33919](https://github.com/ClickHouse/ClickHouse/issues/33919). [#56946](https://github.com/ClickHouse/ClickHouse/pull/56946) ([Michael Kolupaev](https://github.com/al13n321), [Michael Guzov](https://github.com/koloshmet)). * Introduce `PASTE JOIN`, which allows users to join tables without `ON` clause simply by row numbers. Example: `SELECT * FROM (SELECT number AS a FROM numbers(2)) AS t1 PASTE JOIN (SELECT number AS a FROM numbers(2) ORDER BY a DESC) AS t2`. [#57995](https://github.com/ClickHouse/ClickHouse/pull/57995) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). * The `ORDER BY` clause now supports specifying `ALL`, meaning that ClickHouse sorts by all columns in the `SELECT` clause. Example: `SELECT col1, col2 FROM tab WHERE [...] ORDER BY ALL`. [#57875](https://github.com/ClickHouse/ClickHouse/pull/57875) ([zhongyuankai](https://github.com/zhongyuankai)). @@ -51,7 +51,7 @@ doc_type: 'changelog' * Added a new setting `readonly` which can be used to specify an S3 disk is read only. It can be useful to create a table on a disk of `s3_plain` type, while having read only access to the underlying S3 bucket. [#57977](https://github.com/ClickHouse/ClickHouse/pull/57977) ([Pengyuan Bian](https://github.com/bianpengyuan)). * The primary key analysis in MergeTree tables will now be applied to predicates that include the virtual column `_part_offset` (optionally with `_part`). This feature can serve as a special kind of a secondary index. [#58224](https://github.com/ClickHouse/ClickHouse/pull/58224) ([Amos Bird](https://github.com/amosbird)). -#### Performance Improvement {#performance-improvement} +#### Performance improvement {#performance-improvement} * Extract non-intersecting parts ranges from MergeTree table during FINAL processing. That way we can avoid additional FINAL logic for this non-intersecting parts ranges. In case when amount of duplicate values with same primary key is low, performance will be almost the same as without FINAL. Improve reading performance for MergeTree FINAL when `do_not_merge_across_partitions_select_final` setting is set. [#58120](https://github.com/ClickHouse/ClickHouse/pull/58120) ([Maksim Kita](https://github.com/kitaisreal)). * Made copy between s3 disks using a s3-server-side copy instead of copying through the buffer. Improves `BACKUP/RESTORE` operations and `clickhouse-disks copy` command. [#56744](https://github.com/ClickHouse/ClickHouse/pull/56744) ([MikhailBurdukov](https://github.com/MikhailBurdukov)). * Hash JOIN respects setting `max_joined_block_size_rows` and do not produce large blocks for `ALL JOIN`. [#56996](https://github.com/ClickHouse/ClickHouse/pull/56996) ([vdimir](https://github.com/vdimir)). @@ -114,7 +114,7 @@ doc_type: 'changelog' * Add base backup name to `system.backups` and `system.backup_log` tables [#58178](https://github.com/ClickHouse/ClickHouse/pull/58178) ([Pradeep Chhetri](https://github.com/chhetripradeep)). * Add support for specifying query parameters in the command line in clickhouse-local [#58210](https://github.com/ClickHouse/ClickHouse/pull/58210) ([Pradeep Chhetri](https://github.com/chhetripradeep)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement} * Randomize more settings [#39663](https://github.com/ClickHouse/ClickHouse/pull/39663) ([Anton Popov](https://github.com/CurtizJ)). * Randomize disabled optimizations in CI [#57315](https://github.com/ClickHouse/ClickHouse/pull/57315) ([Raúl Marín](https://github.com/Algunenano)). * Allow usage of Azure-related table engines/functions on macOS. [#51866](https://github.com/ClickHouse/ClickHouse/pull/51866) ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -123,7 +123,7 @@ doc_type: 'changelog' * Remove the usage of a harmful C/POSIX `select` function from external libraries. [#57467](https://github.com/ClickHouse/ClickHouse/pull/57467) ([Igor Nikonov](https://github.com/devcrafter)). * Settings only available in ClickHouse Cloud will be also present in the open-source ClickHouse build for convenience. [#57638](https://github.com/ClickHouse/ClickHouse/pull/57638) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). -#### Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release} +#### Bug fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release} * Fixed a possibility of sorting order breakage in TTL GROUP BY [#49103](https://github.com/ClickHouse/ClickHouse/pull/49103) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Fix: split `lttb` bucket strategy, first bucket and last bucket should only contain single point [#57003](https://github.com/ClickHouse/ClickHouse/pull/57003) ([FFish](https://github.com/wxybear)). * Fix possible deadlock in the `Template` format during sync after error [#57004](https://github.com/ClickHouse/ClickHouse/pull/57004) ([Kruglov Pavel](https://github.com/Avogar)). @@ -162,16 +162,16 @@ doc_type: 'changelog' * Fix a slow-down of CREATE VIEW with an enormous number of subqueries [#58220](https://github.com/ClickHouse/ClickHouse/pull/58220) ([Tao Wang](https://github.com/wangtZJU)). * Fix parallel parsing for JSONCompactEachRow [#58181](https://github.com/ClickHouse/ClickHouse/pull/58181) ([Alexey Milovidov](https://github.com/alexey-milovidov)). [#58250](https://github.com/ClickHouse/ClickHouse/pull/58250) ([Kruglov Pavel](https://github.com/Avogar)). -### ClickHouse release 23.11, 2023-12-06 {#2311} +### ClickHouse release 23.11, 2023-12-06 {#2311} -#### Backward Incompatible Change {#backward-incompatible-change-1} +#### Backward incompatible change {#backward-incompatible-change-1} * The default ClickHouse server configuration file has enabled `access_management` (user manipulation by SQL queries) and `named_collection_control` (manipulation of named collection by SQL queries) for the `default` user by default. This closes [#56482](https://github.com/ClickHouse/ClickHouse/issues/56482). [#56619](https://github.com/ClickHouse/ClickHouse/pull/56619) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Multiple improvements for `RESPECT NULLS`/`IGNORE NULLS` for window functions. If you use them as aggregate functions and store the states of aggregate functions with these modifiers, they might become incompatible. [#57189](https://github.com/ClickHouse/ClickHouse/pull/57189) ([Raúl Marín](https://github.com/Algunenano)). * Remove optimization `optimize_move_functions_out_of_any`. [#57190](https://github.com/ClickHouse/ClickHouse/pull/57190) ([Raúl Marín](https://github.com/Algunenano)). * Formatters `%l`/`%k`/`%c` in function `parseDateTime` are now able to parse hours/months without leading zeros, e.g. `select parseDateTime('2023-11-26 8:14', '%F %k:%i')` now works. Set `parsedatetime_parse_without_leading_zeros = 0` to restore the previous behavior which required two digits. Function `formatDateTime` is now also able to print hours/months without leading zeros. This is controlled by setting `formatdatetime_format_without_leading_zeros` but off by default to not break existing use cases. [#55872](https://github.com/ClickHouse/ClickHouse/pull/55872) ([Azat Khuzhin](https://github.com/azat)). * You can no longer use the aggregate function `avgWeighted` with arguments of type `Decimal`. Workaround: convert arguments to `Float64`. This closes [#43928](https://github.com/ClickHouse/ClickHouse/issues/43928). This closes [#31768](https://github.com/ClickHouse/ClickHouse/issues/31768). This closes [#56435](https://github.com/ClickHouse/ClickHouse/issues/56435). If you have used this function inside materialized views or projections with `Decimal` arguments, contact support@clickhouse.com. Fixed error in aggregate function `sumMap` and made it slower around 1.5..2 times. It does not matter because the function is garbage anyway. This closes [#54955](https://github.com/ClickHouse/ClickHouse/issues/54955). This closes [#53134](https://github.com/ClickHouse/ClickHouse/issues/53134). This closes [#55148](https://github.com/ClickHouse/ClickHouse/issues/55148). Fix a bug in function `groupArraySample` - it used the same random seed in case more than one aggregate state is generated in a query. [#56350](https://github.com/ClickHouse/ClickHouse/pull/56350) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### New Feature {#new-feature-1} +#### New feature {#new-feature-1} * Added server setting `async_load_databases` for asynchronous loading of databases and tables. Speeds up the server start time. Applies to databases with `Ordinary`, `Atomic` and `Replicated` engines. Their tables load metadata asynchronously. Query to a table increases the priority of the load job and waits for it to be done. Added a new table `system.asynchronous_loader` for introspection. [#49351](https://github.com/ClickHouse/ClickHouse/pull/49351) ([Sergei Trifonov](https://github.com/serxa)). * Add system table `blob_storage_log`. It allows auditing all the data written to S3 and other object storages. [#52918](https://github.com/ClickHouse/ClickHouse/pull/52918) ([vdimir](https://github.com/vdimir)). * Use statistics to order prewhere conditions better. [#53240](https://github.com/ClickHouse/ClickHouse/pull/53240) ([Han Fei](https://github.com/hanfei1991)). @@ -197,7 +197,7 @@ doc_type: 'changelog' * Setting `join_algorithm` respects specified order [#51745](https://github.com/ClickHouse/ClickHouse/pull/51745) ([vdimir](https://github.com/vdimir)). * Add support for the [well-known Protobuf types](https://protobuf.dev/reference/protobuf/google.protobuf/) in the Protobuf format. [#56741](https://github.com/ClickHouse/ClickHouse/pull/56741) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). -#### Performance Improvement {#performance-improvement-1} +#### Performance improvement {#performance-improvement-1} * Adaptive timeouts for interacting with S3. The first attempt is made with low send and receive timeouts. [#56314](https://github.com/ClickHouse/ClickHouse/pull/56314) ([Sema Checherinda](https://github.com/CheSema)). * Increase the default value of `max_concurrent_queries` from 100 to 1000. This makes sense when there is a large number of connecting clients, which are slowly sending or receiving data, so the server is not limited by CPU, or when the number of CPU cores is larger than 100. Also, enable the concurrency control by default, and set the desired number of query processing threads in total as twice the number of CPU cores. It improves performance in scenarios with a very large number of concurrent queries. [#46927](https://github.com/ClickHouse/ClickHouse/pull/46927) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Support parallel evaluation of window functions. Fixes [#34688](https://github.com/ClickHouse/ClickHouse/issues/34688). [#39631](https://github.com/ClickHouse/ClickHouse/pull/39631) ([Dmitry Novik](https://github.com/novikd)). @@ -282,7 +282,7 @@ doc_type: 'changelog' * Increase default `replicated_deduplication_window` of MergeTree settings from 100 to 1k. [#57335](https://github.com/ClickHouse/ClickHouse/pull/57335) ([sichenzhao](https://github.com/sichenzhao)). * Stop using `INCONSISTENT_METADATA_FOR_BACKUP` that much. If possible prefer to continue scanning instead of stopping and starting the scanning for backup from the beginning. [#57385](https://github.com/ClickHouse/ClickHouse/pull/57385) ([Vitaly Baranov](https://github.com/vitlibar)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-1} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-1} * Add SQLLogic test. [#56078](https://github.com/ClickHouse/ClickHouse/pull/56078) ([Han Fei](https://github.com/hanfei1991)). * Make `clickhouse-local` and `clickhouse-client` available under short names (`ch`, `chl`, `chc`) for usability. [#56634](https://github.com/ClickHouse/ClickHouse/pull/56634) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Optimized build size further by removing unused code from external libraries. [#56786](https://github.com/ClickHouse/ClickHouse/pull/56786) ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -299,7 +299,7 @@ doc_type: 'changelog' * Enable temporary_data_in_cache in S3 tests in CI. [#48425](https://github.com/ClickHouse/ClickHouse/pull/48425) ([vdimir](https://github.com/vdimir)). * Set the max memory usage for clickhouse-client (`1G`) in the CI. [#56873](https://github.com/ClickHouse/ClickHouse/pull/56873) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). -#### Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-1} +#### Bug fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-1} * Fix exerimental Analyzer - insertion from select with subquery referencing insertion table should process only insertion block. [#50857](https://github.com/ClickHouse/ClickHouse/pull/50857) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). * Fix a bug in `str_to_map` function. [#56423](https://github.com/ClickHouse/ClickHouse/pull/56423) ([Arthur Passos](https://github.com/arthurpassos)). * Keeper `reconfig`: add timeout before yielding/taking leadership [#53481](https://github.com/ClickHouse/ClickHouse/pull/53481) ([Mike Kot](https://github.com/myrrc)). @@ -371,9 +371,9 @@ doc_type: 'changelog' * MergeTree mutations reuse source part index granularity [#57352](https://github.com/ClickHouse/ClickHouse/pull/57352) ([Maksim Kita](https://github.com/kitaisreal)). * FS cache: add a limit for background download [#57424](https://github.com/ClickHouse/ClickHouse/pull/57424) ([Kseniia Sumarokova](https://github.com/kssenii)). -### ClickHouse release 23.10, 2023-11-02 {#2310} +### ClickHouse release 23.10, 2023-11-02 {#2310} -#### Backward Incompatible Change {#backward-incompatible-change-2} +#### Backward incompatible change {#backward-incompatible-change-2} * There is no longer an option to automatically remove broken data parts. This closes [#55174](https://github.com/ClickHouse/ClickHouse/issues/55174). [#55184](https://github.com/ClickHouse/ClickHouse/pull/55184) ([Alexey Milovidov](https://github.com/alexey-milovidov)). [#55557](https://github.com/ClickHouse/ClickHouse/pull/55557) ([Jihyuk Bok](https://github.com/tomahawk28)). * The obsolete in-memory data parts can no longer be read from the write-ahead log. If you have configured in-memory parts before, they have to be removed before the upgrade. [#55186](https://github.com/ClickHouse/ClickHouse/pull/55186) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Remove the integration with Meilisearch. Reason: it was compatible only with the old version 0.18. The recent version of Meilisearch changed the protocol and does not work anymore. Note: we would appreciate it if you help to return it back. [#55189](https://github.com/ClickHouse/ClickHouse/pull/55189) ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -383,7 +383,7 @@ doc_type: 'changelog' * Rewrited the experimental `S3Queue` table engine completely: changed the way we keep information in zookeeper which allows to make less zookeeper requests, added caching of zookeeper state in cases when we know the state will not change, improved the polling from s3 process to make it less aggressive, changed the way ttl and max set for trached files is maintained, now it is a background process. Added `system.s3queue` and `system.s3queue_log` tables. Closes [#54998](https://github.com/ClickHouse/ClickHouse/issues/54998). [#54422](https://github.com/ClickHouse/ClickHouse/pull/54422) ([Kseniia Sumarokova](https://github.com/kssenii)). * Arbitrary paths on HTTP endpoint are no longer interpreted as a request to the `/query` endpoint. [#55521](https://github.com/ClickHouse/ClickHouse/pull/55521) ([Konstantin Bogdanov](https://github.com/thevar1able)). -#### New Feature {#new-feature-2} +#### New feature {#new-feature-2} * Add function `arrayFold(accumulator, x1, ..., xn -> expression, initial, array1, ..., arrayn)` which applies a lambda function to multiple arrays of the same cardinality and collects the result in an accumulator. [#49794](https://github.com/ClickHouse/ClickHouse/pull/49794) ([Lirikl](https://github.com/Lirikl)). * Support for `Npy` format. `SELECT * FROM file('example_array.npy', Npy)`. [#55982](https://github.com/ClickHouse/ClickHouse/pull/55982) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). * If a table has a space-filling curve in its key, e.g., `ORDER BY mortonEncode(x, y)`, the conditions on its arguments, e.g., `x >= 10 AND x <= 20 AND y >= 20 AND y <= 30` can be used for indexing. A setting `analyze_index_with_space_filling_curves` is added to enable or disable this analysis. This closes [#41195](https://github.com/ClickHouse/ClickHouse/issue/41195). Continuation of [#4538](https://github.com/ClickHouse/ClickHouse/pull/4538). Continuation of [#6286](https://github.com/ClickHouse/ClickHouse/pull/6286). Continuation of [#28130](https://github.com/ClickHouse/ClickHouse/pull/28130). Continuation of [#41753](https://github.com/ClickHouse/ClickHouse/pull/#41753). [#55642](https://github.com/ClickHouse/ClickHouse/pull/55642) ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -408,7 +408,7 @@ doc_type: 'changelog' * Add support for `SHOW MERGES` query. [#55815](https://github.com/ClickHouse/ClickHouse/pull/55815) ([megao](https://github.com/jetgm)). * Introduce a setting `create_table_empty_primary_key_by_default` for default `ORDER BY ()`. [#55899](https://github.com/ClickHouse/ClickHouse/pull/55899) ([Srikanth Chekuri](https://github.com/srikanthccv)). -#### Performance Improvement {#performance-improvement-2} +#### Performance improvement {#performance-improvement-2} * Add option `query_plan_preserve_num_streams_after_window_functions` to preserve the number of streams after evaluating window functions to allow parallel stream processing. [#50771](https://github.com/ClickHouse/ClickHouse/pull/50771) ([frinkr](https://github.com/frinkr)). * Release more streams if data is small. [#53867](https://github.com/ClickHouse/ClickHouse/pull/53867) ([Jiebin Sun](https://github.com/jiebinn)). * RoaringBitmaps being optimized before serialization. [#55044](https://github.com/ClickHouse/ClickHouse/pull/55044) ([UnamedRus](https://github.com/UnamedRus)). @@ -478,7 +478,7 @@ doc_type: 'changelog' * Fixed a typo in SQL function `minSampleSizeContinous` (renamed `minSampleSizeContinuous`). Old name is preserved for backward compatibility. This closes: [#56139](https://github.com/ClickHouse/ClickHouse/issues/56139). [#56143](https://github.com/ClickHouse/ClickHouse/pull/56143) ([Dorota Szeremeta](https://github.com/orotaday)). * Print path for broken parts on disk before shutting down the server. Before this change if a part is corrupted on disk and server cannot start, it was almost impossible to understand which part is broken. This is fixed. [#56181](https://github.com/ClickHouse/ClickHouse/pull/56181) ([Duc Canh Le](https://github.com/canhld94)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-2} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-2} * If the database in Docker is already initialized, it doesn't need to be initialized again upon subsequent launches. This can potentially fix the issue of infinite container restarts when the database fails to load within 1000 attempts (relevant for very large databases and multi-node setups). [#50724](https://github.com/ClickHouse/ClickHouse/pull/50724) ([Alexander Nikolaev](https://github.com/AlexNik)). * Resource with source code including submodules is built in Darwin special build task. It may be used to build ClickHouse without checking out the submodules. [#51435](https://github.com/ClickHouse/ClickHouse/pull/51435) ([Ilya Yatsishin](https://github.com/qoega)). * An error was occuring when building ClickHouse with the AVX series of instructions enabled globally (which isn't recommended). The reason is that snappy does not enable `SNAPPY_HAVE_X86_CRC32`. [#55049](https://github.com/ClickHouse/ClickHouse/pull/55049) ([monchickey](https://github.com/monchickey)). @@ -487,7 +487,7 @@ doc_type: 'changelog' * Modified the error message difference between openssl and boringssl to fix the functional test. [#55975](https://github.com/ClickHouse/ClickHouse/pull/55975) ([MeenaRenganathan22](https://github.com/MeenaRenganathan22)). * Use upstream repo for apache datasketches. [#55787](https://github.com/ClickHouse/ClickHouse/pull/55787) ([Nikita Taranov](https://github.com/nickitat)). -#### Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-2} +#### Bug fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-2} * Skip hardlinking inverted index files in mutation [#47663](https://github.com/ClickHouse/ClickHouse/pull/47663) ([cangyin](https://github.com/cangyin)). * Fixed bug of `match` function (regex) with pattern containing alternation produces incorrect key condition. Closes #53222. [#54696](https://github.com/ClickHouse/ClickHouse/pull/54696) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). * Fix 'Cannot find column' in read-in-order optimization with ARRAY JOIN [#51746](https://github.com/ClickHouse/ClickHouse/pull/51746) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -549,14 +549,14 @@ doc_type: 'changelog' * Fix schema cache for fallback JSON->JSONEachRow with changed settings [#56172](https://github.com/ClickHouse/ClickHouse/pull/56172) ([Kruglov Pavel](https://github.com/Avogar)). * Add error handler to odbc-bridge [#56185](https://github.com/ClickHouse/ClickHouse/pull/56185) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). -### ClickHouse release 23.9, 2023-09-28 {#239} +### ClickHouse release 23.9, 2023-09-28 {#239} -#### Backward Incompatible Change {#backward-incompatible-change-3} +#### Backward incompatible change {#backward-incompatible-change-3} * Remove the `status_info` configuration option and dictionaries status from the default Prometheus handler. [#54090](https://github.com/ClickHouse/ClickHouse/pull/54090) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * The experimental parts metadata cache is removed from the codebase. [#54215](https://github.com/ClickHouse/ClickHouse/pull/54215) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Disable setting `input_format_json_try_infer_numbers_from_strings` by default, so we don't try to infer numbers from strings in JSON formats by default to avoid possible parsing errors when sample data contains strings that looks like a number. [#55099](https://github.com/ClickHouse/ClickHouse/pull/55099) ([Kruglov Pavel](https://github.com/Avogar)). -#### New Feature {#new-feature-3} +#### New feature {#new-feature-3} * Improve schema inference from JSON formats: 1) Now it's possible to infer named Tuples from JSON objects without experimental JSON type under a setting `input_format_json_try_infer_named_tuples_from_objects` in JSON formats. Previously without experimental type JSON we could only infer JSON objects as Strings or Maps, now we can infer named Tuple. Resulting Tuple type will conain all keys of objects that were read in data sample during schema inference. It can be useful for reading structured JSON data without sparse objects. The setting is enabled by default. 2) Allow parsing JSON array into a column with type String under setting `input_format_json_read_arrays_as_strings`. It can help reading arrays with values with different types. 3) Allow to use type String for JSON keys with unkown types (`null`/`[]`/`{}`) in sample data under setting `input_format_json_infer_incomplete_types_as_strings`. Now in JSON formats we can read any value into String column and we can avoid getting error `Cannot determine type for column 'column_name' by first 25000 rows of data, most likely this column contains only Nulls or empty Arrays/Maps` during schema inference by using type String for unknown types, so the data will be read successfully. [#54427](https://github.com/ClickHouse/ClickHouse/pull/54427) ([Kruglov Pavel](https://github.com/Avogar)). * Added IO scheduling support for remote disks. Storage configuration for disk types `s3`, `s3_plain`, `hdfs` and `azure_blob_storage` can now contain `read_resource` and `write_resource` elements holding resource names. Scheduling policies for these resources can be configured in a separate server configuration section `resources`. Queries can be marked using setting `workload` and classified using server configuration section `workload_classifiers` to achieve diverse resource scheduling goals. More details in [the docs](/operations/workload-scheduling). [#47009](https://github.com/ClickHouse/ClickHouse/pull/47009) ([Sergei Trifonov](https://github.com/serxa)). Added "bandwidth_limit" IO scheduling node type. It allows you to specify `max_speed` and `max_burst` constraints on traffic passing though this node. [#54618](https://github.com/ClickHouse/ClickHouse/pull/54618) ([Sergei Trifonov](https://github.com/serxa)). * Added new type of authentication based on SSH keys. It works only for the native TCP protocol. [#41109](https://github.com/ClickHouse/ClickHouse/pull/41109) ([George Gamezardashvili](https://github.com/InfJoker)). @@ -578,7 +578,7 @@ doc_type: 'changelog' * Allow S3-style URLs for table functions `s3`, `gcs`, `oss`. URL is automatically converted to HTTP. Example: `'s3://clickhouse-public-datasets/hits.csv'` is converted to `'https://clickhouse-public-datasets.s3.amazonaws.com/hits.csv'`. [#54931](https://github.com/ClickHouse/ClickHouse/pull/54931) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). * Add new setting `print_pretty_type_names` to print pretty deep nested types like Tuple/Maps/Arrays. [#55095](https://github.com/ClickHouse/ClickHouse/pull/55095) ([Kruglov Pavel](https://github.com/Avogar)). -#### Performance Improvement {#performance-improvement-3} +#### Performance improvement {#performance-improvement-3} * Speed up reading from S3 by enabling prefetches by default. [#53709](https://github.com/ClickHouse/ClickHouse/pull/53709) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Do not implicitly read PK and version columns in lonely parts if unnecessary for queries with FINAL. [#53919](https://github.com/ClickHouse/ClickHouse/pull/53919) ([Duc Canh Le](https://github.com/canhld94)). * Optimize group by constant keys. Will optimize queries with group by `_file/_path` after https://github.com/ClickHouse/ClickHouse/pull/53529. [#53549](https://github.com/ClickHouse/ClickHouse/pull/53549) ([Kruglov Pavel](https://github.com/Avogar)). @@ -599,7 +599,7 @@ doc_type: 'changelog' * Reuse HTTP connections in the `s3` table function. [#54812](https://github.com/ClickHouse/ClickHouse/pull/54812) ([Michael Kolupaev](https://github.com/al13n321)). * Replace the linear search in `MergeTreeRangeReader::Stream::ceilRowsToCompleteGranules` with a binary search. [#54869](https://github.com/ClickHouse/ClickHouse/pull/54869) ([usurai](https://github.com/usurai)). -#### Experimental Feature {#experimental-feature} +#### Experimental feature {#experimental-feature} * The creation of `Annoy` indexes can now be parallelized using setting `max_threads_for_annoy_index_creation`. [#54047](https://github.com/ClickHouse/ClickHouse/pull/54047) ([Robert Schulze](https://github.com/rschu1ze)). * Parallel replicas over distributed don't read from all replicas [#54199](https://github.com/ClickHouse/ClickHouse/pull/54199) ([Igor Nikonov](https://github.com/devcrafter)). @@ -649,7 +649,7 @@ doc_type: 'changelog' * The Keeper dynamically adjusts log levels. [#50372](https://github.com/ClickHouse/ClickHouse/pull/50372) ([helifu](https://github.com/helifu)). * Added function `timestamp` for compatibility with MySQL. Closes [#54275](https://github.com/ClickHouse/ClickHouse/issues/54275). [#54639](https://github.com/ClickHouse/ClickHouse/pull/54639) ([Nikolay Degterinsky](https://github.com/evillique)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-3} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-3} * Bumped the compiler of official and continuous integration builds of ClickHouse from Clang 16 to 17. [#53831](https://github.com/ClickHouse/ClickHouse/pull/53831) ([Robert Schulze](https://github.com/rschu1ze)). * Regenerated tld data for lookups (`tldLookup.generated.cpp`). [#54269](https://github.com/ClickHouse/ClickHouse/pull/54269) ([Bharat Nallan](https://github.com/bharatnc)). * Remove the redundant `clickhouse-keeper-client` symlink. [#54587](https://github.com/ClickHouse/ClickHouse/pull/54587) ([Tomas Barton](https://github.com/deric)). @@ -658,7 +658,7 @@ doc_type: 'changelog' * If the linker is different than LLD, stop with a fatal error. [#55036](https://github.com/ClickHouse/ClickHouse/pull/55036) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Replaced the library to handle (encode/decode) base64 values from Turbo-Base64 to aklomp-base64. Both are SIMD-accelerated on x86 and ARM but 1. the license of the latter (BSD-2) is more favorable for ClickHouse, Turbo64 switched in the meantime to GPL-3, 2. with more GitHub stars, aklomp-base64 seems more future-proof, 3. aklomp-base64 has a slightly nicer API (which is arguably subjective), and 4. aklomp-base64 does not require us to hack around bugs (like non-threadsafe initialization). Note: aklomp-base64 rejects unpadded base64 values whereas Turbo-Base64 decodes them on a best-effort basis. RFC-4648 leaves it open whether padding is mandatory or not, but depending on the context this may be a behavioral change to be aware of. [#54119](https://github.com/ClickHouse/ClickHouse/pull/54119) ([Mikhail Koviazin](https://github.com/mkmkme)). -#### Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-3} +#### Bug fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-3} * Fix REPLACE/MOVE PARTITION with zero-copy replication (note: "zero-copy replication" is an experimental feature) [#54193](https://github.com/ClickHouse/ClickHouse/pull/54193) ([Alexander Tokmakov](https://github.com/tavplubix)). * Fix zero copy locks with hardlinks (note: "zero-copy replication" is an experimental feature) [#54859](https://github.com/ClickHouse/ClickHouse/pull/54859) ([Alexander Tokmakov](https://github.com/tavplubix)). * Fix zero copy garbage (note: "zero-copy replication" is an experimental feature) [#54550](https://github.com/ClickHouse/ClickHouse/pull/54550) ([Alexander Tokmakov](https://github.com/tavplubix)). @@ -715,16 +715,16 @@ doc_type: 'changelog' * Fix: insert quorum w/o keeper retries [#55026](https://github.com/ClickHouse/ClickHouse/pull/55026) ([Igor Nikonov](https://github.com/devcrafter)). * Fix simple state with nullable [#55030](https://github.com/ClickHouse/ClickHouse/pull/55030) ([Pedro Riera](https://github.com/priera)). -### ClickHouse release 23.8 LTS, 2023-08-31 {#238} +### ClickHouse release 23.8 lts, 2023-08-31 {#238} -#### Backward Incompatible Change {#backward-incompatible-change-4} +#### Backward incompatible change {#backward-incompatible-change-4} * If a dynamic disk contains a name, it should be specified as `disk = disk(name = 'disk_name'`, ...) in disk function arguments. In previous version it could be specified as `disk = disk_(...)`, which is no longer supported. [#52820](https://github.com/ClickHouse/ClickHouse/pull/52820) ([Kseniia Sumarokova](https://github.com/kssenii)). * `clickhouse-benchmark` will establish connections in parallel when invoked with `--concurrency` more than one. Previously it was unusable if you ran it with 1000 concurrent connections from Europe to the US. Correct calculation of QPS for connections with high latency. Backward incompatible change: the option for JSON output of `clickhouse-benchmark` is removed. If you've used this option, you can also extract data from the `system.query_log` in JSON format as a workaround. [#53293](https://github.com/ClickHouse/ClickHouse/pull/53293) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * The `microseconds` column is removed from the `system.text_log`, and the `milliseconds` column is removed from the `system.metric_log`, because they are redundant in the presence of the `event_time_microseconds` column. [#53601](https://github.com/ClickHouse/ClickHouse/pull/53601) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Deprecate the metadata cache feature. It is experimental and we have never used it. The feature is dangerous: [#51182](https://github.com/ClickHouse/ClickHouse/issues/51182). Remove the `system.merge_tree_metadata_cache` system table. The metadata cache is still available in this version but will be removed soon. This closes [#39197](https://github.com/ClickHouse/ClickHouse/issues/39197). [#51303](https://github.com/ClickHouse/ClickHouse/pull/51303) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Disable support for 3DES in TLS connections. [#52893](https://github.com/ClickHouse/ClickHouse/pull/52893) ([Kenji Noguchi](https://github.com/knoguchi)). -#### New Feature {#new-feature-4} +#### New feature {#new-feature-4} * Direct import from zip/7z/tar archives. Example: `file('*.zip :: *.csv')`. [#50321](https://github.com/ClickHouse/ClickHouse/pull/50321) ([nikitakeba](https://github.com/nikitakeba)). * Add column `ptr` to `system.trace_log` for `trace_type = 'MemorySample'`. This column contains an address of allocation. Added function `flameGraph` which can build flamegraph containing allocated and not released memory. Reworking of [#38391](https://github.com/ClickHouse/ClickHouse/issues/38391). [#45322](https://github.com/ClickHouse/ClickHouse/pull/45322) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Added table function `azureBlobStorageCluster`. The supported set of features is very similar to table function `s3Cluster`. [#50795](https://github.com/ClickHouse/ClickHouse/pull/50795) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni)). @@ -750,12 +750,12 @@ doc_type: 'changelog' * Added support for adding and subtracting arrays: `[5,2] + [1,7]`. Division and multiplication were not implemented due to confusion between pointwise multiplication and the scalar product of arguments. Closes [#49939](https://github.com/ClickHouse/ClickHouse/issues/49939). [#52625](https://github.com/ClickHouse/ClickHouse/pull/52625) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). * Add support for string literals as table names. Closes [#52178](https://github.com/ClickHouse/ClickHouse/issues/52178). [#52635](https://github.com/ClickHouse/ClickHouse/pull/52635) ([hendrik-m](https://github.com/hendrik-m)). -#### Experimental Feature {#experimental-feature-1} +#### Experimental feature {#experimental-feature-1} * Add new table engine `S3Queue` for streaming data import from s3. Closes [#37012](https://github.com/ClickHouse/ClickHouse/issues/37012). [#49086](https://github.com/ClickHouse/ClickHouse/pull/49086) ([s-kat](https://github.com/s-kat)). It is not ready to use. Do not use it. * Enable parallel reading from replicas over distributed table. Related to [#49708](https://github.com/ClickHouse/ClickHouse/issues/49708). [#53005](https://github.com/ClickHouse/ClickHouse/pull/53005) ([Igor Nikonov](https://github.com/devcrafter)). * Add experimental support for HNSW as approximate neighbor search method. [#53447](https://github.com/ClickHouse/ClickHouse/pull/53447) ([Davit Vardanyan](https://github.com/davvard)). This is currently intended for those who continue working on the implementation. Do not use it. -#### Performance Improvement {#performance-improvement-4} +#### Performance improvement {#performance-improvement-4} * Parquet filter pushdown. I.e. when reading Parquet files, row groups (chunks of the file) are skipped based on the WHERE condition and the min/max values in each column. In particular, if the file is roughly sorted by some column, queries that filter by a short range of that column will be much faster. [#52951](https://github.com/ClickHouse/ClickHouse/pull/52951) ([Michael Kolupaev](https://github.com/al13n321)). * Optimize reading small row groups by batching them together in Parquet. Closes [#53069](https://github.com/ClickHouse/ClickHouse/issues/53069). [#53281](https://github.com/ClickHouse/ClickHouse/pull/53281) ([Kruglov Pavel](https://github.com/Avogar)). * Optimize count from files in most input formats. Closes [#44334](https://github.com/ClickHouse/ClickHouse/issues/44334). [#53637](https://github.com/ClickHouse/ClickHouse/pull/53637) ([Kruglov Pavel](https://github.com/Avogar)). @@ -848,7 +848,7 @@ doc_type: 'changelog' * Make reconnect limit in RAFT limits configurable for keeper. This configuration can help to make keeper to rebuild connection with peers quicker if the current connection is broken. [#53817](https://github.com/ClickHouse/ClickHouse/pull/53817) ([Pengyuan Bian](https://github.com/bianpengyuan)). * Ignore foreign keys in tables definition to improve compatibility with MySQL, so a user wouldn't need to rewrite his SQL of the foreign key part, ref [#53380](https://github.com/ClickHouse/ClickHouse/issues/53380). [#53864](https://github.com/ClickHouse/ClickHouse/pull/53864) ([jsc0218](https://github.com/jsc0218)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-4} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-4} * Don't expose symbols from ClickHouse binary to dynamic linker. It might fix [#43933](https://github.com/ClickHouse/ClickHouse/issues/43933). [#47475](https://github.com/ClickHouse/ClickHouse/pull/47475) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Add `clickhouse-keeper-client` symlink to the clickhouse-server package. [#51882](https://github.com/ClickHouse/ClickHouse/pull/51882) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). * Add https://github.com/elliotchance/sqltest to CI to report the SQL 2016 conformance. [#52293](https://github.com/ClickHouse/ClickHouse/pull/52293) ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -870,7 +870,7 @@ doc_type: 'changelog' * Upgrade snappy to 1.1.10. [#53672](https://github.com/ClickHouse/ClickHouse/pull/53672) ([李扬](https://github.com/taiyang-li)). * Slightly improve cmake build by sanitizing some dependencies and removing some duplicates. Each commit includes a short description of the changes made. [#53759](https://github.com/ClickHouse/ClickHouse/pull/53759) ([Amos Bird](https://github.com/amosbird)). -#### Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-4} +#### Bug fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-4} * Do not reset (experimental) Annoy index during build-up with more than one mark [#51325](https://github.com/ClickHouse/ClickHouse/pull/51325) ([Tian Xinhui](https://github.com/xinhuitian)). * Fix usage of temporary directories during RESTORE [#51493](https://github.com/ClickHouse/ClickHouse/pull/51493) ([Azat Khuzhin](https://github.com/azat)). * Fix binary arithmetic for Nullable(IPv4) [#51642](https://github.com/ClickHouse/ClickHouse/pull/51642) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). @@ -935,16 +935,16 @@ doc_type: 'changelog' * Fix possible segfault while using PostgreSQL engine [#53847](https://github.com/ClickHouse/ClickHouse/pull/53847) ([Kseniia Sumarokova](https://github.com/kssenii)). * Fix named_collection_admin alias [#54066](https://github.com/ClickHouse/ClickHouse/pull/54066) ([Kseniia Sumarokova](https://github.com/kssenii)). -### ClickHouse release 23.7, 2023-07-27 {#237} +### ClickHouse release 23.7, 2023-07-27 {#237} -#### Backward Incompatible Change {#backward-incompatible-change-5} +#### Backward incompatible change {#backward-incompatible-change-5} * Add `NAMED COLLECTION` access type (aliases `USE NAMED COLLECTION`, `NAMED COLLECTION USAGE`). This PR is backward incompatible because this access type is disabled by default (because a parent access type `NAMED COLLECTION ADMIN` is disabled by default as well). Proposed in [#50277](https://github.com/ClickHouse/ClickHouse/issues/50277). To grant use `GRANT NAMED COLLECTION ON collection_name TO user` or `GRANT NAMED COLLECTION ON * TO user`, to be able to give these grants `named_collection_admin` is required in config (previously it was named `named_collection_control`, so will remain as an alias). [#50625](https://github.com/ClickHouse/ClickHouse/pull/50625) ([Kseniia Sumarokova](https://github.com/kssenii)). * Fixing a typo in the `system.parts` column name `last_removal_attemp_time`. Now it is named `last_removal_attempt_time`. [#52104](https://github.com/ClickHouse/ClickHouse/pull/52104) ([filimonov](https://github.com/filimonov)). * Bump version of the distributed_ddl_entry_format_version to 5 by default (enables opentelemetry and initial_query_idd pass through). This will not allow to process existing entries for distributed DDL after *downgrade* (but note, that usually there should be no such unprocessed entries). [#52128](https://github.com/ClickHouse/ClickHouse/pull/52128) ([Azat Khuzhin](https://github.com/azat)). * Check projection metadata the same way we check ordinary metadata. This change may prevent the server from starting in case there was a table with an invalid projection. An example is a projection that created positional columns in PK (e.g. `projection p (select * order by 1, 4)` which is not allowed in table PK and can cause a crash during insert/merge). Drop such projections before the update. Fixes [#52353](https://github.com/ClickHouse/ClickHouse/issues/52353). [#52361](https://github.com/ClickHouse/ClickHouse/pull/52361) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * The experimental feature `hashid` is removed due to a bug. The quality of implementation was questionable at the start, and it didn't get through the experimental status. This closes [#52406](https://github.com/ClickHouse/ClickHouse/issues/52406). [#52449](https://github.com/ClickHouse/ClickHouse/pull/52449) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### New Feature {#new-feature-5} +#### New feature {#new-feature-5} * Added `Overlay` database engine to combine multiple databases into one. Added `Filesystem` database engine to represent a directory in the filesystem as a set of implicitly available tables with auto-detected formats and structures. A new `S3` database engine allows to read-only interact with s3 storage by representing a prefix as a set of tables. A new `HDFS` database engine allows to interact with HDFS storage in the same way. [#48821](https://github.com/ClickHouse/ClickHouse/pull/48821) ([alekseygolub](https://github.com/alekseygolub)). * Add support for external disks in Keeper for storing snapshots and logs. [#50098](https://github.com/ClickHouse/ClickHouse/pull/50098) ([Antonio Andelic](https://github.com/antonio2368)). * Add support for multi-directory selection (`{}`) globs. [#50559](https://github.com/ClickHouse/ClickHouse/pull/50559) ([Andrey Zvonov](https://github.com/zvonand)). @@ -976,7 +976,7 @@ doc_type: 'changelog' * Support `async_deduplication_token` for async insert. [#52136](https://github.com/ClickHouse/ClickHouse/pull/52136) ([Han Fei](https://github.com/hanfei1991)). * Add new setting `disable_url_encoding` that allows to disable decoding/encoding path in uri in URL engine. [#52337](https://github.com/ClickHouse/ClickHouse/pull/52337) ([Kruglov Pavel](https://github.com/Avogar)). -#### Performance Improvement {#performance-improvement-5} +#### Performance improvement {#performance-improvement-5} * Enable automatic selection of the sparse serialization format by default. It improves performance. The format is supported since version 22.1. After this change, downgrading to versions older than 22.1 might not be possible. A downgrade may require to set `ratio_of_defaults_for_sparse_serialization=0.9375` [55153](https://github.com/ClickHouse/ClickHouse/issues/55153). You can turn off the usage of the sparse serialization format by providing the `ratio_of_defaults_for_sparse_serialization = 1` setting for your MergeTree tables. [#49631](https://github.com/ClickHouse/ClickHouse/pull/49631) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Enable `move_all_conditions_to_prewhere` and `enable_multiple_prewhere_read_steps` settings by default. [#46365](https://github.com/ClickHouse/ClickHouse/pull/46365) ([Alexander Gololobov](https://github.com/davenger)). * Improves performance of some queries by tuning allocator. [#46416](https://github.com/ClickHouse/ClickHouse/pull/46416) ([Azat Khuzhin](https://github.com/azat)). @@ -1037,7 +1037,7 @@ doc_type: 'changelog' * Wait for zero copy replication lock even if some disks don't support it. [#52376](https://github.com/ClickHouse/ClickHouse/pull/52376) ([Raúl Marín](https://github.com/Algunenano)). * Now interserver port will be closed only after tables are shut down. [#52498](https://github.com/ClickHouse/ClickHouse/pull/52498) ([alesapin](https://github.com/alesapin)). -#### Experimental Feature {#experimental-feature-2} +#### Experimental feature {#experimental-feature-2} * Writing parquet files is 10x faster, it's multi-threaded now. Almost the same speed as reading. [#49367](https://github.com/ClickHouse/ClickHouse/pull/49367) ([Michael Kolupaev](https://github.com/al13n321)). This is controlled by the setting `output_format_parquet_use_custom_encoder` which is disabled by default, because the feature is non-ideal. * Added support for [PRQL](https://prql-lang.org/) as a query language. [#50686](https://github.com/ClickHouse/ClickHouse/pull/50686) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). * Allow to add disk name for custom disks. Previously custom disks would use an internal generated disk name. Now it will be possible with `disk = disk_(...)` (e.g. disk will have name `name`) . [#51552](https://github.com/ClickHouse/ClickHouse/pull/51552) ([Kseniia Sumarokova](https://github.com/kssenii)). This syntax can be changed in this release. @@ -1050,7 +1050,7 @@ doc_type: 'changelog' * The `session_timezone` setting (new in version 23.6) is demoted to experimental. [#52445](https://github.com/ClickHouse/ClickHouse/pull/52445) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Support ZooKeeper `reconfig` command for ClickHouse Keeper with incremental reconfiguration which can be enabled via `keeper_server.enable_reconfiguration` setting. Support adding servers, removing servers, and changing server priorities. [#49450](https://github.com/ClickHouse/ClickHouse/pull/49450) ([Mike Kot](https://github.com/myrrc)). It is suspected that this feature is incomplete. -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-5} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-5} * Add experimental ClickHouse builds for Linux RISC-V 64 to CI. [#31398](https://github.com/ClickHouse/ClickHouse/pull/31398) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Add integration test check with the enabled Analyzer. [#50926](https://github.com/ClickHouse/ClickHouse/pull/50926) [#52210](https://github.com/ClickHouse/ClickHouse/pull/52210) ([Dmitry Novik](https://github.com/novikd)). * Reproducible builds for Rust. [#52395](https://github.com/ClickHouse/ClickHouse/pull/52395) ([Azat Khuzhin](https://github.com/azat)). @@ -1060,7 +1060,7 @@ doc_type: 'changelog' * The `system.licenses` table now includes the hard-forked library Poco. This closes [#52066](https://github.com/ClickHouse/ClickHouse/issues/52066). [#52127](https://github.com/ClickHouse/ClickHouse/pull/52127) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Check that there are no cases of bad punctuation: whitespace before a comma like `Hello ,world` instead of `Hello, world`. [#52549](https://github.com/ClickHouse/ClickHouse/pull/52549) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-5} +#### Bug fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-5} * Fix MaterializedPostgreSQL syncTables [#49698](https://github.com/ClickHouse/ClickHouse/pull/49698) ([Kseniia Sumarokova](https://github.com/kssenii)). * Fix projection with optimize_aggregators_of_group_by_keys [#49709](https://github.com/ClickHouse/ClickHouse/pull/49709) ([Amos Bird](https://github.com/amosbird)). * Fix optimize_skip_unused_shards with JOINs [#51037](https://github.com/ClickHouse/ClickHouse/pull/51037) ([Azat Khuzhin](https://github.com/azat)). @@ -1109,15 +1109,15 @@ doc_type: 'changelog' * Fix lightweight delete after drop of projection [#52517](https://github.com/ClickHouse/ClickHouse/pull/52517) ([Anton Popov](https://github.com/CurtizJ)). * Fix possible error "Cannot drain connections: cancel first" [#52585](https://github.com/ClickHouse/ClickHouse/pull/52585) ([Kruglov Pavel](https://github.com/Avogar)). -### ClickHouse release 23.6, 2023-06-29 {#236} +### ClickHouse release 23.6, 2023-06-29 {#236} -#### Backward Incompatible Change {#backward-incompatible-change-6} +#### Backward incompatible change {#backward-incompatible-change-6} * Delete feature `do_not_evict_index_and_mark_files` in the fs cache. This feature was only making things worse. [#51253](https://github.com/ClickHouse/ClickHouse/pull/51253) ([Kseniia Sumarokova](https://github.com/kssenii)). * Remove ALTER support for experimental LIVE VIEW. [#51287](https://github.com/ClickHouse/ClickHouse/pull/51287) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Decrease the default values for `http_max_field_value_size` and `http_max_field_name_size` to 128 KiB. [#51163](https://github.com/ClickHouse/ClickHouse/pull/51163) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). * CGroups metrics related to CPU are replaced with one metric, `CGroupMaxCPU` for better usability. The `Normalized` CPU usage metrics will be normalized to CGroups limits instead of the total number of CPUs when they are set. This closes [#50836](https://github.com/ClickHouse/ClickHouse/issues/50836). [#50835](https://github.com/ClickHouse/ClickHouse/pull/50835) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### New Feature {#new-feature-6} +#### New feature {#new-feature-6} * The function `transform` as well as `CASE` with value matching started to support all data types. This closes [#29730](https://github.com/ClickHouse/ClickHouse/issues/29730). This closes [#32387](https://github.com/ClickHouse/ClickHouse/issues/32387). This closes [#50827](https://github.com/ClickHouse/ClickHouse/issues/50827). This closes [#31336](https://github.com/ClickHouse/ClickHouse/issues/31336). This closes [#40493](https://github.com/ClickHouse/ClickHouse/issues/40493). [#51351](https://github.com/ClickHouse/ClickHouse/pull/51351) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Added option `--rename_files_after_processing `. This closes [#34207](https://github.com/ClickHouse/ClickHouse/issues/34207). [#49626](https://github.com/ClickHouse/ClickHouse/pull/49626) ([alekseygolub](https://github.com/alekseygolub)). * Add support for `TRUNCATE` modifier in `INTO OUTFILE` clause. Suggest using `APPEND` or `TRUNCATE` for `INTO OUTFILE` when file exists. [#50950](https://github.com/ClickHouse/ClickHouse/pull/50950) ([alekar](https://github.com/alekar)). @@ -1128,7 +1128,7 @@ doc_type: 'changelog' * Add setting `session_timezone`; it is used as the default timezone for a session when not explicitly specified. [#44149](https://github.com/ClickHouse/ClickHouse/pull/44149) ([Andrey Zvonov](https://github.com/zvonand)). * Codec DEFLATE_QPL is now controlled via server setting "enable_deflate_qpl_codec" (default: false) instead of setting "allow_experimental_codecs". This marks DEFLATE_QPL non-experimental. [#50775](https://github.com/ClickHouse/ClickHouse/pull/50775) ([Robert Schulze](https://github.com/rschu1ze)). -#### Performance Improvement {#performance-improvement-6} +#### Performance improvement {#performance-improvement-6} * Improved scheduling of merge selecting and cleanup tasks in `ReplicatedMergeTree`. The tasks will not be executed too frequently when there's nothing to merge or cleanup. Added settings `max_merge_selecting_sleep_ms`, `merge_selecting_sleep_slowdown_factor`, `max_cleanup_delay_period` and `cleanup_thread_preferred_points_per_iteration`. It should close [#31919](https://github.com/ClickHouse/ClickHouse/issues/31919). [#50107](https://github.com/ClickHouse/ClickHouse/pull/50107) ([Alexander Tokmakov](https://github.com/tavplubix)). * Make filter push down through cross join. [#50605](https://github.com/ClickHouse/ClickHouse/pull/50605) ([Han Fei](https://github.com/hanfei1991)). * Improve performance with enabled QueryProfiler using thread-local timer_id instead of global object. [#48778](https://github.com/ClickHouse/ClickHouse/pull/48778) ([Jiebin Sun](https://github.com/jiebinn)). @@ -1141,7 +1141,7 @@ doc_type: 'changelog' * Fix data lakes slowness because of synchronous head requests. (Related to Iceberg/Deltalake/Hudi being slow with a lot of files). [#50976](https://github.com/ClickHouse/ClickHouse/pull/50976) ([Kseniia Sumarokova](https://github.com/kssenii)). * Do not read all the columns from right GLOBAL JOIN table. [#50721](https://github.com/ClickHouse/ClickHouse/pull/50721) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). -#### Experimental Feature {#experimental-feature-3} +#### Experimental feature {#experimental-feature-3} * Support parallel replicas with the analyzer. [#50441](https://github.com/ClickHouse/ClickHouse/pull/50441) ([Raúl Marín](https://github.com/Algunenano)). * Add random sleep before large merges/mutations execution to split load more evenly between replicas in case of zero-copy replication. [#51282](https://github.com/ClickHouse/ClickHouse/pull/51282) ([alesapin](https://github.com/alesapin)). * Do not replicate `ALTER PARTITION` queries and mutations through `Replicated` database if it has only one shard and the underlying table is `ReplicatedMergeTree`. [#51049](https://github.com/ClickHouse/ClickHouse/pull/51049) ([Alexander Tokmakov](https://github.com/tavplubix)). @@ -1166,7 +1166,7 @@ doc_type: 'changelog' * Better checking of data parts on disks with filesystem cache. [#51164](https://github.com/ClickHouse/ClickHouse/pull/51164) ([Anton Popov](https://github.com/CurtizJ)). * Fix sometimes not correct current_elements_num in fs cache. [#51242](https://github.com/ClickHouse/ClickHouse/pull/51242) ([Kseniia Sumarokova](https://github.com/kssenii)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-6} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-6} * Add embedded keeper-client to standalone keeper binary. [#50964](https://github.com/ClickHouse/ClickHouse/pull/50964) ([pufit](https://github.com/pufit)). * Actual LZ4 version is used now. [#50621](https://github.com/ClickHouse/ClickHouse/pull/50621) ([Nikita Taranov](https://github.com/nickitat)). * ClickHouse server will print the list of changed settings on fatal errors. This closes [#51137](https://github.com/ClickHouse/ClickHouse/issues/51137). [#51138](https://github.com/ClickHouse/ClickHouse/pull/51138) ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -1174,7 +1174,7 @@ doc_type: 'changelog' * [SQLancer](https://github.com/sqlancer/sqlancer) check is considered stable as bugs that were triggered by it are fixed. Now failures of SQLancer check will be reported as failed check status. [#51340](https://github.com/ClickHouse/ClickHouse/pull/51340) ([Ilya Yatsishin](https://github.com/qoega)). * Split huge `RUN` in Dockerfile into smaller conditional. Install the necessary tools on demand in the same `RUN` layer, and remove them after that. Upgrade the OS only once at the beginning. Use a modern way to check the signed repository. Downgrade the base repo to ubuntu:20.04 to address the issues on older docker versions. Upgrade golang version to address golang vulnerabilities. [#51504](https://github.com/ClickHouse/ClickHouse/pull/51504) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). -#### Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-6} +#### Bug fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-6} * Report loading status for executable dictionaries correctly [#48775](https://github.com/ClickHouse/ClickHouse/pull/48775) ([Anton Kozlov](https://github.com/tonickkozlov)). * Proper mutation of skip indices and projections [#50104](https://github.com/ClickHouse/ClickHouse/pull/50104) ([Amos Bird](https://github.com/amosbird)). @@ -1208,9 +1208,9 @@ doc_type: 'changelog' * Fix fuzzer failure in ActionsDAG [#51301](https://github.com/ClickHouse/ClickHouse/pull/51301) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Remove garbage from function `transform` [#51350](https://github.com/ClickHouse/ClickHouse/pull/51350) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -### ClickHouse release 23.5, 2023-06-08 {#235} +### ClickHouse release 23.5, 2023-06-08 {#235} -#### Upgrade Notes {#upgrade-notes} +#### Upgrade notes {#upgrade-notes} * Compress marks and primary key by default. It significantly reduces the cold query time. Upgrade notes: the support for compressed marks and primary key has been added in version 22.9. If you turned on compressed marks or primary key or installed version 23.5 or newer, which has compressed marks or primary key on by default, you will not be able to downgrade to version 22.8 or earlier. You can also explicitly disable compressed marks or primary keys by specifying the `compress_marks` and `compress_primary_key` settings in the `` section of the server configuration file. **Upgrade notes:** If you upgrade from versions prior to 22.9, you should either upgrade all replicas at once or disable the compression before upgrade, or upgrade through an intermediate version, where the compressed marks are supported but not enabled by default, such as 23.3. [#42587](https://github.com/ClickHouse/ClickHouse/pull/42587) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Make local object storage work consistently with s3 object storage, fix problem with append (closes [#48465](https://github.com/ClickHouse/ClickHouse/issues/48465)), make it configurable as independent storage. The change is backward incompatible because the cache on top of local object storage is not compatible to previous versions. [#48791](https://github.com/ClickHouse/ClickHouse/pull/48791) ([Kseniia Sumarokova](https://github.com/kssenii)). * The experimental feature "in-memory data parts" is removed. The data format is still supported, but the settings are no-op, and compact or wide parts will be used instead. This closes [#45409](https://github.com/ClickHouse/ClickHouse/issues/45409). [#49429](https://github.com/ClickHouse/ClickHouse/pull/49429) ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -1220,7 +1220,7 @@ doc_type: 'changelog' * Revert the "`groupArray` returns cannot be nullable" change (due to binary compatibility breakage for `groupArray`/`groupArrayLast`/`groupArraySample` over `Nullable` types, which likely will lead to `TOO_LARGE_ARRAY_SIZE` or `CANNOT_READ_ALL_DATA`). [#49971](https://github.com/ClickHouse/ClickHouse/pull/49971) ([Azat Khuzhin](https://github.com/azat)). * Setting `enable_memory_bound_merging_of_aggregation_results` is enabled by default. If you update from version prior to 22.12, we recommend to set this flag to `false` until update is finished. [#50319](https://github.com/ClickHouse/ClickHouse/pull/50319) ([Nikita Taranov](https://github.com/nickitat)). -#### New Feature {#new-feature-7} +#### New feature {#new-feature-7} * Added storage engine AzureBlobStorage and azureBlobStorage table function. The supported set of features is very similar to storage/table function S3 [#50604] (https://github.com/ClickHouse/ClickHouse/pull/50604) ([alesapin](https://github.com/alesapin)) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni). * Added native ClickHouse Keeper CLI Client, it is available as `clickhouse keeper-client` [#47414](https://github.com/ClickHouse/ClickHouse/pull/47414) ([pufit](https://github.com/pufit)). * Add `urlCluster` table function. Refactor all *Cluster table functions to reduce code duplication. Make schema inference work for all possible *Cluster function signatures and for named collections. Closes [#38499](https://github.com/ClickHouse/ClickHouse/issues/38499). [#45427](https://github.com/ClickHouse/ClickHouse/pull/45427) ([attack204](https://github.com/attack204)), Pavel Kruglov. @@ -1252,7 +1252,7 @@ doc_type: 'changelog' * Add server and format settings `display_secrets_in_show_and_select` for displaying secrets of tables, databases, table functions, and dictionaries. Add privilege `displaySecretsInShowAndSelect` controlling which users can view secrets. [#46528](https://github.com/ClickHouse/ClickHouse/pull/46528) ([Mike Kot](https://github.com/myrrc)). * Allow to set up a ROW POLICY for all tables that belong to a DATABASE. [#47640](https://github.com/ClickHouse/ClickHouse/pull/47640) ([Ilya Golshtein](https://github.com/ilejn)). -#### Performance Improvement {#performance-improvement-7} +#### Performance improvement {#performance-improvement-7} * Compress marks and primary key by default. It significantly reduces the cold query time. Upgrade notes: the support for compressed marks and primary key has been added in version 22.9. If you turned on compressed marks or primary key or installed version 23.5 or newer, which has compressed marks or primary key on by default, you will not be able to downgrade to version 22.8 or earlier. You can also explicitly disable compressed marks or primary keys by specifying the `compress_marks` and `compress_primary_key` settings in the `` section of the server configuration file. [#42587](https://github.com/ClickHouse/ClickHouse/pull/42587) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * New setting s3_max_inflight_parts_for_one_file sets the limit of concurrently loaded parts with multipart upload request in scope of one file. [#49961](https://github.com/ClickHouse/ClickHouse/pull/49961) ([Sema Checherinda](https://github.com/CheSema)). * When reading from multiple files reduce parallel parsing threads for each file. Resolves [#42192](https://github.com/ClickHouse/ClickHouse/issues/42192). [#46661](https://github.com/ClickHouse/ClickHouse/pull/46661) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni)). @@ -1278,7 +1278,7 @@ doc_type: 'changelog' * Don't send head request for all keys in Iceberg schema inference, only for keys that are used for reaing data. [#50203](https://github.com/ClickHouse/ClickHouse/pull/50203) ([Kruglov Pavel](https://github.com/Avogar)). * Setting `enable_memory_bound_merging_of_aggregation_results` is enabled by default. [#50319](https://github.com/ClickHouse/ClickHouse/pull/50319) ([Nikita Taranov](https://github.com/nickitat)). -#### Experimental Feature {#experimental-feature-4} +#### Experimental feature {#experimental-feature-4} * `DEFLATE_QPL` codec lower the minimum simd version to SSE 4.2. [doc change in qpl](https://github.com/intel/qpl/commit/3f8f5cea27739f5261e8fd577dc233ffe88bf679) - Intel® QPL relies on a run-time kernels dispatcher and cpuid check to choose the best available implementation(sse/avx2/avx512) - restructured cmakefile for qpl build in clickhouse to align with latest upstream qpl. [#49811](https://github.com/ClickHouse/ClickHouse/pull/49811) ([jasperzhu](https://github.com/jinjunzh)). * Add initial support to do JOINs with pure parallel replicas. [#49544](https://github.com/ClickHouse/ClickHouse/pull/49544) ([Raúl Marín](https://github.com/Algunenano)). * More parallelism on `Outdated` parts removal with "zero-copy replication". [#49630](https://github.com/ClickHouse/ClickHouse/pull/49630) ([Alexander Tokmakov](https://github.com/tavplubix)). @@ -1355,7 +1355,7 @@ doc_type: 'changelog' * In case JSON parse fails due to the large size of the object output the last position to allow debugging. [#50474](https://github.com/ClickHouse/ClickHouse/pull/50474) ([Valentin Alexeev](https://github.com/valentinalexeev)). * Support decimals with not fixed size. Closes [#49130](https://github.com/ClickHouse/ClickHouse/issues/49130). [#50586](https://github.com/ClickHouse/ClickHouse/pull/50586) ([Kruglov Pavel](https://github.com/Avogar)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-7} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-7} * New and improved `keeper-bench`. Everything can be customized from YAML/XML file: - request generator - each type of request generator can have a specific set of fields - multi requests can be generated just by doing the same under `multi` key - for each request or subrequest in multi a `weight` field can be defined to control distribution - define trees that need to be setup for a test run - hosts can be defined with all timeouts customizable and it's possible to control how many sessions to generate for each host - integers defined with `min_value` and `max_value` fields are random number generators. [#48547](https://github.com/ClickHouse/ClickHouse/pull/48547) ([Antonio Andelic](https://github.com/antonio2368)). * Io_uring is not supported on macos, don't choose it when running tests on local to avoid occasional failures. [#49250](https://github.com/ClickHouse/ClickHouse/pull/49250) ([Frank Chen](https://github.com/FrankChen021)). * Support named fault injection for testing. [#49361](https://github.com/ClickHouse/ClickHouse/pull/49361) ([Han Fei](https://github.com/hanfei1991)). @@ -1374,7 +1374,7 @@ doc_type: 'changelog' * Bump c-ares to v1.19.1 (bogus CVE-2023-32067, bogus CVE-2023-31130, bogus CVE-2023-31147). [#50403](https://github.com/ClickHouse/ClickHouse/pull/50403) ([Robert Schulze](https://github.com/rschu1ze)). * Fix bogus CVE-2022-2469 in libgsasl. [#50404](https://github.com/ClickHouse/ClickHouse/pull/50404) ([Robert Schulze](https://github.com/rschu1ze)). -#### Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-7} +#### Bug fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-7} * ActionsDAG: fix wrong optimization [#47584](https://github.com/ClickHouse/ClickHouse/pull/47584) ([Salvatore Mesoraca](https://github.com/aiven-sal)). * Correctly handle concurrent snapshots in Keeper [#48466](https://github.com/ClickHouse/ClickHouse/pull/48466) ([Antonio Andelic](https://github.com/antonio2368)). @@ -1460,14 +1460,14 @@ doc_type: 'changelog' * Fix converting Null to LowCardinality(Nullable) in values table function [#50637](https://github.com/ClickHouse/ClickHouse/pull/50637) ([Kruglov Pavel](https://github.com/Avogar)). * Revert invalid RegExpTreeDictionary optimization [#50642](https://github.com/ClickHouse/ClickHouse/pull/50642) ([Johann Gan](https://github.com/johanngan)). -### ClickHouse release 23.4, 2023-04-26 {#234} +### ClickHouse release 23.4, 2023-04-26 {#234} -#### Backward Incompatible Change {#backward-incompatible-change-7} +#### Backward incompatible change {#backward-incompatible-change-7} * Formatter '%M' in function formatDateTime() now prints the month name instead of the minutes. This makes the behavior consistent with MySQL. The previous behavior can be restored using setting "formatdatetime_parsedatetime_m_is_month_name = 0". [#47246](https://github.com/ClickHouse/ClickHouse/pull/47246) ([Robert Schulze](https://github.com/rschu1ze)). * This change makes sense only if you are using the virtual filesystem cache. If `path` in the virtual filesystem cache configuration is not empty and is not an absolute path, then it will be put in `/caches/`. [#48784](https://github.com/ClickHouse/ClickHouse/pull/48784) ([Kseniia Sumarokova](https://github.com/kssenii)). * Primary/secondary indices and sorting keys with identical expressions are now rejected. This behavior can be disabled using setting `allow_suspicious_indices`. [#48536](https://github.com/ClickHouse/ClickHouse/pull/48536) ([凌涛](https://github.com/lingtaolf)). -#### New Feature {#new-feature-8} +#### New feature {#new-feature-8} * Support new aggregate function `quantileGK`/`quantilesGK`, like [approx_percentile](https://spark.apache.org/docs/latest/api/sql/index.html#approx_percentile) in spark. Greenwald-Khanna algorithm refer to http://infolab.stanford.edu/~datar/courses/cs361a/papers/quantiles.pdf. [#46428](https://github.com/ClickHouse/ClickHouse/pull/46428) ([李扬](https://github.com/taiyang-li)). * Add a statement `SHOW COLUMNS` which shows distilled information from system.columns. [#48017](https://github.com/ClickHouse/ClickHouse/pull/48017) ([Robert Schulze](https://github.com/rschu1ze)). * Added `LIGHTWEIGHT` and `PULL` modifiers for `SYSTEM SYNC REPLICA` query. `LIGHTWEIGHT` version waits for fetches and drop-ranges only (merges and mutations are ignored). `PULL` version pulls new entries from ZooKeeper and does not wait for them. Fixes [#47794](https://github.com/ClickHouse/ClickHouse/issues/47794). [#48085](https://github.com/ClickHouse/ClickHouse/pull/48085) ([Alexander Tokmakov](https://github.com/tavplubix)). @@ -1483,7 +1483,7 @@ doc_type: 'changelog' * Functions replaceOne(), replaceAll(), replaceRegexpOne() and replaceRegexpAll() can now be called with non-const pattern and replacement arguments. [#46589](https://github.com/ClickHouse/ClickHouse/pull/46589) ([Robert Schulze](https://github.com/rschu1ze)). * Added functions to work with columns of type `Map`: `mapConcat`, `mapSort`, `mapExists`. [#48071](https://github.com/ClickHouse/ClickHouse/pull/48071) ([Anton Popov](https://github.com/CurtizJ)). -#### Performance Improvement {#performance-improvement-8} +#### Performance improvement {#performance-improvement-8} * Reading files in `Parquet` format is now much faster. IO and decoding are parallelized (controlled by `max_threads` setting), and only required data ranges are read. [#47964](https://github.com/ClickHouse/ClickHouse/pull/47964) ([Michael Kolupaev](https://github.com/al13n321)). * If we run a mutation with IN (subquery) like this: `ALTER TABLE t UPDATE col='new value' WHERE id IN (SELECT id FROM huge_table)` and the table `t` has multiple parts than for each part a set for subquery `SELECT id FROM huge_table` is built in memory. And if there are many parts then this might consume a lot of memory (and lead to an OOM) and CPU. The solution is to introduce a short-lived cache of sets that are currently being built by mutation tasks. If another task of the same mutation is executed concurrently it can look up the set in the cache, wait for it to be built and reuse it. [#46835](https://github.com/ClickHouse/ClickHouse/pull/46835) ([Alexander Gololobov](https://github.com/davenger)). * Only check dependencies if necessary when applying `ALTER TABLE` queries. [#48062](https://github.com/ClickHouse/ClickHouse/pull/48062) ([Raúl Marín](https://github.com/Algunenano)). @@ -1495,7 +1495,7 @@ doc_type: 'changelog' * Reduce memory usage for multiple `ALTER DELETE` mutations. [#48522](https://github.com/ClickHouse/ClickHouse/pull/48522) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Remove the excessive connection attempts if the `skip_unavailable_shards` setting is enabled. [#48771](https://github.com/ClickHouse/ClickHouse/pull/48771) ([Azat Khuzhin](https://github.com/azat)). -#### Experimental Feature {#experimental-feature-5} +#### Experimental feature {#experimental-feature-5} * Entries in the query cache are now squashed to max_block_size and compressed. [#45912](https://github.com/ClickHouse/ClickHouse/pull/45912) ([Robert Schulze](https://github.com/rschu1ze)). * It is now possible to define per-user quotas in the query cache. [#48284](https://github.com/ClickHouse/ClickHouse/pull/48284) ([Robert Schulze](https://github.com/rschu1ze)). * Some fixes for parallel replicas [#48433](https://github.com/ClickHouse/ClickHouse/pull/48433) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). @@ -1558,7 +1558,7 @@ doc_type: 'changelog' * In previous versions, the `LineAsString` format worked inconsistently when the parallel parsing was enabled or not, in presence of DOS or macOS Classic line breaks. This closes [#49039](https://github.com/ClickHouse/ClickHouse/issues/49039). [#49052](https://github.com/ClickHouse/ClickHouse/pull/49052) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * The exception message about the unparsed query parameter will also tell about the name of the parameter. Reimplement [#48878](https://github.com/ClickHouse/ClickHouse/issues/48878). Close [#48772](https://github.com/ClickHouse/ClickHouse/issues/48772). [#49061](https://github.com/ClickHouse/ClickHouse/pull/49061) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-8} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-8} * Update time zones. The following were updated: Africa/Cairo, Africa/Casablanca, Africa/El_Aaiun, America/Bogota, America/Cambridge_Bay, America/Ciudad_Juarez, America/Godthab, America/Inuvik, America/Iqaluit, America/Nuuk, America/Ojinaga, America/Pangnirtung, America/Rankin_Inlet, America/Resolute, America/Whitehorse, America/Yellowknife, Asia/Gaza, Asia/Hebron, Asia/Kuala_Lumpur, Asia/Singapore, Canada/Yukon, Egypt, Europe/Kirov, Europe/Volgograd, Singapore. [#48572](https://github.com/ClickHouse/ClickHouse/pull/48572) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Reduce the number of dependencies in the header files to speed up the build. [#47984](https://github.com/ClickHouse/ClickHouse/pull/47984) ([Dmitry Novik](https://github.com/novikd)). * Randomize compression of marks and indices in tests. [#48286](https://github.com/ClickHouse/ClickHouse/pull/48286) ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -1568,7 +1568,7 @@ doc_type: 'changelog' * Remove remainders of GCC support. [#48671](https://github.com/ClickHouse/ClickHouse/pull/48671) ([Robert Schulze](https://github.com/rschu1ze)). * Add CI run with new analyzer infrastructure enabled. [#48719](https://github.com/ClickHouse/ClickHouse/pull/48719) ([Dmitry Novik](https://github.com/novikd)). -#### Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-8} +#### Bug fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-8} * Fix system.query_views_log for MVs that are pushed from background threads [#46668](https://github.com/ClickHouse/ClickHouse/pull/46668) ([Azat Khuzhin](https://github.com/azat)). * Fix several `RENAME COLUMN` bugs [#46946](https://github.com/ClickHouse/ClickHouse/pull/46946) ([alesapin](https://github.com/alesapin)). @@ -1607,9 +1607,9 @@ doc_type: 'changelog' * Fix `UNKNOWN_IDENTIFIER` error while selecting from table with row policy and column with dots [#48976](https://github.com/ClickHouse/ClickHouse/pull/48976) ([Kruglov Pavel](https://github.com/Avogar)). * Fix aggregation by empty nullable strings [#48999](https://github.com/ClickHouse/ClickHouse/pull/48999) ([LiuNeng](https://github.com/liuneng1994)). -### ClickHouse release 23.3 LTS, 2023-03-30 {#233} +### ClickHouse release 23.3 lts, 2023-03-30 {#233} -#### Upgrade Notes {#upgrade-notes-1} +#### Upgrade notes {#upgrade-notes-1} * Lightweight DELETEs are production ready and enabled by default. The `DELETE` query for MergeTree tables is now available by default. * The behavior of `*domain*RFC` and `netloc` functions is slightly changed: relaxed the set of symbols that are allowed in the URL authority for better conformance. [#46841](https://github.com/ClickHouse/ClickHouse/pull/46841) ([Azat Khuzhin](https://github.com/azat)). * Prohibited creating tables based on KafkaEngine with DEFAULT/EPHEMERAL/ALIAS/MATERIALIZED statements for columns. [#47138](https://github.com/ClickHouse/ClickHouse/pull/47138) ([Aleksandr Musorin](https://github.com/AVMusorin)). @@ -1618,7 +1618,7 @@ doc_type: 'changelog' * Make backup_threads/restore_threads server settings (instead of user settings). [#47881](https://github.com/ClickHouse/ClickHouse/pull/47881) ([Azat Khuzhin](https://github.com/azat)). * Do not allow const and non-deterministic secondary indices [#46839](https://github.com/ClickHouse/ClickHouse/pull/46839) ([Anton Popov](https://github.com/CurtizJ)). -#### New Feature {#new-feature-9} +#### New feature {#new-feature-9} * Add a new mode for splitting the work on replicas using settings `parallel_replicas_custom_key` and `parallel_replicas_custom_key_filter_type`. If the cluster consists of a single shard with multiple replicas, up to `max_parallel_replicas` will be randomly picked and turned into shards. For each shard, a corresponding filter is added to the query on the initiator before being sent to the shard. If the cluster consists of multiple shards, it will behave the same as `sample_key` but with the possibility to define an arbitrary key. [#45108](https://github.com/ClickHouse/ClickHouse/pull/45108) ([Antonio Andelic](https://github.com/antonio2368)). * An option to display partial result on cancel: Added query setting `partial_result_on_first_cancel` allowing the canceled query (e.g. due to Ctrl-C) to return a partial result. [#45689](https://github.com/ClickHouse/ClickHouse/pull/45689) ([Alexey Perevyshin](https://github.com/alexX512)). * Added support of arbitrary tables engines for temporary tables (except for Replicated and KeeperMap engines). Close [#31497](https://github.com/ClickHouse/ClickHouse/issues/31497). [#46071](https://github.com/ClickHouse/ClickHouse/pull/46071) ([Roman Vasin](https://github.com/rvasin)). @@ -1642,7 +1642,7 @@ doc_type: 'changelog' * Add *OrNull() and *OrZero() variants for `parseDateTime`, add alias `str_to_date` for MySQL parity. [#48000](https://github.com/ClickHouse/ClickHouse/pull/48000) ([Robert Schulze](https://github.com/rschu1ze)). * Added operator `REGEXP` (similar to operators "LIKE", "IN", "MOD" etc.) for better compatibility with MySQL [#47869](https://github.com/ClickHouse/ClickHouse/pull/47869) ([Robert Schulze](https://github.com/rschu1ze)). -#### Performance Improvement {#performance-improvement-9} +#### Performance improvement {#performance-improvement-9} * Marks in memory are now compressed, using 3-6x less memory. [#47290](https://github.com/ClickHouse/ClickHouse/pull/47290) ([Michael Kolupaev](https://github.com/al13n321)). * Backups for large numbers of files were unbelievably slow in previous versions. Not anymore. Now they are unbelievably fast. [#47251](https://github.com/ClickHouse/ClickHouse/pull/47251) ([Alexey Milovidov](https://github.com/alexey-milovidov)). Introduced a separate thread pool for backup's IO operations. This will allow scaling it independently of other pools and increase performance. [#47174](https://github.com/ClickHouse/ClickHouse/pull/47174) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). Use MultiRead request and retries for collecting metadata at the final stage of backup processing. [#47243](https://github.com/ClickHouse/ClickHouse/pull/47243) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). If a backup and restoring data are both in S3 then server-side copy should be used from now on. [#47546](https://github.com/ClickHouse/ClickHouse/pull/47546) ([Vitaly Baranov](https://github.com/vitlibar)). * Fixed excessive reading in queries with `FINAL`. [#47801](https://github.com/ClickHouse/ClickHouse/pull/47801) ([Nikita Taranov](https://github.com/nickitat)). @@ -1658,7 +1658,7 @@ doc_type: 'changelog' * Even more batching in Keeper. Improve performance by avoiding breaking batches on read requests. [#47978](https://github.com/ClickHouse/ClickHouse/pull/47978) ([Antonio Andelic](https://github.com/antonio2368)). * Allow PREWHERE for Merge with different DEFAULT expressions for columns. [#46831](https://github.com/ClickHouse/ClickHouse/pull/46831) ([Azat Khuzhin](https://github.com/azat)). -#### Experimental Feature {#experimental-feature-6} +#### Experimental feature {#experimental-feature-6} * Parallel replicas: Improved the overall performance by better utilizing the local replica, and forbid the reading with parallel replicas from non-replicated MergeTree by default. [#47858](https://github.com/ClickHouse/ClickHouse/pull/47858) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Support filter push down to left table for JOIN with `Join`, `Dictionary` and `EmbeddedRocksDB` tables if the experimental Analyzer is enabled. [#47280](https://github.com/ClickHouse/ClickHouse/pull/47280) ([Maksim Kita](https://github.com/kitaisreal)). * Now ReplicatedMergeTree with zero copy replication has less load to Keeper. [#47676](https://github.com/ClickHouse/ClickHouse/pull/47676) ([alesapin](https://github.com/alesapin)). @@ -1713,7 +1713,7 @@ doc_type: 'changelog' * Support big integers `(U)Int128/(U)Int256`, `Map` with any key type and `DateTime64` with any precision (not only 3 and 6). [#48119](https://github.com/ClickHouse/ClickHouse/pull/48119) ([Kruglov Pavel](https://github.com/Avogar)). * Allow skipping errors related to unknown enum values in row input formats. [#48133](https://github.com/ClickHouse/ClickHouse/pull/48133) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-9} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-9} * ClickHouse now builds with `C++23`. [#47424](https://github.com/ClickHouse/ClickHouse/pull/47424) ([Robert Schulze](https://github.com/rschu1ze)). * Fuzz `EXPLAIN` queries in the AST Fuzzer. [#47803](https://github.com/ClickHouse/ClickHouse/pull/47803) [#47852](https://github.com/ClickHouse/ClickHouse/pull/47852) ([flynn](https://github.com/ucasfl)). * Split stress test and the automated backward compatibility check (now Upgrade check). [#44879](https://github.com/ClickHouse/ClickHouse/pull/44879) ([Kruglov Pavel](https://github.com/Avogar)). @@ -1728,7 +1728,7 @@ doc_type: 'changelog' * Add a unit test to assert Apache Arrow's fatal logging does not abort. It covers the changes in [ClickHouse/arrow#16](https://github.com/ClickHouse/arrow/pull/16). [#47958](https://github.com/ClickHouse/ClickHouse/pull/47958) ([Arthur Passos](https://github.com/arthurpassos)). * Restore the ability of native macOS debug server build to start. [#48050](https://github.com/ClickHouse/ClickHouse/pull/48050) ([Robert Schulze](https://github.com/rschu1ze)). Note: this change is only relevant for development, as the ClickHouse official builds are done with cross-compilation. -#### Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-9} +#### Bug fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-9} * Fix formats parser resetting, test processing bad messages in `Kafka` [#45693](https://github.com/ClickHouse/ClickHouse/pull/45693) ([Kruglov Pavel](https://github.com/Avogar)). * Fix data size calculation in Keeper [#46086](https://github.com/ClickHouse/ClickHouse/pull/46086) ([Antonio Andelic](https://github.com/antonio2368)). * Fixed a bug in automatic retries of `DROP TABLE` query with `ReplicatedMergeTree` tables and `Atomic` databases. In rare cases it could lead to `Can't get data for node /zk_path/log_pointer` and `The specified key does not exist` errors if the ZooKeeper session expired during DROP and a new replicated table with the same path in ZooKeeper was created in parallel. [#46384](https://github.com/ClickHouse/ClickHouse/pull/46384) ([Alexander Tokmakov](https://github.com/tavplubix)). @@ -1791,9 +1791,9 @@ doc_type: 'changelog' * Correctly set TCP/HTTP socket timeouts in Keeper [#48108](https://github.com/ClickHouse/ClickHouse/pull/48108) ([Antonio Andelic](https://github.com/antonio2368)). * Fix possible member call on null pointer in `Avro` format [#48184](https://github.com/ClickHouse/ClickHouse/pull/48184) ([Kruglov Pavel](https://github.com/Avogar)). -### ClickHouse release 23.2, 2023-02-23 {#232} +### ClickHouse release 23.2, 2023-02-23 {#232} -#### Backward Incompatible Change {#backward-incompatible-change-8} +#### Backward incompatible change {#backward-incompatible-change-8} * Extend function "toDayOfWeek()" (alias: "DAYOFWEEK") with a mode argument that encodes whether the week starts on Monday or Sunday and whether counting starts at 0 or 1. For consistency with other date time functions, the mode argument was inserted between the time and the time zone arguments. This breaks existing usage of the (previously undocumented) 2-argument syntax "toDayOfWeek(time, time_zone)". A fix is to rewrite the function into "toDayOfWeek(time, 0, time_zone)". [#45233](https://github.com/ClickHouse/ClickHouse/pull/45233) ([Robert Schulze](https://github.com/rschu1ze)). * Rename setting `max_query_cache_size` to `filesystem_cache_max_download_size`. [#45614](https://github.com/ClickHouse/ClickHouse/pull/45614) ([Kseniia Sumarokova](https://github.com/kssenii)). * The `default` user will not have permissions for access type `SHOW NAMED COLLECTION` by default (e.g. `default` user will no longer be able to grant ALL to other users as it was before, therefore this PR is backward incompatible). [#46010](https://github.com/ClickHouse/ClickHouse/pull/46010) ([Kseniia Sumarokova](https://github.com/kssenii)). @@ -1802,7 +1802,7 @@ doc_type: 'changelog' * Slightly improve performance of `countDigits` on realistic datasets. This closed [#44518](https://github.com/ClickHouse/ClickHouse/issues/44518). In previous versions, `countDigits(0)` returned `0`; now it returns `1`, which is more correct, and follows the existing documentation. [#46187](https://github.com/ClickHouse/ClickHouse/pull/46187) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Disallow creation of new columns compressed by a combination of codecs "Delta" or "DoubleDelta" followed by codecs "Gorilla" or "FPC". This can be bypassed using setting "allow_suspicious_codecs = true". [#45652](https://github.com/ClickHouse/ClickHouse/pull/45652) ([Robert Schulze](https://github.com/rschu1ze)). -#### New Feature {#new-feature-10} +#### New feature {#new-feature-10} * Add `StorageIceberg` and table function `iceberg` to access iceberg table store on S3. [#45384](https://github.com/ClickHouse/ClickHouse/pull/45384) ([flynn](https://github.com/ucasfl)). * Allow configuring storage as `SETTINGS disk = ''` (instead of `storage_policy`) and with explicit disk creation `SETTINGS disk = disk(type=s3, ...)`. [#41976](https://github.com/ClickHouse/ClickHouse/pull/41976) ([Kseniia Sumarokova](https://github.com/kssenii)). * Expose `ProfileEvents` counters in `system.part_log`. [#38614](https://github.com/ClickHouse/ClickHouse/pull/38614) ([Bharat Nallan](https://github.com/bharatnc)). @@ -1822,7 +1822,7 @@ doc_type: 'changelog' * Add new function `regexpExtract`, like spark function `REGEXP_EXTRACT` for compatibility. It is similar to the existing function `extract`. [#46469](https://github.com/ClickHouse/ClickHouse/pull/46469) ([李扬](https://github.com/taiyang-li)). * Add new function `JSONArrayLength`, which returns the number of elements in the outermost JSON array. The function returns NULL if the input JSON string is invalid. [#46631](https://github.com/ClickHouse/ClickHouse/pull/46631) ([李扬](https://github.com/taiyang-li)). -#### Performance Improvement {#performance-improvement-10} +#### Performance improvement {#performance-improvement-10} * The introduced logic works if PREWHERE condition is a conjunction of multiple conditions (cond1 AND cond2 AND ... ). It groups those conditions that require reading the same columns into steps. After each step the corresponding part of the full condition is computed and the result rows might be filtered. This allows to read fewer rows in the next steps thus saving IO bandwidth and doing less computation. This logic is disabled by default for now. It will be enabled by default in one of the future releases once it is known to not have any regressions, so it is highly encouraged to be used for testing. It can be controlled by 2 settings: "enable_multiple_prewhere_read_steps" and "move_all_conditions_to_prewhere". [#46140](https://github.com/ClickHouse/ClickHouse/pull/46140) ([Alexander Gololobov](https://github.com/davenger)). * An option added to aggregate partitions independently if table partition key and group by key are compatible. Controlled by the setting `allow_aggregate_partitions_independently`. Disabled by default because of limited applicability (please refer to the docs). [#45364](https://github.com/ClickHouse/ClickHouse/pull/45364) ([Nikita Taranov](https://github.com/nickitat)). * Allow using Vertical merge algorithm with parts in Compact format. This will allow ClickHouse server to use much less memory for background operations. This closes [#46084](https://github.com/ClickHouse/ClickHouse/issues/46084). [#45681](https://github.com/ClickHouse/ClickHouse/pull/45681) [#46282](https://github.com/ClickHouse/ClickHouse/pull/46282) ([Anton Popov](https://github.com/CurtizJ)). @@ -1912,7 +1912,7 @@ doc_type: 'changelog' * Support for IN clause with parameter in parameterized views. [#46583](https://github.com/ClickHouse/ClickHouse/pull/46583) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni)). * Do not load named collections on server startup (load them on first access instead). [#46607](https://github.com/ClickHouse/ClickHouse/pull/46607) ([Kseniia Sumarokova](https://github.com/kssenii)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-10} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-10} * Introduce GWP-ASan implemented by the LLVM runtime. This closes [#27039](https://github.com/ClickHouse/ClickHouse/issues/27039). [#45226](https://github.com/ClickHouse/ClickHouse/pull/45226) ([Han Fei](https://github.com/hanfei1991)). * We want to make our tests less stable and more flaky: add randomization for merge tree settings in tests. [#38983](https://github.com/ClickHouse/ClickHouse/pull/38983) ([Anton Popov](https://github.com/CurtizJ)). * Enable the HDFS support in PowerPC and which helps to fixes the following functional tests 02113_hdfs_assert.sh, 02244_hdfs_cluster.sql and 02368_cancel_write_into_hdfs.sh. [#44949](https://github.com/ClickHouse/ClickHouse/pull/44949) ([MeenaRenganathan22](https://github.com/MeenaRenganathan22)). @@ -1924,7 +1924,7 @@ doc_type: 'changelog' * Raised the minimum Clang version needed to build ClickHouse from 12 to 15. [#46710](https://github.com/ClickHouse/ClickHouse/pull/46710) ([Robert Schulze](https://github.com/rschu1ze)). * Upgrade Intel QPL from v0.3.0 to v1.0.0 2. Build libaccel-config and link it statically to QPL library instead of dynamically. [#45809](https://github.com/ClickHouse/ClickHouse/pull/45809) ([jasperzhu](https://github.com/jinjunzh)). -#### Bug Fix (user-visible misbehavior in official stable release) {#bug-fix-user-visible-misbehavior-in-official-stable-release} +#### Bug fix (user-visible misbehavior in official stable release) {#bug-fix-user-visible-misbehavior-in-official-stable-release} * Flush data exactly by `rabbitmq_flush_interval_ms` or by `rabbitmq_max_block_size` in `StorageRabbitMQ`. Closes [#42389](https://github.com/ClickHouse/ClickHouse/issues/42389). Closes [#45160](https://github.com/ClickHouse/ClickHouse/issues/45160). [#44404](https://github.com/ClickHouse/ClickHouse/pull/44404) ([Kseniia Sumarokova](https://github.com/kssenii)). * Use PODArray to render in sparkBar function, so we can control the memory usage. Close [#44467](https://github.com/ClickHouse/ClickHouse/issues/44467). [#44489](https://github.com/ClickHouse/ClickHouse/pull/44489) ([Duc Canh Le](https://github.com/canhld94)). @@ -1970,11 +1970,11 @@ doc_type: 'changelog' * Allocated during asynchronous inserts memory buffers were deallocated in the global context and MemoryTracker counters for corresponding user and query were not updated correctly. That led to false positive OOM exceptions. [#46622](https://github.com/ClickHouse/ClickHouse/pull/46622) ([Dmitry Novik](https://github.com/novikd)). * Updated to not clear on_expression from table_join as its used by future analyze runs resolves [#45185](https://github.com/ClickHouse/ClickHouse/issues/45185). [#46487](https://github.com/ClickHouse/ClickHouse/pull/46487) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni)). -### ClickHouse release 23.1, 2023-01-26 {#231} +### ClickHouse release 23.1, 2023-01-26 {#231} ### ClickHouse release 23.1 {#clickhouse-release-231} -#### Upgrade Notes {#upgrade-notes-2} +#### Upgrade notes {#upgrade-notes-2} * The `SYSTEM RESTART DISK` query becomes a no-op. [#44647](https://github.com/ClickHouse/ClickHouse/pull/44647) ([alesapin](https://github.com/alesapin)). * The `PREALLOCATE` option for `HASHED`/`SPARSE_HASHED` dictionaries becomes a no-op. [#45388](https://github.com/ClickHouse/ClickHouse/pull/45388) ([Azat Khuzhin](https://github.com/azat)). It does not give significant advantages anymore. * Disallow `Gorilla` codec on columns of non-Float32 or non-Float64 type. [#45252](https://github.com/ClickHouse/ClickHouse/pull/45252) ([Robert Schulze](https://github.com/rschu1ze)). It was pointless and led to inconsistencies. @@ -1983,7 +1983,7 @@ doc_type: 'changelog' * Forbid paths in timezone names. For example, a timezone name like `/usr/share/zoneinfo/Asia/Aden` is not allowed; the IANA timezone database name like `Asia/Aden` should be used. [#44225](https://github.com/ClickHouse/ClickHouse/pull/44225) ([Kruglov Pavel](https://github.com/Avogar)). * Queries combining equijoin and constant expressions (e.g., `JOIN ON t1.x = t2.x AND 1 = 1`) are forbidden due to incorrect results. [#44016](https://github.com/ClickHouse/ClickHouse/pull/44016) ([Vladimir C](https://github.com/vdimir)). -#### New Feature {#new-feature-11} +#### New feature {#new-feature-11} * Dictionary source for extracting keys by traversing regular expressions tree. It can be used for User-Agent parsing. [#40878](https://github.com/ClickHouse/ClickHouse/pull/40878) ([Vage Ogannisian](https://github.com/nooblose)). [#43858](https://github.com/ClickHouse/ClickHouse/pull/43858) ([Han Fei](https://github.com/hanfei1991)). * Added parametrized view functionality, now it's possible to specify query parameters for the View table engine. resolves [#40907](https://github.com/ClickHouse/ClickHouse/issues/40907). [#41687](https://github.com/ClickHouse/ClickHouse/pull/41687) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni)). * Add `quantileInterpolatedWeighted`/`quantilesInterpolatedWeighted` functions. [#38252](https://github.com/ClickHouse/ClickHouse/pull/38252) ([Bharat Nallan](https://github.com/bharatnc)). @@ -2002,13 +2002,13 @@ doc_type: 'changelog' * Try to autodetect headers with column names (and maybe types) for CSV/TSV/CustomSeparated input formats. Add settings input_format_tsv/csv/custom_detect_header that enable this behaviour (enabled by default). Closes [#44640](https://github.com/ClickHouse/ClickHouse/issues/44640). [#44953](https://github.com/ClickHouse/ClickHouse/pull/44953) ([Kruglov Pavel](https://github.com/Avogar)). -#### Experimental Feature {#experimental-feature-7} +#### Experimental feature {#experimental-feature-7} * Add an experimental inverted index as a new secondary index type for efficient text search. [#38667](https://github.com/ClickHouse/ClickHouse/pull/38667) ([larryluogit](https://github.com/larryluogit)). * Add experimental query result cache. [#43797](https://github.com/ClickHouse/ClickHouse/pull/43797) ([Robert Schulze](https://github.com/rschu1ze)). * Added extendable and configurable scheduling subsystem for IO requests (not yet integrated with IO code itself). [#41840](https://github.com/ClickHouse/ClickHouse/pull/41840) ([Sergei Trifonov](https://github.com/serxa)). This feature does nothing at all, enjoy. * Added `SYSTEM DROP DATABASE REPLICA` that removes metadata of a dead replica of a `Replicated` database. Resolves [#41794](https://github.com/ClickHouse/ClickHouse/issues/41794). [#42807](https://github.com/ClickHouse/ClickHouse/pull/42807) ([Alexander Tokmakov](https://github.com/tavplubix)). -#### Performance Improvement {#performance-improvement-11} +#### Performance improvement {#performance-improvement-11} * Do not load inactive parts at startup of `MergeTree` tables. [#42181](https://github.com/ClickHouse/ClickHouse/pull/42181) ([Anton Popov](https://github.com/CurtizJ)). * Improved latency of reading from storage `S3` and table function `s3` with large numbers of small files. Now settings `remote_filesystem_read_method` and `remote_filesystem_read_prefetch` take effect while reading from storage `S3`. [#43726](https://github.com/ClickHouse/ClickHouse/pull/43726) ([Anton Popov](https://github.com/CurtizJ)). * Optimization for reading struct fields in Parquet/ORC files. Only the required fields are loaded. [#44484](https://github.com/ClickHouse/ClickHouse/pull/44484) ([lgbo](https://github.com/lgbo-ustc)). @@ -2068,7 +2068,7 @@ Add settings input_format_tsv/csv/custom_detect_header that enable this behaviou * Added settings to disallow concurrent backups and restores resolves [#43891](https://github.com/ClickHouse/ClickHouse/issues/43891) Implementation: * Added server-level settings to disallow concurrent backups and restores, which are read and set when BackupWorker is created in Context. * Settings are set to true by default. * Before starting backup or restores, added a check to see if any other backups/restores are running. For internal requests, it checks if it is from the self node using backup_uuid. [#45072](https://github.com/ClickHouse/ClickHouse/pull/45072) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni)). * Add `` config parameter for system logs. [#45320](https://github.com/ClickHouse/ClickHouse/pull/45320) ([Stig Bakken](https://github.com/stigsb)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-11} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-11} * Statically link with the `skim` library (it is written in Rust) for fuzzy search in clickhouse client/local history. [#44239](https://github.com/ClickHouse/ClickHouse/pull/44239) ([Azat Khuzhin](https://github.com/azat)). * We removed support for shared linking because of Rust. Actually, Rust is only an excuse for this removal, and we wanted to remove it nevertheless. [#44828](https://github.com/ClickHouse/ClickHouse/pull/44828) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Remove the dependency on the `adduser` tool from the packages, because we don't use it. This fixes [#44934](https://github.com/ClickHouse/ClickHouse/issues/44934). [#45011](https://github.com/ClickHouse/ClickHouse/pull/45011) ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -2078,7 +2078,7 @@ Add settings input_format_tsv/csv/custom_detect_header that enable this behaviou * Automatically merge green backport PRs and green approved PRs [#41110](https://github.com/ClickHouse/ClickHouse/pull/41110) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). * Introduce a [website](https://aretestsgreenyet.com/) for the status of ClickHouse CI. [Source](https://github.com/ClickHouse/aretestsgreenyet). -#### Bug Fix {#bug-fix} +#### Bug fix {#bug-fix} * Replace domain IP types (IPv4, IPv6) with native. [#43221](https://github.com/ClickHouse/ClickHouse/pull/43221) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). It automatically fixes some missing implementations in the code. * Fix the backup process if mutations get killed during the backup process. [#45351](https://github.com/ClickHouse/ClickHouse/pull/45351) ([Vitaly Baranov](https://github.com/vitlibar)). diff --git a/docs/whats-new/changelog/2024.md b/docs/whats-new/changelog/2024.md index 766b5f47445..b232583aa19 100644 --- a/docs/whats-new/changelog/2024.md +++ b/docs/whats-new/changelog/2024.md @@ -8,7 +8,7 @@ keywords: ['ClickHouse 2024', 'changelog 2024', 'release notes', 'version histor doc_type: 'changelog' --- -### Table of Contents {#table-of-contents} +### Table of contents {#table-of-contents} **[ClickHouse release v24.12, 2024-12-19](/whats-new/changelog/2024#a-id2412a-clickhouse-release-2412-2024-12-19)**
**[ClickHouse release v24.11, 2024-11-26](/whats-new/changelog/2024#a-id2411a-clickhouse-release-2411-2024-11-26)**
**[ClickHouse release v24.10, 2024-10-31](/whats-new/changelog/2024#a-id2410a-clickhouse-release-2410-2024-10-31)**
@@ -23,13 +23,13 @@ doc_type: 'changelog' **[ClickHouse release v24.1, 2024-01-30](/whats-new/changelog/2024#a-id241a-clickhouse-release-241-2024-01-30)**
**[Changelog for 2023](/whats-new/changelog/2023/)**
-### ClickHouse release 24.12, 2024-12-19 {#a-id2412a-clickhouse-release-2412-2024-12-19} +### ClickHouse release 24.12, 2024-12-19 {#a-id2412a-clickhouse-release-2412-2024-12-19} -#### Backward Incompatible Change {#backward-incompatible-change} +#### Backward incompatible change {#backward-incompatible-change} * Functions `greatest` and `least` now ignore NULL input values, whereas they previously returned NULL if one of the arguments was NULL. For example, `SELECT greatest(1, 2, NULL)` now returns 2. This makes the behavior compatible with PostgreSQL, but at the same time it breaks the compatibility with MySQL which returns NULL. To retain the previous behavior, set setting `least_greatest_legacy_null_behavior` (default: `false`) to `true`. [#65519](https://github.com/ClickHouse/ClickHouse/pull/65519) [#73344](https://github.com/ClickHouse/ClickHouse/pull/73344) ([kevinyhzou](https://github.com/KevinyhZou)). * A new MongoDB integration is now the default. Users who like to use the legacy MongoDB driver (based on the Poco driver) can enable server setting `use_legacy_mongodb_integration`. [#73359](https://github.com/ClickHouse/ClickHouse/pull/73359) ([Kirill Nikiforov](https://github.com/allmazz). -#### New Feature {#new-feature} +#### New feature {#new-feature} * Move `JSON`/`Dynamic`/`Variant` types from experimental features to beta. [#72294](https://github.com/ClickHouse/ClickHouse/pull/72294) ([Pavel Kruglov](https://github.com/Avogar)). We also backported all fixes as well as this change to 24.11. * Schema evolution for the [Iceberg data storage](https://iceberg.apache.org/spec/#file-system-operations) format provides the user with extensive options for modifying the schema of their table. The order of columns, column names, and simple type extensions can be changed under the hood. [#69445](https://github.com/ClickHouse/ClickHouse/pull/69445) ([Daniil Ivanik](https://github.com/divanik)). * Integrate with Iceberg REST Catalog: a new database engine, named Iceberg, which plugs the whole catalog into ClickHouse. [#71542](https://github.com/ClickHouse/ClickHouse/pull/71542) ([Kseniia Sumarokova](https://github.com/kssenii)). @@ -45,10 +45,10 @@ doc_type: 'changelog' * Allows to use a delimiter as an optional second argument for aggregate function `groupConcat`. [#72540](https://github.com/ClickHouse/ClickHouse/pull/72540) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). * Function `translate` now supports character deletion if the `from` argument contains more characters than the `to` argument. Example: `SELECT translate('clickhouse', 'clickhouse', 'CLICK')` now returns `CLICK`. [#71441](https://github.com/ClickHouse/ClickHouse/pull/71441) ([shuai.xu](https://github.com/shuai-xu)). -#### Experimental Features {#experimental-features} +#### Experimental features {#experimental-features} * A new MergeTree setting `allow_experimental_reverse_key` that enables support for descending sort order in MergeTree sorting keys. This is useful for time series analysis, especially TopN queries. Example usage: `ENGINE = MergeTree ORDER BY (time DESC, key)`- descending order for the `time` field. [#71095](https://github.com/ClickHouse/ClickHouse/pull/71095) ([Amos Bird](https://github.com/amosbird)). -#### Performance Improvement {#performance-improvement} +#### Performance improvement {#performance-improvement} * JOIN reordering. Added an option to select the side of the join that will act as the inner (build) table in the query plan. This is controlled by `query_plan_join_swap_table`, which can be set to `auto`. In this mode, ClickHouse will try to choose the table with the smallest number of rows. [#71577](https://github.com/ClickHouse/ClickHouse/pull/71577) ([Vladimir Cherkasov](https://github.com/vdimir)). * Now `parallel_hash` algorithm will be used (if applicable) when the `join_algorithm` setting is set to `default`. Two previous alternatives (`direct` and `hash`) are still considered when `parallel_hash` cannot be used. [#70788](https://github.com/ClickHouse/ClickHouse/pull/70788) ([Nikita Taranov](https://github.com/nickitat)). * Add option to extract common expressions from `WHERE` and `ON` expressions in order to reduce the number of hash tables used during joins. This makes sense when the JOIN ON condition has common parts inside AND in different OR parts. Can be enabled by `optimize_extract_common_expressions = 1`. [#71537](https://github.com/ClickHouse/ClickHouse/pull/71537) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). @@ -95,7 +95,7 @@ doc_type: 'changelog' * Added `JoinBuildTableRowCount`/`JoinProbeTableRowCount/JoinResultRowCount` profile events. [#72842](https://github.com/ClickHouse/ClickHouse/pull/72842) ([Vladimir Cherkasov](https://github.com/vdimir)). * Support subcolumns in MergeTree sorting key and skip indexes. [#72644](https://github.com/ClickHouse/ClickHouse/pull/72644) ([Pavel Kruglov](https://github.com/Avogar)). -#### Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release} +#### Bug fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release} * Fix possible intersecting parts for MergeTree (after an operation of moving part to the detached directory has been failed, possibly due to operation on object storage). [#70476](https://github.com/ClickHouse/ClickHouse/pull/70476) ([Azat Khuzhin](https://github.com/azat)). * Fixes an error detection when a table name is too long. Provide a diagnostic telling the maximum length. Add a new function `getMaxTableNameLengthForDatabase`. [#70810](https://github.com/ClickHouse/ClickHouse/pull/70810) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). * Fixed zombie processes after a crash of `clickhouse-library-bridge` (this program allows to run unsafe libraries). [#71301](https://github.com/ClickHouse/ClickHouse/pull/71301) ([MikhailBurdukov](https://github.com/MikhailBurdukov)). @@ -133,15 +133,15 @@ doc_type: 'changelog' * Fix possible overestimation of memory tracking (when the difference between `MemoryTracking` and `MemoryResident` kept growing). [#73081](https://github.com/ClickHouse/ClickHouse/pull/73081) ([Azat Khuzhin](https://github.com/azat)). * Check for duplicate JSON keys during Tuple parsing. Previously it could lead to a logical error `Invalid number of rows in Chunk` during parsing. [#73082](https://github.com/ClickHouse/ClickHouse/pull/73082) ([Pavel Kruglov](https://github.com/Avogar)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement} * All small utilities previously stored in `/utils` folder and required manual compilation from sources are now a part of main ClickHouse bundle. This closes: [#72404](https://github.com/ClickHouse/ClickHouse/issues/72404). [#72426](https://github.com/ClickHouse/ClickHouse/pull/72426) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Get rid of `/etc/systemd/system/clickhouse-server.service` removal introduced in 22.3 [#39323](https://github.com/ClickHouse/ClickHouse/issues/39323). [#72259](https://github.com/ClickHouse/ClickHouse/pull/72259) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). * Split large translation units to avoid compilation failures due to memory/cpu limitations. [#72352](https://github.com/ClickHouse/ClickHouse/pull/72352) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). * OSX: Build with ICU support, which enables collations, charset conversions and other localization features. [#73083](https://github.com/ClickHouse/ClickHouse/pull/73083) ([Raúl Marín](https://github.com/Algunenano)). -### ClickHouse release 24.11, 2024-11-26 {#a-id2411a-clickhouse-release-2411-2024-11-26} +### ClickHouse release 24.11, 2024-11-26 {#a-id2411a-clickhouse-release-2411-2024-11-26} -#### Backward Incompatible Change {#backward-incompatible-change-1} +#### Backward incompatible change {#backward-incompatible-change-1} * Remove system tables `generate_series` and `generateSeries`. They were added by mistake here: [#59390](https://github.com/ClickHouse/ClickHouse/issues/59390). [#71091](https://github.com/ClickHouse/ClickHouse/pull/71091) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Remove `StorageExternalDistributed`. Closes [#70600](https://github.com/ClickHouse/ClickHouse/issues/70600).[#71176](https://github.com/ClickHouse/ClickHouse/pull/71176) ([flynn](https://github.com/ucasfl)). * The table engines Kafka, NATS and RabbitMQ are now covered by their own grants in the `SOURCES` hierarchy. Add grants to any non-default database users that create tables with these engine types. [#71250](https://github.com/ClickHouse/ClickHouse/pull/71250) ([Christoph Wurm](https://github.com/cwurm)). @@ -158,7 +158,7 @@ doc_type: 'changelog' * Don't allow Variant/Dynamic types in ORDER BY/GROUP BY/PARTITION BY/PRIMARY KEY by default because it may lead to unexpected results. [#69731](https://github.com/ClickHouse/ClickHouse/pull/69731) ([Pavel Kruglov](https://github.com/Avogar)). * Forbid Dynamic/Variant types in min/max functions to avoid confusion. [#71761](https://github.com/ClickHouse/ClickHouse/pull/71761) ([Pavel Kruglov](https://github.com/Avogar)). -#### New Feature {#new-feature-1} +#### New feature {#new-feature-1} * Added SQL syntax to describe workload and resource management. https://clickhouse.com/docs/operations/workload-scheduling. [#69187](https://github.com/ClickHouse/ClickHouse/pull/69187) ([Sergei Trifonov](https://github.com/serxa)). * A new data type, `BFloat16`, represents 16-bit floating point numbers with 8-bit exponent, sign, and 7-bit mantissa. This closes [#44206](https://github.com/ClickHouse/ClickHouse/issues/44206). This closes [#49937](https://github.com/ClickHouse/ClickHouse/issues/49937). [#64712](https://github.com/ClickHouse/ClickHouse/pull/64712) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Add `CHECK GRANT` query to check whether the current user/role has been granted the specific privilege and whether the corresponding table/column exists in the memory. [#68885](https://github.com/ClickHouse/ClickHouse/pull/68885) ([Unalian](https://github.com/Unalian)). @@ -168,7 +168,7 @@ doc_type: 'changelog' * Allow each authentication method to have its own expiration date, remove from user entity. [#70090](https://github.com/ClickHouse/ClickHouse/pull/70090) ([Arthur Passos](https://github.com/arthurpassos)). * Added new functions `parseDateTime64`, `parseDateTime64OrNull` and `parseDateTime64OrZero`. Compared to the existing function `parseDateTime` (and variants), they return a value of type `DateTime64` instead of `DateTime`. [#71581](https://github.com/ClickHouse/ClickHouse/pull/71581) ([kevinyhzou](https://github.com/KevinyhZou)). -#### Performance Improvement {#performance-improvement-1} +#### Performance improvement {#performance-improvement-1} * Optimized memory usage for values of index granularity if granularity is constant for part. Added an ability to always select constant granularity for part (setting `use_const_adaptive_granularity`), which helps to ensure that it is always optimized in memory. It helps in large workloads (trillions of rows in shared storage) to avoid constantly growing memory usage by metadata (values of index granularity) of data parts. [#71786](https://github.com/ClickHouse/ClickHouse/pull/71786) ([Anton Popov](https://github.com/CurtizJ)). * Now we don't copy input blocks columns for `join_algorithm = 'parallel_hash'` when distribute them between threads for parallel processing. [#67782](https://github.com/ClickHouse/ClickHouse/pull/67782) ([Nikita Taranov](https://github.com/nickitat)). * Optimized `Replacing` merge algorithm for non-intersecting parts. [#70977](https://github.com/ClickHouse/ClickHouse/pull/70977) ([Anton Popov](https://github.com/CurtizJ)). @@ -224,7 +224,7 @@ doc_type: 'changelog' * On the advanced dashboard HTML page added a dropdown selector for the dashboard from `system.dashboards` table. [#72081](https://github.com/ClickHouse/ClickHouse/pull/72081) ([Sergei Trifonov](https://github.com/serxa)). * Check if default database is present after authorization. Fixes [#71097](https://github.com/ClickHouse/ClickHouse/issues/71097). [#71140](https://github.com/ClickHouse/ClickHouse/pull/71140) ([Konstantin Bogdanov](https://github.com/thevar1able)). -#### Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-1} +#### Bug fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-1} * The parts deduplicated during `ATTACH PART` query don't get stuck with the `attaching_` prefix anymore. [#65636](https://github.com/ClickHouse/ClickHouse/pull/65636) ([Kirill](https://github.com/kirillgarbar)). * Fix for the bug when DateTime64 losing precision for the `IN` function. [#67230](https://github.com/ClickHouse/ClickHouse/pull/67230) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). * Fix possible logical error when using functions with `IGNORE/RESPECT NULLS` in `ORDER BY ... WITH FILL`, close [#57609](https://github.com/ClickHouse/ClickHouse/issues/57609). [#68234](https://github.com/ClickHouse/ClickHouse/pull/68234) ([Vladimir Cherkasov](https://github.com/vdimir)). @@ -294,14 +294,14 @@ doc_type: 'changelog' * This is a fix for "zero-copy" replication, which is unsupported and will be removed entirely. Don't delete a blob when there are nodes using it in ReplicatedMergeTree with zero-copy replication. [#71186](https://github.com/ClickHouse/ClickHouse/pull/71186) ([Antonio Andelic](https://github.com/antonio2368)). * This is a fix for "zero-copy" replication, which is unsupported and will be removed entirely. Acquiring zero-copy shared lock before moving a part to zero-copy disk to prevent possible data loss if Keeper is unavailable. [#71845](https://github.com/ClickHouse/ClickHouse/pull/71845) ([Aleksei Filatov](https://github.com/aalexfvk)). -### ClickHouse release 24.10, 2024-10-31 {#a-id2410a-clickhouse-release-2410-2024-10-31} +### ClickHouse release 24.10, 2024-10-31 {#a-id2410a-clickhouse-release-2410-2024-10-31} -#### Backward Incompatible Change {#backward-incompatible-change-2} +#### Backward incompatible change {#backward-incompatible-change-2} * Allow to write `SETTINGS` before `FORMAT` in a chain of queries with `UNION` when subqueries are inside parentheses. This closes [#39712](https://github.com/ClickHouse/ClickHouse/issues/39712). Change the behavior when a query has the SETTINGS clause specified twice in a sequence. The closest SETTINGS clause will have a preference for the corresponding subquery. In the previous versions, the outermost SETTINGS clause could take a preference over the inner one. [#68614](https://github.com/ClickHouse/ClickHouse/pull/68614) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Reordering of filter conditions from `[PRE]WHERE` clause is now allowed by default. It could be disabled by setting `allow_reorder_prewhere_conditions` to `false`. [#70657](https://github.com/ClickHouse/ClickHouse/pull/70657) ([Nikita Taranov](https://github.com/nickitat)). * Remove the `idxd-config` library, which has an incompatible license. This also removes the experimental Intel DeflateQPL codec. [#70987](https://github.com/ClickHouse/ClickHouse/pull/70987) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### New Feature {#new-feature-2} +#### New feature {#new-feature-2} * Allow to grant access to the wildcard prefixes. `GRANT SELECT ON db.table_pefix_* TO user`. [#65311](https://github.com/ClickHouse/ClickHouse/pull/65311) ([pufit](https://github.com/pufit)). * If you press space bar during query runtime, the client will display a real-time table with detailed metrics. You can enable it globally with the new `--progress-table` option in clickhouse-client; a new `--enable-progress-table-toggle` is associated with the `--progress-table` option, and toggles the rendering of the progress table by pressing the control key (Space). [#63689](https://github.com/ClickHouse/ClickHouse/pull/63689) ([Maria Khristenko](https://github.com/mariaKhr)), [#70423](https://github.com/ClickHouse/ClickHouse/pull/70423) ([Julia Kartseva](https://github.com/jkartseva)). * Allow to cache read files for object storage table engines and data lakes using hash from ETag + file path as cache key. [#70135](https://github.com/ClickHouse/ClickHouse/pull/70135) ([Kseniia Sumarokova](https://github.com/kssenii)). @@ -330,7 +330,7 @@ doc_type: 'changelog' * Introduced a special (experimental) mode of a merge selector for MergeTree tables which makes it more aggressive for the partitions that are close to the limit by the number of parts. It is controlled by the `merge_selector_use_blurry_base` MergeTree-level setting. [#70645](https://github.com/ClickHouse/ClickHouse/pull/70645) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Implement generic ser/de between Avro's `Union` and ClickHouse's `Variant` types. Resolves [#69713](https://github.com/ClickHouse/ClickHouse/issues/69713). [#69712](https://github.com/ClickHouse/ClickHouse/pull/69712) ([Jiří Kozlovský](https://github.com/jirislav)). -#### Performance Improvement {#performance-improvement-2} +#### Performance improvement {#performance-improvement-2} * Refactor `IDisk` and `IObjectStorage` for better performance. Tables from `plain` and `plain_rewritable` object storages will initialize faster. [#68146](https://github.com/ClickHouse/ClickHouse/pull/68146) ([Alexey Milovidov](https://github.com/alexey-milovidov), [Julia Kartseva](https://github.com/jkartseva)). Do not call the LIST object storage API when determining if a file or directory exists on the plain rewritable disk, as it can be cost-inefficient. [#70852](https://github.com/ClickHouse/ClickHouse/pull/70852) ([Julia Kartseva](https://github.com/jkartseva)). Reduce the number of object storage HEAD API requests in the plain_rewritable disk. [#70915](https://github.com/ClickHouse/ClickHouse/pull/70915) ([Julia Kartseva](https://github.com/jkartseva)). * Added an ability to parse data directly into sparse columns. [#69828](https://github.com/ClickHouse/ClickHouse/pull/69828) ([Anton Popov](https://github.com/CurtizJ)). * Improved performance of parsing formats with high number of missed values (e.g. `JSONEachRow`). [#69875](https://github.com/ClickHouse/ClickHouse/pull/69875) ([Anton Popov](https://github.com/CurtizJ)). @@ -390,7 +390,7 @@ doc_type: 'changelog' * Support for atomic `CREATE OR REPLACE VIEW`. [#70536](https://github.com/ClickHouse/ClickHouse/pull/70536) ([tuanpach](https://github.com/tuanpach)) * Added `strict_once` mode to aggregate function `windowFunnel` to avoid counting one event several times in case it matches multiple conditions, close [#21835](https://github.com/ClickHouse/ClickHouse/issues/21835). [#69738](https://github.com/ClickHouse/ClickHouse/pull/69738) ([Vladimir Cherkasov](https://github.com/vdimir)). -#### Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-2} +#### Bug fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-2} * Apply configuration updates in global context object. It fixes issues like [#62308](https://github.com/ClickHouse/ClickHouse/issues/62308). [#62944](https://github.com/ClickHouse/ClickHouse/pull/62944) ([Amos Bird](https://github.com/amosbird)). * Fix `ReadSettings` not using user set values, because defaults were only used. [#65625](https://github.com/ClickHouse/ClickHouse/pull/65625) ([Kseniia Sumarokova](https://github.com/kssenii)). * Fix type mismatch issue in `sumMapFiltered` when using signed arguments. [#58408](https://github.com/ClickHouse/ClickHouse/pull/58408) ([Chen768959](https://github.com/Chen768959)). @@ -452,14 +452,14 @@ doc_type: 'changelog' * Fix a logical error due to negative zeros in the two-level hash table. This closes [#70973](https://github.com/ClickHouse/ClickHouse/issues/70973). [#70979](https://github.com/ClickHouse/ClickHouse/pull/70979) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Fix `limit by`, `limit with ties` for distributed and parallel replicas. [#70880](https://github.com/ClickHouse/ClickHouse/pull/70880) ([Nikita Taranov](https://github.com/nickitat)). -### ClickHouse release 24.9, 2024-09-26 {#a-id249a-clickhouse-release-249-2024-09-26} +### ClickHouse release 24.9, 2024-09-26 {#a-id249a-clickhouse-release-249-2024-09-26} -#### Backward Incompatible Change {#backward-incompatible-change-3} +#### Backward incompatible change {#backward-incompatible-change-3} * Expressions like `a[b].c` are supported for named tuples, as well as named subscripts from arbitrary expressions, e.g., `expr().name`. This is useful for processing JSON. This closes [#54965](https://github.com/ClickHouse/ClickHouse/issues/54965). In previous versions, an expression of form `expr().name` was parsed as `tupleElement(expr(), name)`, and the query analyzer was searching for a column `name` rather than for the corresponding tuple element; while in the new version, it is changed to `tupleElement(expr(), 'name')`. In most cases, the previous version was not working, but it is possible to imagine a very unusual scenario when this change could lead to incompatibility: if you stored names of tuple elements in a column or an alias, that was named differently than the tuple element's name: `SELECT 'b' AS a, CAST([tuple(123)] AS 'Array(Tuple(b UInt8))') AS t, t[1].a`. It is very unlikely that you used such queries, but we still have to mark this change as potentially backward incompatible. [#68435](https://github.com/ClickHouse/ClickHouse/pull/68435) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * When the setting `print_pretty_type_names` is enabled, it will print `Tuple` data type in a pretty form in `SHOW CREATE TABLE` statements, `formatQuery` function, and in the interactive mode in `clickhouse-client` and `clickhouse-local`. In previous versions, this setting was only applied to `DESCRIBE` queries and `toTypeName`. This closes [#65753](https://github.com/ClickHouse/ClickHouse/issues/65753). [#68492](https://github.com/ClickHouse/ClickHouse/pull/68492) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Do not allow explicitly specifying UUID when creating a table in `Replicated` databases. Also, do not allow explicitly specifying Keeper path and replica name for *MergeTree tables in Replicated databases. It introduces a new setting `database_replicated_allow_explicit_uuid` and changes the type of `database_replicated_allow_replicated_engine_arguments` from Bool to UInt64 [#66104](https://github.com/ClickHouse/ClickHouse/pull/66104) ([Alexander Tokmakov](https://github.com/tavplubix)). -#### New Feature {#new-feature-3} +#### New feature {#new-feature-3} * Allow a user to have multiple authentication methods instead of only one. Allow authentication methods to be reset to most recently added method. If you want to run instances on 24.8 and one on 24.9 for some time, it's better to set `max_authentication_methods_per_user` = 1 for that period to avoid potential incompatibilities. [#65277](https://github.com/ClickHouse/ClickHouse/pull/65277) ([Arthur Passos](https://github.com/arthurpassos)). * Add support for `ATTACH PARTITION ALL FROM`. [#61987](https://github.com/ClickHouse/ClickHouse/pull/61987) ([Kirill Nikiforov](https://github.com/allmazz)). * Add the `input_format_json_empty_as_default` setting which, when enabled, treats empty fields in JSON inputs as default values. Closes [#59339](https://github.com/ClickHouse/ClickHouse/issues/59339). [#66782](https://github.com/ClickHouse/ClickHouse/pull/66782) ([Alexis Arnaud](https://github.com/a-a-f)). @@ -486,7 +486,7 @@ doc_type: 'changelog' * Improve castOrDefault from Variant/Dynamic columns so it works when inner types are not convertible at all. [#67150](https://github.com/ClickHouse/ClickHouse/pull/67150) ([Kruglov Pavel](https://github.com/Avogar)). * Replication of subset of columns is now available through MaterializedPostgreSQL. Closes [#33748](https://github.com/ClickHouse/ClickHouse/issues/33748). [#69092](https://github.com/ClickHouse/ClickHouse/pull/69092) ([Kruglov Kirill](https://github.com/1on)). -#### Performance Improvement {#performance-improvement-3} +#### Performance improvement {#performance-improvement-3} * Implemented reading of required files only for Hive partitioning. [#68963](https://github.com/ClickHouse/ClickHouse/pull/68963) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). * Improve the JOIN performance by rearranging the right table by keys while the table keys are dense in the LEFT or INNER hash joins. [#60341](https://github.com/ClickHouse/ClickHouse/pull/60341) ([kevinyhzou](https://github.com/KevinyhZou)). * Improve ALL JOIN performance by appending the list of rows lazily. [#63677](https://github.com/ClickHouse/ClickHouse/pull/63677) ([kevinyhzou](https://github.com/KevinyhZou)). @@ -535,7 +535,7 @@ doc_type: 'changelog' * Masking sensitive info in `gcs` table function. [#69611](https://github.com/ClickHouse/ClickHouse/pull/69611) ([Vitaly Baranov](https://github.com/vitlibar)). * Rebuild projection for merges that reduce number of rows. [#62364](https://github.com/ClickHouse/ClickHouse/pull/62364) ([cangyin](https://github.com/cangyin)). -#### Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-3} +#### Bug fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-3} * Fix attaching table when pg dbname contains "-" in the experimental and unsupported MaterializedPostgreSQL engine. [#62730](https://github.com/ClickHouse/ClickHouse/pull/62730) ([takakawa](https://github.com/takakawa)). * Fixed error on generated columns in the experimental and totally unsupported MaterializedPostgreSQL engine when adnum ordering is broken [#63161](https://github.com/ClickHouse/ClickHouse/issues/63161). Fixed error on id column with nextval expression as default in the experimental and totally unsupported MaterializedPostgreSQL when there are generated columns in table. Fixed error on dropping publication with symbols except \[a-z1-9-\]. [#67664](https://github.com/ClickHouse/ClickHouse/pull/67664) ([Kruglov Kirill](https://github.com/1on)). * Storage Join to support Nullable columns in the left table, close [#61247](https://github.com/ClickHouse/ClickHouse/issues/61247). [#66926](https://github.com/ClickHouse/ClickHouse/pull/66926) ([vdimir](https://github.com/vdimir)). @@ -613,9 +613,9 @@ doc_type: 'changelog' * Use tryconvertfieldtotype in gethyperrectangleforrowgroup. [#69745](https://github.com/ClickHouse/ClickHouse/pull/69745) ([Miсhael Stetsyuk](https://github.com/mstetsyuk)). * Revert "Fix prewhere without columns and without adaptive index granularity (almost w/o anything)"'. Due to the reverted changes some errors might happen when reading data parts produced by old CH releases (presumably 2021 or older). [#68897](https://github.com/ClickHouse/ClickHouse/pull/68897) ([Alexander Gololobov](https://github.com/davenger)). -### ClickHouse release 24.8 LTS, 2024-08-20 {#a-id248a-clickhouse-release-248-lts-2024-08-20} +### ClickHouse release 24.8 lts, 2024-08-20 {#a-id248a-clickhouse-release-248-lts-2024-08-20} -#### Backward Incompatible Change {#backward-incompatible-change-4} +#### Backward incompatible change {#backward-incompatible-change-4} * `clickhouse-client` and `clickhouse-local` now default to multi-query mode (instead single-query mode). As an example, `clickhouse-client -q "SELECT 1; SELECT 2"` now works, whereas users previously had to add `--multiquery` (or `-n`). The `--multiquery/-n` switch became obsolete. INSERT queries in multi-query statements are treated specially based on their FORMAT clause: If the FORMAT is `VALUES` (the most common case), the end of the INSERT statement is represented by a trailing semicolon `;` at the end of the query. For all other FORMATs (e.g. `CSV` or `JSONEachRow`), the end of the INSERT statement is represented by two newlines `\n\n` at the end of the query. [#63898](https://github.com/ClickHouse/ClickHouse/pull/63898) ([FFish](https://github.com/wxybear)). * In previous versions, it was possible to use an alternative syntax for `LowCardinality` data types by appending `WithDictionary` to the name of the data type. It was an initial working implementation, and it was never documented or exposed to the public. Now, it is deprecated. If you have used this syntax, you have to ALTER your tables and rename the data types to `LowCardinality`. [#66842](https://github.com/ClickHouse/ClickHouse/pull/66842) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Fix logical errors with storage `Buffer` used with distributed destination table. It's a backward incompatible change: queries using `Buffer` with a distributed destination table may stop working if the table appears more than once in the query (e.g., in a self-join). [#67015](https://github.com/ClickHouse/ClickHouse/pull/67015) ([vdimir](https://github.com/vdimir)). @@ -625,7 +625,7 @@ doc_type: 'changelog' * Fix REPLACE modifier formatting (forbid omitting brackets). [#67774](https://github.com/ClickHouse/ClickHouse/pull/67774) ([Azat Khuzhin](https://github.com/azat)). * Backported in [#68349](https://github.com/ClickHouse/ClickHouse/issues/68349): Reimplement `Dynamic` type. Now when the limit of dynamic data types is reached new types are not cast to String but stored in a special data structure in binary format with binary encoded data type. Now any type ever inserted into `Dynamic` column can be read from it as subcolumn. [#68132](https://github.com/ClickHouse/ClickHouse/pull/68132) ([Kruglov Pavel](https://github.com/Avogar)). -#### New Feature {#new-feature-4} +#### New feature {#new-feature-4} * Added a new `MergeTree` setting `deduplicate_merge_projection_mode` to control the projections during merges (for specific engines) and `OPTIMIZE DEDUPLICATE` query. Supported options: `throw` (throw an exception in case the projection is not fully supported for *MergeTree engine), `drop` (remove projection during merge if it can't be merged itself consistently) and `rebuild` (rebuild projection from scratch, which is a heavy operation). [#66672](https://github.com/ClickHouse/ClickHouse/pull/66672) ([jsc0218](https://github.com/jsc0218)). * Add `_etag` virtual column for S3 table engine. Fixes [#65312](https://github.com/ClickHouse/ClickHouse/issues/65312). [#65386](https://github.com/ClickHouse/ClickHouse/pull/65386) ([skyoct](https://github.com/skyoct)). * Added a tagging (namespace) mechanism for the query cache. The same queries with different tags are considered different by the query cache. Example: `SELECT 1 SETTINGS use_query_cache = 1, query_cache_tag = 'abc'` and `SELECT 1 SETTINGS use_query_cache = 1, query_cache_tag = 'def'` now create different query cache entries. [#68235](https://github.com/ClickHouse/ClickHouse/pull/68235) ([sakulali](https://github.com/sakulali)). @@ -640,14 +640,14 @@ doc_type: 'changelog' * Support `OPTIMIZE` query on `Join` tables to reduce their memory footprint. [#67883](https://github.com/ClickHouse/ClickHouse/pull/67883) ([Duc Canh Le](https://github.com/canhld94)). * Allow run query instantly in play if you add `&run=1` in the URL [#66457](https://github.com/ClickHouse/ClickHouse/pull/66457) ([Aleksandr Musorin](https://github.com/AVMusorin)). -#### Experimental Feature {#experimental-feature-3} +#### Experimental feature {#experimental-feature-3} * Implement a new `JSON` data type. [#66444](https://github.com/ClickHouse/ClickHouse/pull/66444) ([Kruglov Pavel](https://github.com/Avogar)). * Add the new `TimeSeries` table engine. [#64183](https://github.com/ClickHouse/ClickHouse/pull/64183) ([Vitaly Baranov](https://github.com/vitlibar)). * Add new experimental `Kafka` storage engine to store offsets in Keeper instead of relying on committing them to Kafka. It makes the commit to ClickHouse tables atomic with regard to consumption from the queue. [#57625](https://github.com/ClickHouse/ClickHouse/pull/57625) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). * Use adaptive read task size calculation method (adaptive meaning it depends on read column sizes) for parallel replicas. [#60377](https://github.com/ClickHouse/ClickHouse/pull/60377) ([Nikita Taranov](https://github.com/nickitat)). * Added statistics type `count_min` (count-min sketches) which provide selectivity estimations for equality predicates like `col = 'val'`. Supported data types are string, date, datatime and numeric types. [#65521](https://github.com/ClickHouse/ClickHouse/pull/65521) ([JackyWoo](https://github.com/JackyWoo)). -#### Performance Improvement {#performance-improvement-4} +#### Performance improvement {#performance-improvement-4} * Setting `optimize_functions_to_subcolumns` is enabled by default. [#68053](https://github.com/ClickHouse/ClickHouse/pull/68053) ([Anton Popov](https://github.com/CurtizJ)). * Store the `plain_rewritable` disk directory metadata in `__meta` layout, separately from the merge tree data in the object storage. Move the `plain_rewritable` disk to a flat directory structure. [#65751](https://github.com/ClickHouse/ClickHouse/pull/65751) ([Julia Kartseva](https://github.com/jkartseva)). * Improve columns squashing (an operation happening in INSERT queries) for `String`/`Array`/`Map`/`Variant`/`Dynamic` types by reserving required memory in advance for all subcolumns. [#67043](https://github.com/ClickHouse/ClickHouse/pull/67043) ([Kruglov Pavel](https://github.com/Avogar)). @@ -684,13 +684,13 @@ doc_type: 'changelog' * Added profile events for merges and mutations for better introspection. [#68015](https://github.com/ClickHouse/ClickHouse/pull/68015) ([Anton Popov](https://github.com/CurtizJ)). * Remove unnecessary logs for non-replicated `MergeTree`. [#68238](https://github.com/ClickHouse/ClickHouse/pull/68238) ([Daniil Ivanik](https://github.com/divanik)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-1} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-1} * Integration tests flaky check will not run each test case multiple times to find more issues in tests and make them more reliable. It is using `pytest-repeat` library to run test case multiple times for the same environment. It is important to cleanup tables and other entities in the end of a test case to pass. Repeating works much faster than several pytest runs as it starts necessary containers only once. [#66986](https://github.com/ClickHouse/ClickHouse/pull/66986) ([Ilya Yatsishin](https://github.com/qoega)). * Unblock the usage of CLion with ClickHouse. In previous versions, CLion freezed for a minute on every keypress. This closes [#66994](https://github.com/ClickHouse/ClickHouse/issues/66994). [#66995](https://github.com/ClickHouse/ClickHouse/pull/66995) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * getauxval: avoid a crash under a sanitizer re-exec due to high ASLR entropy in newer Linux kernels. [#67081](https://github.com/ClickHouse/ClickHouse/pull/67081) ([Raúl Marín](https://github.com/Algunenano)). * Some parts of client code are extracted to a single file and highest possible level optimization is applied to them even for debug builds. This closes: [#65745](https://github.com/ClickHouse/ClickHouse/issues/65745). [#67215](https://github.com/ClickHouse/ClickHouse/pull/67215) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). -#### Bug Fix {#bug-fix} +#### Bug fix {#bug-fix} * Only relevant to the experimental Variant data type. Fix crash with Variant + AggregateFunction type. [#67122](https://github.com/ClickHouse/ClickHouse/pull/67122) ([Kruglov Pavel](https://github.com/Avogar)). * Fix crash in DistributedAsyncInsert when connection is empty. [#67219](https://github.com/ClickHouse/ClickHouse/pull/67219) ([Pablo Marcos](https://github.com/pamarcos)). * Fix crash of `uniq` and `uniqTheta ` with `tuple()` argument. Closes [#67303](https://github.com/ClickHouse/ClickHouse/issues/67303). [#67306](https://github.com/ClickHouse/ClickHouse/pull/67306) ([flynn](https://github.com/ucasfl)). @@ -757,9 +757,9 @@ doc_type: 'changelog' * Try fix postgres crash when query is cancelled. [#68288](https://github.com/ClickHouse/ClickHouse/pull/68288) ([Kseniia Sumarokova](https://github.com/kssenii)). * Fix missing sync replica mode in query `SYSTEM SYNC REPLICA`. [#68326](https://github.com/ClickHouse/ClickHouse/pull/68326) ([Duc Canh Le](https://github.com/canhld94)). -### ClickHouse release 24.7, 2024-07-30 {#a-id247a-clickhouse-release-247-2024-07-30} +### ClickHouse release 24.7, 2024-07-30 {#a-id247a-clickhouse-release-247-2024-07-30} -#### Backward Incompatible Change {#backward-incompatible-change-5} +#### Backward incompatible change {#backward-incompatible-change-5} * Forbid `CRATE MATERIALIZED VIEW ... ENGINE Replicated*MergeTree POPULATE AS SELECT ...` with Replicated databases. [#63963](https://github.com/ClickHouse/ClickHouse/pull/63963) ([vdimir](https://github.com/vdimir)). * `clickhouse-keeper-client` will only accept paths in string literals, such as `ls '/hello/world'`, not bare strings such as `ls /hello/world`. [#65494](https://github.com/ClickHouse/ClickHouse/pull/65494) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Metric `KeeperOutstandingRequets` was renamed to `KeeperOutstandingRequests`. [#66206](https://github.com/ClickHouse/ClickHouse/pull/66206) ([Robert Schulze](https://github.com/rschu1ze)). @@ -768,7 +768,7 @@ doc_type: 'changelog' * Change how deduplication for Materialized Views works. Fixed a lot of cases like: - on destination table: data is split for 2 or more blocks and that blocks is considered as duplicate when that block is inserted in parallel. - on MV destination table: the equal blocks are deduplicated, that happens when MV often produces equal data as a result for different input data due to performing aggregation. - on MV destination table: the equal blocks which comes from different MV are deduplicated. [#61601](https://github.com/ClickHouse/ClickHouse/pull/61601) ([Sema Checherinda](https://github.com/CheSema)). * Functions `bitShiftLeft` and `bitShitfRight` return an error for out of bounds shift positions [#65838](https://github.com/ClickHouse/ClickHouse/pull/65838) ([Pablo Marcos](https://github.com/pamarcos)). -#### New Feature {#new-feature-5} +#### New feature {#new-feature-5} * Add `ASOF JOIN` support for `full_sorting_join` algorithm. [#55051](https://github.com/ClickHouse/ClickHouse/pull/55051) ([vdimir](https://github.com/vdimir)). * Support JWT authentication in `clickhouse-client` (will be available only in ClickHouse Cloud). [#62829](https://github.com/ClickHouse/ClickHouse/pull/62829) ([Konstantin Bogdanov](https://github.com/thevar1able)). * Add SQL functions `changeYear`, `changeMonth`, `changeDay`, `changeHour`, `changeMinute`, `changeSecond`. For example, `SELECT changeMonth(toDate('2024-06-14'), 7)` returns date `2024-07-14`. [#63186](https://github.com/ClickHouse/ClickHouse/pull/63186) ([cucumber95](https://github.com/cucumber95)). @@ -785,7 +785,7 @@ doc_type: 'changelog' * When lightweight delete happens on a table with projection(s), users have choices either throw an exception (by default) or drop the projection [#65594](https://github.com/ClickHouse/ClickHouse/pull/65594) ([jsc0218](https://github.com/jsc0218)). * Add system tables with main information about all detached tables. [#65400](https://github.com/ClickHouse/ClickHouse/pull/65400) ([Konstantin Morozov](https://github.com/k-morozov)). -#### Experimental Feature {#experimental-feature-4} +#### Experimental feature {#experimental-feature-4} * Change binary serialization of the `Variant` data type: add `compact` mode to avoid writing the same discriminator multiple times for granules with single variant or with only NULL values. Add MergeTree setting `use_compact_variant_discriminators_serialization` that is enabled by default. Note that Variant type is still experimental and backward-incompatible change in serialization is ok. [#62774](https://github.com/ClickHouse/ClickHouse/pull/62774) ([Kruglov Pavel](https://github.com/Avogar)). * Support on-disk backend storage for clickhouse-keeper. [#56626](https://github.com/ClickHouse/ClickHouse/pull/56626) ([Han Fei](https://github.com/hanfei1991)). * Refactor JSONExtract functions, support more types including experimental Dynamic type. [#66046](https://github.com/ClickHouse/ClickHouse/pull/66046) ([Kruglov Pavel](https://github.com/Avogar)). @@ -793,7 +793,7 @@ doc_type: 'changelog' * Fix reading `Dynamic` subcolumns from altered `Memory` table. Previously if `max_types` parameter of a Dynamic type was changed in Memory table via alter, further subcolumns reading can return wrong result. [#66066](https://github.com/ClickHouse/ClickHouse/pull/66066) ([Kruglov Pavel](https://github.com/Avogar)). * Add support for `cluster_for_parallel_replicas` when using custom key parallel replicas. It allows you to use parallel replicas with custom key with MergeTree tables. [#65453](https://github.com/ClickHouse/ClickHouse/pull/65453) ([Antonio Andelic](https://github.com/antonio2368)). -#### Performance Improvement {#performance-improvement-5} +#### Performance improvement {#performance-improvement-5} * Replace int to string algorithm with a faster one (from a modified amdn/itoa to a modified jeaiii/itoa). [#61661](https://github.com/ClickHouse/ClickHouse/pull/61661) ([Raúl Marín](https://github.com/Algunenano)). * Sizes of hash tables created by join (`parallel_hash` algorithm) are collected and cached now. This information will be used to preallocate space in hash tables for subsequent query executions and save time on hash table resizes. [#64553](https://github.com/ClickHouse/ClickHouse/pull/64553) ([Nikita Taranov](https://github.com/nickitat)). * Optimized queries with `ORDER BY` primary key and `WHERE` that have a condition with high selectivity by using buffering. It is controlled by setting `read_in_order_use_buffering` (enabled by default) and can increase memory usage of query. [#64607](https://github.com/ClickHouse/ClickHouse/pull/64607) ([Anton Popov](https://github.com/CurtizJ)). @@ -847,7 +847,7 @@ doc_type: 'changelog' * Disable suspending on `Ctrl+Z` in interactive mode. This is a common trap and is not expected behavior for almost all users. I imagine only a few extreme power users could appreciate suspending terminal applications to the background, but I don't know any. [#66511](https://github.com/ClickHouse/ClickHouse/pull/66511) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Add option for validating the primary key type in Dictionaries. Without this option for simple layouts any column type will be implicitly converted to UInt64. [#66595](https://github.com/ClickHouse/ClickHouse/pull/66595) ([MikhailBurdukov](https://github.com/MikhailBurdukov)). -#### Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-4} +#### Bug fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-4} * Check cyclic dependencies on CREATE/REPLACE/RENAME/EXCHANGE queries and throw an exception if there is a cyclic dependency. Previously such cyclic dependencies could lead to a deadlock during server startup. Also fix some bugs in dependencies creation. [#65405](https://github.com/ClickHouse/ClickHouse/pull/65405) ([Kruglov Pavel](https://github.com/Avogar)). * Fix unexpected sizes of `LowCardinality` columns in function calls. [#65298](https://github.com/ClickHouse/ClickHouse/pull/65298) ([Raúl Marín](https://github.com/Algunenano)). * Fix crash in maxIntersections. [#65689](https://github.com/ClickHouse/ClickHouse/pull/65689) ([Raúl Marín](https://github.com/Algunenano)). @@ -924,16 +924,16 @@ doc_type: 'changelog' * Fix `indexHint` function case found by fuzzer. [#66286](https://github.com/ClickHouse/ClickHouse/pull/66286) ([Anton Popov](https://github.com/CurtizJ)). * Fix AST formatting of 'create table b empty as a'. [#64951](https://github.com/ClickHouse/ClickHouse/pull/64951) ([Michael Kolupaev](https://github.com/al13n321)). -### ClickHouse release 24.6, 2024-07-01 {#a-id246a-clickhouse-release-246-2024-07-01} +### ClickHouse release 24.6, 2024-07-01 {#a-id246a-clickhouse-release-246-2024-07-01} -#### Backward Incompatible Change {#backward-incompatible-change-6} +#### Backward incompatible change {#backward-incompatible-change-6} * Enable asynchronous load of databases and tables by default. See the `async_load_databases` in config.xml. While this change is fully compatible, it can introduce a difference in behavior. When `async_load_databases` is false, as in the previous versions, the server will not accept connections until all tables are loaded. When `async_load_databases` is true, as in the new version, the server can accept connections before all the tables are loaded. If a query is made to a table that is not yet loaded, it will wait for the table's loading, which can take considerable time. It can change the behavior of the server if it is part of a large distributed system under a load balancer. In the first case, the load balancer can get a connection refusal and quickly failover to another server. In the second case, the load balancer can connect to a server that is still loading the tables, and the query will have a higher latency. Moreover, if many queries accumulate in the waiting state, it can lead to a "thundering herd" problem when they start processing simultaneously. This can make a difference only for highly loaded distributed backends. You can set the value of `async_load_databases` to false to avoid this problem. [#57695](https://github.com/ClickHouse/ClickHouse/pull/57695) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Setting `replace_long_file_name_to_hash` is enabled by default for `MergeTree` tables. [#64457](https://github.com/ClickHouse/ClickHouse/pull/64457) ([Anton Popov](https://github.com/CurtizJ)). This setting is fully compatible, and no actions needed during upgrade. The new data format is supported from all versions starting from 23.9. After enabling this setting, you can no longer downgrade to a version 23.8 or older. * Some invalid queries will fail earlier during parsing. Note: disabled the support for inline KQL expressions (the experimental Kusto language) when they are put into a `kql` table function without a string literal, e.g. `kql(garbage | trash)` instead of `kql('garbage | trash')` or `kql($$garbage | trash$$)`. This feature was introduced unintentionally and should not exist. [#61500](https://github.com/ClickHouse/ClickHouse/pull/61500) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Rework parallel processing in `Ordered` mode of storage `S3Queue`. This PR is backward incompatible for Ordered mode if you used settings `s3queue_processing_threads_num` or `s3queue_total_shards_num`. Setting `s3queue_total_shards_num` is deleted, previously it was allowed to use only under `s3queue_allow_experimental_sharded_mode`, which is now deprecated. A new setting is added - `s3queue_buckets`. [#64349](https://github.com/ClickHouse/ClickHouse/pull/64349) ([Kseniia Sumarokova](https://github.com/kssenii)). * New functions `snowflakeIDToDateTime`, `snowflakeIDToDateTime64`, `dateTimeToSnowflakeID`, and `dateTime64ToSnowflakeID` were added. Unlike the existing functions `snowflakeToDateTime`, `snowflakeToDateTime64`, `dateTimeToSnowflake`, and `dateTime64ToSnowflake`, the new functions are compatible with function `generateSnowflakeID`, i.e. they accept the snowflake IDs generated by `generateSnowflakeID` and produce snowflake IDs of the same type as `generateSnowflakeID` (i.e. `UInt64`). Furthermore, the new functions default to the UNIX epoch (aka. 1970-01-01), just like `generateSnowflakeID`. If necessary, a different epoch, e.g. Twitter's/X's epoch 2010-11-04 aka. 1288834974657 msec since UNIX epoch, can be passed. The old conversion functions are deprecated and will be removed after a transition period: to use them regardless, enable setting `allow_deprecated_snowflake_conversion_functions`. [#64948](https://github.com/ClickHouse/ClickHouse/pull/64948) ([Robert Schulze](https://github.com/rschu1ze)). -#### New Feature {#new-feature-6} +#### New feature {#new-feature-6} * Allow to store named collections in ClickHouse Keeper. [#64574](https://github.com/ClickHouse/ClickHouse/pull/64574) ([Kseniia Sumarokova](https://github.com/kssenii)). * Support empty tuples. [#55061](https://github.com/ClickHouse/ClickHouse/pull/55061) ([Amos Bird](https://github.com/amosbird)). * Add Hilbert Curve encode and decode functions. [#60156](https://github.com/ClickHouse/ClickHouse/pull/60156) ([Artem Mustafin](https://github.com/Artemmm91)). @@ -954,13 +954,13 @@ doc_type: 'changelog' * Introduced two additional columns in the `system.query_log`: `used_privileges` and `missing_privileges`. `used_privileges` is populated with the privileges that were checked during query execution, and `missing_privileges` contains required privileges that are missing. [#64597](https://github.com/ClickHouse/ClickHouse/pull/64597) ([Alexey Katsman](https://github.com/alexkats)). * Added a setting `output_format_pretty_display_footer_column_names` which when enabled displays column names at the end of the table for long tables (50 rows by default), with the threshold value for minimum number of rows controlled by `output_format_pretty_display_footer_column_names_min_rows`. [#65144](https://github.com/ClickHouse/ClickHouse/pull/65144) ([Shaun Struwig](https://github.com/Blargian)). -#### Experimental Feature {#experimental-feature-5} +#### Experimental feature {#experimental-feature-5} * Introduce statistics of type "number of distinct values". [#59357](https://github.com/ClickHouse/ClickHouse/pull/59357) ([Han Fei](https://github.com/hanfei1991)). * Support statistics with ReplicatedMergeTree. [#64934](https://github.com/ClickHouse/ClickHouse/pull/64934) ([Han Fei](https://github.com/hanfei1991)). * If "replica group" is configured for a `Replicated` database, automatically create a cluster that includes replicas from all groups. [#64312](https://github.com/ClickHouse/ClickHouse/pull/64312) ([Alexander Tokmakov](https://github.com/tavplubix)). * Add settings `parallel_replicas_custom_key_range_lower` and `parallel_replicas_custom_key_range_upper` to control how parallel replicas with dynamic shards parallelizes queries when using a range filter. [#64604](https://github.com/ClickHouse/ClickHouse/pull/64604) ([josh-hildred](https://github.com/josh-hildred)). -#### Performance Improvement {#performance-improvement-6} +#### Performance improvement {#performance-improvement-6} * Add the ability to reshuffle rows during insert to optimize for size without violating the order set by `PRIMARY KEY`. It's controlled by the setting `optimize_row_order` (off by default). [#63578](https://github.com/ClickHouse/ClickHouse/pull/63578) ([Igor Markelov](https://github.com/ElderlyPassionFruit)). * Add a native parquet reader, which can read parquet binary to ClickHouse Columns directly. It's controlled by the setting `input_format_parquet_use_native_reader` (disabled by default). [#60361](https://github.com/ClickHouse/ClickHouse/pull/60361) ([ZhiHong Zhang](https://github.com/copperybean)). * Support partial trivial count optimization when the query filter is able to select exact ranges from merge tree tables. [#60463](https://github.com/ClickHouse/ClickHouse/pull/60463) ([Amos Bird](https://github.com/amosbird)). @@ -1017,7 +1017,7 @@ doc_type: 'changelog' * Disable transactions for unsupported storages even for materialized views. [#64918](https://github.com/ClickHouse/ClickHouse/pull/64918) ([alesapin](https://github.com/alesapin)). * Forbid `QUALIFY` clause in the old analyzer. The old analyzer ignored `QUALIFY`, so it could lead to unexpected data removal in mutations. [#65356](https://github.com/ClickHouse/ClickHouse/pull/65356) ([Dmitry Novik](https://github.com/novikd)). -#### Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-5} +#### Bug fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-5} * A bug in Apache ORC library was fixed: Fixed ORC statistics calculation, when writing, for unsigned types on all platforms and Int8 on ARM. [#64563](https://github.com/ClickHouse/ClickHouse/pull/64563) ([Michael Kolupaev](https://github.com/al13n321)). * Returned back the behaviour of how ClickHouse works and interprets Tuples in CSV format. This change effectively reverts https://github.com/ClickHouse/ClickHouse/pull/60994 and makes it available only under a few settings: `output_format_csv_serialize_tuple_into_separate_columns`, `input_format_csv_deserialize_separate_columns_into_tuple` and `input_format_csv_try_infer_strings_from_quoted_tuples`. [#65170](https://github.com/ClickHouse/ClickHouse/pull/65170) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Fix a permission error where a user in a specific situation can escalate their privileges on the default database without necessary grants. [#64769](https://github.com/ClickHouse/ClickHouse/pull/64769) ([pufit](https://github.com/pufit)). @@ -1077,7 +1077,7 @@ doc_type: 'changelog' * Fix the bug in Hashed and Hashed_Array dictionary short circuit evaluation, which may read uninitialized number, leading to various errors. [#65256](https://github.com/ClickHouse/ClickHouse/pull/65256) ([jsc0218](https://github.com/jsc0218)). * This PR ensures that the type of the constant(IN operator's second parameter) is always visible during the IN operator's type conversion process. Otherwise, losing type information may cause some conversions to fail, such as the conversion from DateTime to Date. This fixes ([#64487](https://github.com/ClickHouse/ClickHouse/issues/64487)). [#65315](https://github.com/ClickHouse/ClickHouse/pull/65315) ([pn](https://github.com/chloro-pn)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-2} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-2} * Add support for LLVM XRay. [#64592](https://github.com/ClickHouse/ClickHouse/pull/64592) [#64837](https://github.com/ClickHouse/ClickHouse/pull/64837) ([Tomer Shafir](https://github.com/tomershafir)). * Unite s3/hdfs/azure storage implementations into a single class working with IObjectStorage. Same for *Cluster, data lakes and Queue storages. [#59767](https://github.com/ClickHouse/ClickHouse/pull/59767) ([Kseniia Sumarokova](https://github.com/kssenii)). * Refactor data part writer to remove dependencies on MergeTreeData and DataPart. [#63620](https://github.com/ClickHouse/ClickHouse/pull/63620) ([Alexander Gololobov](https://github.com/davenger)). @@ -1087,14 +1087,14 @@ doc_type: 'changelog' * Reduce the size of some slow tests. [#64387](https://github.com/ClickHouse/ClickHouse/pull/64387) [#64452](https://github.com/ClickHouse/ClickHouse/pull/64452) ([Raúl Marín](https://github.com/Algunenano)). * Replay ZooKeeper logs using keeper-bench. [#62481](https://github.com/ClickHouse/ClickHouse/pull/62481) ([Antonio Andelic](https://github.com/antonio2368)). -### ClickHouse release 24.5, 2024-05-30 {#a-id245a-clickhouse-release-245-2024-05-30} +### ClickHouse release 24.5, 2024-05-30 {#a-id245a-clickhouse-release-245-2024-05-30} -#### Backward Incompatible Change {#backward-incompatible-change-7} +#### Backward incompatible change {#backward-incompatible-change-7} * Renamed "inverted indexes" to "full-text indexes" which is a less technical / more user-friendly name. This also changes internal table metadata and breaks tables with existing (experimental) inverted indexes. Please make sure to drop such indexes before upgrade and re-create them after upgrade. [#62884](https://github.com/ClickHouse/ClickHouse/pull/62884) ([Robert Schulze](https://github.com/rschu1ze)). * Usage of functions `neighbor`, `runningAccumulate`, `runningDifferenceStartingWithFirstValue`, `runningDifference` deprecated (because it is error-prone). Proper window functions should be used instead. To enable them back, set `allow_deprecated_error_prone_window_functions = 1` or set `compatibility = '24.4'` or lower. [#63132](https://github.com/ClickHouse/ClickHouse/pull/63132) ([Nikita Taranov](https://github.com/nickitat)). * Queries from `system.columns` will work faster if there is a large number of columns, but many databases or tables are not granted for `SHOW TABLES`. Note that in previous versions, if you grant `SHOW COLUMNS` to individual columns without granting `SHOW TABLES` to the corresponding tables, the `system.columns` table will show these columns, but in a new version, it will skip the table entirely. Remove trace log messages "Access granted" and "Access denied" that slowed down queries. [#63439](https://github.com/ClickHouse/ClickHouse/pull/63439) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### New Feature {#new-feature-7} +#### New feature {#new-feature-7} * Adds the `Form` format to read/write a single record in the `application/x-www-form-urlencoded` format. [#60199](https://github.com/ClickHouse/ClickHouse/pull/60199) ([Shaun Struwig](https://github.com/Blargian)). * Added possibility to compress in CROSS JOIN. [#60459](https://github.com/ClickHouse/ClickHouse/pull/60459) ([p1rattttt](https://github.com/p1rattttt)). * Added possibility to do `CROSS JOIN` in temporary files if the size exceeds limits. [#63432](https://github.com/ClickHouse/ClickHouse/pull/63432) ([p1rattttt](https://github.com/p1rattttt)). @@ -1113,13 +1113,13 @@ doc_type: 'changelog' * Provide support for `azureBlobStorage` function in ClickHouse server to use Azure Workload identity to authenticate against Azure blob storage. If `use_workload_identity` parameter is set in config, [workload identity](https://github.com/Azure/azure-sdk-for-cpp/tree/main/sdk/identity/azure-identity#authenticate-azure-hosted-applications) is used for authentication. [#57881](https://github.com/ClickHouse/ClickHouse/pull/57881) ([Vinay Suryadevara](https://github.com/vinay92-ch)). * Add TTL information in the `system.parts_columns` table. [#63200](https://github.com/ClickHouse/ClickHouse/pull/63200) ([litlig](https://github.com/litlig)). -#### Experimental Features {#experimental-features-1} +#### Experimental features {#experimental-features-1} * Implement `Dynamic` data type that allows to store values of any type inside it without knowing all of them in advance. `Dynamic` type is available under a setting `allow_experimental_dynamic_type`. Reference: [#54864](https://github.com/ClickHouse/ClickHouse/issues/54864). [#63058](https://github.com/ClickHouse/ClickHouse/pull/63058) ([Kruglov Pavel](https://github.com/Avogar)). * Allowed to create `MaterializedMySQL` database without connection to MySQL. [#63397](https://github.com/ClickHouse/ClickHouse/pull/63397) ([Kirill](https://github.com/kirillgarbar)). * Automatically mark a replica of Replicated database as lost and start recovery if some DDL task fails more than `max_retries_before_automatic_recovery` (100 by default) times in a row with the same error. Also, fixed a bug that could cause skipping DDL entries when an exception is thrown during an early stage of entry execution. [#63549](https://github.com/ClickHouse/ClickHouse/pull/63549) ([Alexander Tokmakov](https://github.com/tavplubix)). * Account failed files in `s3queue_tracked_file_ttl_sec` and `s3queue_traked_files_limit` for `StorageS3Queue`. [#63638](https://github.com/ClickHouse/ClickHouse/pull/63638) ([Kseniia Sumarokova](https://github.com/kssenii)). -#### Performance Improvement {#performance-improvement-7} +#### Performance improvement {#performance-improvement-7} * Less contention in filesystem cache (part 4). Allow to keep filesystem cache not filled to the limit by doing additional eviction in the background (controlled by `keep_free_space_size(elements)_ratio`). This allows to release pressure from space reservation for queries (on `tryReserve` method). Also this is done in a lock free way as much as possible, e.g. should not block normal cache usage. [#61250](https://github.com/ClickHouse/ClickHouse/pull/61250) ([Kseniia Sumarokova](https://github.com/kssenii)). * Skip merging of newly created projection blocks during `INSERT`-s. [#59405](https://github.com/ClickHouse/ClickHouse/pull/59405) ([Nikita Taranov](https://github.com/nickitat)). * Process string functions `...UTF8` 'asciily' if input strings are all ascii chars. Inspired by https://github.com/apache/doris/pull/29799. Overall speed up by 1.07x~1.62x. Notice that peak memory usage had been decreased in some cases. [#61632](https://github.com/ClickHouse/ClickHouse/pull/61632) ([李扬](https://github.com/taiyang-li)). @@ -1166,7 +1166,7 @@ doc_type: 'changelog' * Add metrics, logs, and thread names around parts filtering with indices. [#64130](https://github.com/ClickHouse/ClickHouse/pull/64130) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Ignore `allow_suspicious_primary_key` on `ATTACH` and verify on `ALTER`. [#64202](https://github.com/ClickHouse/ClickHouse/pull/64202) ([Azat Khuzhin](https://github.com/azat)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-3} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-3} * ClickHouse is built with clang-18. A lot of new checks from clang-tidy-18 have been enabled. [#60469](https://github.com/ClickHouse/ClickHouse/pull/60469) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Experimentally support loongarch64 as a new platform for ClickHouse. [#63733](https://github.com/ClickHouse/ClickHouse/pull/63733) ([qiangxuhui](https://github.com/qiangxuhui)). * The Dockerfile is reviewed by the docker official library in https://github.com/docker-library/official-images/pull/15846. [#63400](https://github.com/ClickHouse/ClickHouse/pull/63400) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). @@ -1175,7 +1175,7 @@ doc_type: 'changelog' * Enable GRPC support for aarch64 linux while cross-compiling binary. [#64072](https://github.com/ClickHouse/ClickHouse/pull/64072) ([alesapin](https://github.com/alesapin)). * Fix unwind on SIGSEGV on aarch64 (due to small stack for signal) [#64058](https://github.com/ClickHouse/ClickHouse/pull/64058) ([Azat Khuzhin](https://github.com/azat)). -#### Bug Fix {#bug-fix-1} +#### Bug fix {#bug-fix-1} * Disabled `enable_vertical_final` setting by default. This feature should not be used because it has a bug: [#64543](https://github.com/ClickHouse/ClickHouse/issues/64543). [#64544](https://github.com/ClickHouse/ClickHouse/pull/64544) ([Alexander Tokmakov](https://github.com/tavplubix)). * Fix making backup when multiple shards are used [#57684](https://github.com/ClickHouse/ClickHouse/pull/57684) ([Vitaly Baranov](https://github.com/vitlibar)). * Fix passing projections/indexes/primary key from columns list from CREATE query into inner table of MV [#59183](https://github.com/ClickHouse/ClickHouse/pull/59183) ([Azat Khuzhin](https://github.com/azat)). @@ -1242,16 +1242,16 @@ doc_type: 'changelog' * Ignore `text_log` for Keeper [#64218](https://github.com/ClickHouse/ClickHouse/pull/64218) ([Antonio Andelic](https://github.com/antonio2368)). * Fix Logical error: Bad cast for Buffer table with prewhere. [#64388](https://github.com/ClickHouse/ClickHouse/pull/64388) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). -### ClickHouse release 24.4, 2024-04-30 {#a-id244a-clickhouse-release-244-2024-04-30} +### ClickHouse release 24.4, 2024-04-30 {#a-id244a-clickhouse-release-244-2024-04-30} -#### Upgrade Notes {#upgrade-notes} +#### Upgrade notes {#upgrade-notes} * `clickhouse-odbc-bridge` and `clickhouse-library-bridge` are now separate packages. This closes [#61677](https://github.com/ClickHouse/ClickHouse/issues/61677). [#62114](https://github.com/ClickHouse/ClickHouse/pull/62114) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Don't allow to set max_parallel_replicas (for the experimental parallel reading from replicas) to `0` as it doesn't make sense. Closes [#60140](https://github.com/ClickHouse/ClickHouse/issues/60140). [#61201](https://github.com/ClickHouse/ClickHouse/pull/61201) ([Kruglov Pavel](https://github.com/Avogar)). * Remove support for `INSERT WATCH` query (part of the deprecated `LIVE VIEW` feature). [#62382](https://github.com/ClickHouse/ClickHouse/pull/62382) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Removed the `optimize_monotonous_functions_in_order_by` setting. [#63004](https://github.com/ClickHouse/ClickHouse/pull/63004) ([Raúl Marín](https://github.com/Algunenano)). * Remove experimental tag from the `Replicated` database engine. Now it is in Beta stage. [#62937](https://github.com/ClickHouse/ClickHouse/pull/62937) ([Justin de Guzman](https://github.com/justindeguzman)). -#### New Feature {#new-feature-8} +#### New feature {#new-feature-8} * Support recursive CTEs. [#62074](https://github.com/ClickHouse/ClickHouse/pull/62074) ([Maksim Kita](https://github.com/kitaisreal)). * Support `QUALIFY` clause. Closes [#47819](https://github.com/ClickHouse/ClickHouse/issues/47819). [#62619](https://github.com/ClickHouse/ClickHouse/pull/62619) ([Maksim Kita](https://github.com/kitaisreal)). * Table engines are grantable now, and it won't affect existing users behavior. [#60117](https://github.com/ClickHouse/ClickHouse/pull/60117) ([jsc0218](https://github.com/jsc0218)). @@ -1265,7 +1265,7 @@ doc_type: 'changelog' * Added persistent virtual column `_block_offset` which stores original number of row in block that was assigned at insert. Persistence of column `_block_offset` can be enabled by the MergeTree setting `enable_block_offset_column`. Added virtual column`_part_data_version` which contains either min block number or mutation version of part. Persistent virtual column `_block_number` is not considered experimental anymore. [#60676](https://github.com/ClickHouse/ClickHouse/pull/60676) ([Anton Popov](https://github.com/CurtizJ)). * Add a setting `input_format_json_throw_on_bad_escape_sequence`, disabling it allows saving bad escape sequences in JSON input formats. [#61889](https://github.com/ClickHouse/ClickHouse/pull/61889) ([Kruglov Pavel](https://github.com/Avogar)). -#### Performance Improvement {#performance-improvement-8} +#### Performance improvement {#performance-improvement-8} * JOIN filter push down improvements using equivalent sets. [#61216](https://github.com/ClickHouse/ClickHouse/pull/61216) ([Maksim Kita](https://github.com/kitaisreal)). * Convert OUTER JOIN to INNER JOIN optimization if the filter after JOIN always filters default values. Optimization can be controlled with setting `query_plan_convert_outer_join_to_inner_join`, enabled by default. [#62907](https://github.com/ClickHouse/ClickHouse/pull/62907) ([Maksim Kita](https://github.com/kitaisreal)). * Improvement for AWS S3. Client has to send header 'Keep-Alive: timeout=X' to the server. If a client receives a response from the server with that header, client has to use the value from the server. Also for a client it is better not to use a connection which is nearly expired in order to avoid connection close race. [#62249](https://github.com/ClickHouse/ClickHouse/pull/62249) ([Sema Checherinda](https://github.com/CheSema)). @@ -1282,7 +1282,7 @@ doc_type: 'changelog' * Fix an issue where when a redundant `= 1` or `= 0` is added after a boolean expression involving the primary key, the primary index is not used. For example, both `SELECT * FROM WHERE IN () = 1` and `SELECT * FROM
WHERE NOT IN () = 0` will both perform a full table scan, when the primary index can be used. [#62142](https://github.com/ClickHouse/ClickHouse/pull/62142) ([josh-hildred](https://github.com/josh-hildred)). * Return stream of chunks from `system.remote_data_paths` instead of accumulating the whole result in one big chunk. This allows to consume less memory, show intermediate progress and cancel the query. [#62613](https://github.com/ClickHouse/ClickHouse/pull/62613) ([Alexander Gololobov](https://github.com/davenger)). -#### Experimental Feature {#experimental-feature-6} +#### Experimental feature {#experimental-feature-6} * Support parallel write buffer for Azure Blob Storage managed by setting `azure_allow_parallel_part_upload`. [#62534](https://github.com/ClickHouse/ClickHouse/pull/62534) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni)). * Userspace page cache works with static web storage (`disk(type = web)`) now. Use client setting `use_page_cache_for_disks_without_file_cache=1` to enable. [#61911](https://github.com/ClickHouse/ClickHouse/pull/61911) ([Michael Kolupaev](https://github.com/al13n321)). * Don't treat Bool and number variants as suspicious in the `Variant` type. [#61999](https://github.com/ClickHouse/ClickHouse/pull/61999) ([Kruglov Pavel](https://github.com/Avogar)). @@ -1326,7 +1326,7 @@ doc_type: 'changelog' * Reduce the verbosity of command line argument `--help` in `clickhouse client` and `clickhouse local`. The previous output is now generated by `--help --verbose`. [#62973](https://github.com/ClickHouse/ClickHouse/pull/62973) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). * `log_bin_use_v1_row_events` was removed in MySQL 8.3, and we adjust the experimental `MaterializedMySQL` engine for it [#60479](https://github.com/ClickHouse/ClickHouse/issues/60479). [#63101](https://github.com/ClickHouse/ClickHouse/pull/63101) ([Eugene Klimov](https://github.com/Slach)). Author: Nikolay Yankin. -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-4} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-4} * Vendor in Rust dependencies, so the Rust code (that we use for minor features for hype and lulz) can be built in a sane way, similarly to C++. [#62297](https://github.com/ClickHouse/ClickHouse/pull/62297) ([Raúl Marín](https://github.com/Algunenano)). * ClickHouse now uses OpenSSL 3.2 instead of BoringSSL. [#59870](https://github.com/ClickHouse/ClickHouse/pull/59870) ([Robert Schulze](https://github.com/rschu1ze)). Note that OpenSSL has generally worse engineering culture (such as non-zero number of sanitizer reports, that we had to patch, a complex build system with generated files, etc.) but has better compatibility. * Ignore DROP queries in stress test with 1/2 probability, use TRUNCATE instead of ignoring DROP in upgrade check for Memory/JOIN tables. [#61476](https://github.com/ClickHouse/ClickHouse/pull/61476) ([Kruglov Pavel](https://github.com/Avogar)). @@ -1335,7 +1335,7 @@ doc_type: 'changelog' * Add more tests from issues which are no longer relevant or fixed by analyzer. Closes: [#58985](https://github.com/ClickHouse/ClickHouse/issues/58985) Closes: [#59549](https://github.com/ClickHouse/ClickHouse/issues/59549) Closes: [#36963](https://github.com/ClickHouse/ClickHouse/issues/36963) Closes: [#39453](https://github.com/ClickHouse/ClickHouse/issues/39453) Closes: [#56521](https://github.com/ClickHouse/ClickHouse/issues/56521) Closes: [#47552](https://github.com/ClickHouse/ClickHouse/issues/47552) Closes: [#56503](https://github.com/ClickHouse/ClickHouse/issues/56503) Closes: [#59101](https://github.com/ClickHouse/ClickHouse/issues/59101) Closes: [#50271](https://github.com/ClickHouse/ClickHouse/issues/50271) Closes: [#54954](https://github.com/ClickHouse/ClickHouse/issues/54954) Closes: [#56466](https://github.com/ClickHouse/ClickHouse/issues/56466) Closes: [#11000](https://github.com/ClickHouse/ClickHouse/issues/11000) Closes: [#10894](https://github.com/ClickHouse/ClickHouse/issues/10894) Closes: https://github.com/ClickHouse/ClickHouse/issues/448 Closes: [#8030](https://github.com/ClickHouse/ClickHouse/issues/8030) Closes: [#32139](https://github.com/ClickHouse/ClickHouse/issues/32139) Closes: [#47288](https://github.com/ClickHouse/ClickHouse/issues/47288) Closes: [#50705](https://github.com/ClickHouse/ClickHouse/issues/50705) Closes: [#54511](https://github.com/ClickHouse/ClickHouse/issues/54511) Closes: [#55466](https://github.com/ClickHouse/ClickHouse/issues/55466) Closes: [#58500](https://github.com/ClickHouse/ClickHouse/issues/58500) Closes: [#39923](https://github.com/ClickHouse/ClickHouse/issues/39923) Closes: [#39855](https://github.com/ClickHouse/ClickHouse/issues/39855) Closes: [#4596](https://github.com/ClickHouse/ClickHouse/issues/4596) Closes: [#47422](https://github.com/ClickHouse/ClickHouse/issues/47422) Closes: [#33000](https://github.com/ClickHouse/ClickHouse/issues/33000) Closes: [#14739](https://github.com/ClickHouse/ClickHouse/issues/14739) Closes: [#44039](https://github.com/ClickHouse/ClickHouse/issues/44039) Closes: [#8547](https://github.com/ClickHouse/ClickHouse/issues/8547) Closes: [#22923](https://github.com/ClickHouse/ClickHouse/issues/22923) Closes: [#23865](https://github.com/ClickHouse/ClickHouse/issues/23865) Closes: [#29748](https://github.com/ClickHouse/ClickHouse/issues/29748) Closes: [#4222](https://github.com/ClickHouse/ClickHouse/issues/4222). [#62457](https://github.com/ClickHouse/ClickHouse/pull/62457) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Fixed build errors when OpenSSL is linked dynamically (note: this is generally unsupported and only required for IBM's s390x platforms). [#62888](https://github.com/ClickHouse/ClickHouse/pull/62888) ([Harry Lee](https://github.com/HarryLeeIBM)). -#### Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-6} +#### Bug fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-6} * Fix logical-error when undoing quorum insert transaction. [#61953](https://github.com/ClickHouse/ClickHouse/pull/61953) ([Han Fei](https://github.com/hanfei1991)). * Fix parser error when using COUNT(*) with FILTER clause [#61357](https://github.com/ClickHouse/ClickHouse/pull/61357) ([Duc Canh Le](https://github.com/canhld94)). * Fix logical error in `group_by_use_nulls` + grouping sets + analyzer + materialize/constant [#61567](https://github.com/ClickHouse/ClickHouse/pull/61567) ([Kruglov Pavel](https://github.com/Avogar)). @@ -1402,9 +1402,9 @@ doc_type: 'changelog' * Set server name for SSL handshake in MongoDB engine [#63122](https://github.com/ClickHouse/ClickHouse/pull/63122) ([Alexander Gololobov](https://github.com/davenger)). * Use user specified db instead of "config" for MongoDB wire protocol version check [#63126](https://github.com/ClickHouse/ClickHouse/pull/63126) ([Alexander Gololobov](https://github.com/davenger)). -### ClickHouse release 24.3 LTS, 2024-03-27 {#a-id243a-clickhouse-release-243-lts-2024-03-27} +### ClickHouse release 24.3 lts, 2024-03-27 {#a-id243a-clickhouse-release-243-lts-2024-03-27} -#### Upgrade Notes {#upgrade-notes-1} +#### Upgrade notes {#upgrade-notes-1} * The setting `allow_experimental_analyzer` is enabled by default and it switches the query analysis to a new implementation, which has better compatibility and feature completeness. The feature "analyzer" is considered beta instead of experimental. You can turn the old behavior by setting the `compatibility` to `24.2` or disabling the `allow_experimental_analyzer` setting. Watch the [video on YouTube](https://www.youtube.com/watch?v=zhrOYQpgvkk). * ClickHouse allows arbitrary binary data in the String data type, which is typically UTF-8. Parquet/ORC/Arrow Strings only support UTF-8. That's why you can choose which Arrow's data type to use for the ClickHouse String data type - String or Binary. This is controlled by the settings, `output_format_parquet_string_as_string`, `output_format_orc_string_as_string`, `output_format_arrow_string_as_string`. While Binary would be more correct and compatible, using String by default will correspond to user expectations in most cases. Parquet/ORC/Arrow supports many compression methods, including lz4 and zstd. ClickHouse supports each and every compression method. Some inferior tools lack support for the faster `lz4` compression method, that's why we set `zstd` by default. This is controlled by the settings `output_format_parquet_compression_method`, `output_format_orc_compression_method`, and `output_format_arrow_compression_method`. We changed the default to `zstd` for Parquet and ORC, but not Arrow (it is emphasized for low-level usages). [#61817](https://github.com/ClickHouse/ClickHouse/pull/61817) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * In the new ClickHouse version, the functions `geoDistance`, `greatCircleDistance`, and `greatCircleAngle` will use 64-bit double precision floating point data type for internal calculations and return type if all the arguments are Float64. This closes [#58476](https://github.com/ClickHouse/ClickHouse/issues/58476). In previous versions, the function always used Float32. You can switch to the old behavior by setting `geo_distance_returns_float64_on_float64_arguments` to `false` or setting `compatibility` to `24.2` or earlier. [#61848](https://github.com/ClickHouse/ClickHouse/pull/61848) ([Alexey Milovidov](https://github.com/alexey-milovidov)). Co-authored with [Geet Patel](https://github.com/geetptl). @@ -1416,7 +1416,7 @@ doc_type: 'changelog' * Forbid `SimpleAggregateFunction` in `ORDER BY` of `MergeTree` tables (like `AggregateFunction` is forbidden, but they are forbidden because they are not comparable) by default (use `allow_suspicious_primary_key` to allow them). [#61399](https://github.com/ClickHouse/ClickHouse/pull/61399) ([Azat Khuzhin](https://github.com/azat)). * The `Ordinary` database engine is deprecated. You will receive a warning in clickhouse-client if your server is using it. This closes [#52229](https://github.com/ClickHouse/ClickHouse/issues/52229). [#56942](https://github.com/ClickHouse/ClickHouse/pull/56942) ([shabroo](https://github.com/shabroo)). -#### New Feature {#new-feature-9} +#### New feature {#new-feature-9} * Support reading and writing backups as `tar` (in addition to `zip`). [#59535](https://github.com/ClickHouse/ClickHouse/pull/59535) ([josh-hildred](https://github.com/josh-hildred)). * Implemented support for S3 Express buckets. [#59965](https://github.com/ClickHouse/ClickHouse/pull/59965) ([Nikita Taranov](https://github.com/nickitat)). * Allow to attach parts from a different disk (using copy instead of hard link). [#60112](https://github.com/ClickHouse/ClickHouse/pull/60112) ([Unalian](https://github.com/Unalian)). @@ -1430,7 +1430,7 @@ doc_type: 'changelog' * Added function `toMillisecond` which returns the millisecond component for values of type`DateTime` or `DateTime64`. [#60281](https://github.com/ClickHouse/ClickHouse/pull/60281) ([Shaun Struwig](https://github.com/Blargian)). * Allow configuring HTTP redirect handlers for clickhouse-server. For example, you can make `/` redirect to the Play UI. [#60390](https://github.com/ClickHouse/ClickHouse/pull/60390) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### Performance Improvement {#performance-improvement-9} +#### Performance improvement {#performance-improvement-9} * Optimized function `dotProduct` to omit unnecessary and expensive memory copies. [#60928](https://github.com/ClickHouse/ClickHouse/pull/60928) ([Robert Schulze](https://github.com/rschu1ze)). * 30x faster printing for 256-bit integers. [#61100](https://github.com/ClickHouse/ClickHouse/pull/61100) ([Raúl Marín](https://github.com/Algunenano)). * If the table's primary key contains mostly useless columns, don't keep them in memory. This is controlled by a new setting `primary_key_ratio_of_unique_prefix_values_to_skip_suffix_columns` with the value `0.9` by default, which means: for a composite primary key, if a column changes its value for at least 0.9 of all the times, the next columns after it will be not loaded. [#60255](https://github.com/ClickHouse/ClickHouse/pull/60255) ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -1449,7 +1449,7 @@ doc_type: 'changelog' * Add a new analyzer pass to optimize the IN operator for a single value. [#61564](https://github.com/ClickHouse/ClickHouse/pull/61564) ([LiuNeng](https://github.com/liuneng1994)). * DNSResolver shuffles set of resolved IPs which is needed to uniformly utilize multiple endpoints of AWS S3. [#60965](https://github.com/ClickHouse/ClickHouse/pull/60965) ([Sema Checherinda](https://github.com/CheSema)). -#### Experimental Feature {#experimental-feature-7} +#### Experimental feature {#experimental-feature-7} * Support parallel reading for Azure blob storage. This improves the performance of the experimental Azure object storage. [#61503](https://github.com/ClickHouse/ClickHouse/pull/61503) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni)). * Add asynchronous WriteBuffer for Azure blob storage similar to S3. This improves the performance of the experimental Azure object storage. [#59929](https://github.com/ClickHouse/ClickHouse/pull/59929) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni)). * Use managed identity for backups IO when using Azure Blob Storage. Add a setting to prevent ClickHouse from attempting to create a non-existent container, which requires permissions at the storage account level. [#61785](https://github.com/ClickHouse/ClickHouse/pull/61785) ([Daniel Pozo Escalona](https://github.com/danipozo)). @@ -1507,13 +1507,13 @@ doc_type: 'changelog' * Added `current_user` function as a compatibility alias for MySQL. [#61770](https://github.com/ClickHouse/ClickHouse/pull/61770) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). * Fix inconsistent floating point aggregate function states in mixed x86-64 / ARM clusters [#60610](https://github.com/ClickHouse/ClickHouse/pull/60610) ([Harry Lee](https://github.com/HarryLeeIBM)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-5} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-5} * The real-time query profiler now works on AArch64. In previous versions, it worked only when a program didn't spend time inside a syscall. [#60807](https://github.com/ClickHouse/ClickHouse/pull/60807) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * ClickHouse version has been added to docker labels. Closes [#54224](https://github.com/ClickHouse/ClickHouse/issues/54224). [#60949](https://github.com/ClickHouse/ClickHouse/pull/60949) ([Nikolay Monkov](https://github.com/nikmonkov)). * Upgrade `prqlc` to 0.11.3. [#60616](https://github.com/ClickHouse/ClickHouse/pull/60616) ([Maximilian Roos](https://github.com/max-sixty)). * Add generic query text fuzzer in `clickhouse-local`. [#61508](https://github.com/ClickHouse/ClickHouse/pull/61508) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-7} +#### Bug fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-7} * Fix finished_mutations_to_keep=0 for MergeTree (as docs says 0 is to keep everything) [#60031](https://github.com/ClickHouse/ClickHouse/pull/60031) ([Azat Khuzhin](https://github.com/azat)). * Something was wrong with the FINAL optimization, here is how the author describes it: "PartsSplitter invalid ranges for the same part". [#60041](https://github.com/ClickHouse/ClickHouse/pull/60041) ([Maksim Kita](https://github.com/kitaisreal)). * Something was wrong with Apache Hive, which is experimental and not supported. [#60262](https://github.com/ClickHouse/ClickHouse/pull/60262) ([shanfengp](https://github.com/Aed-p)). @@ -1574,16 +1574,16 @@ doc_type: 'changelog' * Fix a sanitizer report in `multiSearchAllPositionsCaseInsensitiveUTF8` for incorrect UTF-8 [#61749](https://github.com/ClickHouse/ClickHouse/pull/61749) ([pufit](https://github.com/pufit)). * Fix an observation that the RANGE frame is not supported for Nullable columns. [#61766](https://github.com/ClickHouse/ClickHouse/pull/61766) ([YuanLiu](https://github.com/ditgittube)). -### ClickHouse release 24.2, 2024-02-29 {#a-id242a-clickhouse-release-242-2024-02-29} +### ClickHouse release 24.2, 2024-02-29 {#a-id242a-clickhouse-release-242-2024-02-29} -#### Backward Incompatible Change {#backward-incompatible-change-8} +#### Backward incompatible change {#backward-incompatible-change-8} * Validate suspicious/experimental types in nested types. Previously we didn't validate such types (except JSON) in nested types like Array/Tuple/Map. [#59385](https://github.com/ClickHouse/ClickHouse/pull/59385) ([Kruglov Pavel](https://github.com/Avogar)). * Add sanity check for number of threads and block sizes. [#60138](https://github.com/ClickHouse/ClickHouse/pull/60138) ([Raúl Marín](https://github.com/Algunenano)). * Don't infer floats in exponential notation by default. Add a setting `input_format_try_infer_exponent_floats` that will restore previous behaviour (disabled by default). Closes [#59476](https://github.com/ClickHouse/ClickHouse/issues/59476). [#59500](https://github.com/ClickHouse/ClickHouse/pull/59500) ([Kruglov Pavel](https://github.com/Avogar)). * Allow alter operations to be surrounded by parenthesis. The emission of parentheses can be controlled by the `format_alter_operations_with_parentheses` config. By default, in formatted queries the parentheses are emitted as we store the formatted alter operations in some places as metadata (e.g.: mutations). The new syntax clarifies some of the queries where alter operations end in a list. E.g.: `ALTER TABLE x MODIFY TTL date GROUP BY a, b, DROP COLUMN c` cannot be parsed properly with the old syntax. In the new syntax the query `ALTER TABLE x (MODIFY TTL date GROUP BY a, b), (DROP COLUMN c)` is obvious. Older versions are not able to read the new syntax, therefore using the new syntax might cause issues if newer and older version of ClickHouse are mixed in a single cluster. [#59532](https://github.com/ClickHouse/ClickHouse/pull/59532) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). * Fix for the materialized view security issue, which allowed a user to insert into a table without required grants for that. Fix validates that the user has permission to insert not only into a materialized view but also into all underlying tables. This means that some queries, which worked before, now can fail with `Not enough privileges`. To address this problem, the release introduces a new feature of SQL security for views https://clickhouse.com/docs/sql-reference/statements/create/view#sql_security. [#54901](https://github.com/ClickHouse/ClickHouse/pull/54901) [#60439](https://github.com/ClickHouse/ClickHouse/pull/60439) ([pufit](https://github.com/pufit)). -#### New Feature {#new-feature-10} +#### New feature {#new-feature-10} * Added new syntax which allows to specify definer user in view/materialized view. This allows to execute selects/inserts from views without explicit grants for underlying tables. So, a View will encapsulate the grants. [#54901](https://github.com/ClickHouse/ClickHouse/pull/54901) [#60439](https://github.com/ClickHouse/ClickHouse/pull/60439) ([pufit](https://github.com/pufit)). * Try to detect file format automatically during schema inference if it's unknown in `file/s3/hdfs/url/azureBlobStorage` engines. Closes [#50576](https://github.com/ClickHouse/ClickHouse/issues/50576). [#59092](https://github.com/ClickHouse/ClickHouse/pull/59092) ([Kruglov Pavel](https://github.com/Avogar)). * Implement auto-adjustment for asynchronous insert timeouts. The following settings are introduced: async_insert_poll_timeout_ms, async_insert_use_adaptive_busy_timeout, async_insert_busy_timeout_min_ms, async_insert_busy_timeout_max_ms, async_insert_busy_timeout_increase_rate, async_insert_busy_timeout_decrease_rate. [#58486](https://github.com/ClickHouse/ClickHouse/pull/58486) ([Julia Kartseva](https://github.com/jkartseva)). @@ -1602,7 +1602,7 @@ doc_type: 'changelog' * Support specifying a set of permitted users for specific S3 settings in config using `user` key. [#60144](https://github.com/ClickHouse/ClickHouse/pull/60144) ([Antonio Andelic](https://github.com/antonio2368)). * Added table function `mergeTreeIndex`. It represents the contents of index and marks files of `MergeTree` tables. It can be used for introspection. Syntax: `mergeTreeIndex(database, table, [with_marks = true])` where `database.table` is an existing table with `MergeTree` engine. [#58140](https://github.com/ClickHouse/ClickHouse/pull/58140) ([Anton Popov](https://github.com/CurtizJ)). -#### Experimental Feature {#experimental-feature-8} +#### Experimental feature {#experimental-feature-8} * Added function `seriesOutliersDetectTukey` to detect outliers in series data using Tukey's fences algorithm. [#58632](https://github.com/ClickHouse/ClickHouse/pull/58632) ([Bhavna Jindal](https://github.com/bhavnajindal)). Keep in mind that the behavior will be changed in the next patch release. * Add function `variantType` that returns Enum with variant type name for each row. [#59398](https://github.com/ClickHouse/ClickHouse/pull/59398) ([Kruglov Pavel](https://github.com/Avogar)). * Support `LEFT JOIN`, `ALL INNER JOIN`, and simple subqueries for parallel replicas (only with analyzer). New setting `parallel_replicas_prefer_local_join` chooses local `JOIN` execution (by default) vs `GLOBAL JOIN`. All tables should exist on every replica from `cluster_for_parallel_replicas`. New settings `min_external_table_block_size_rows` and `min_external_table_block_size_bytes` are used to squash small blocks that are sent for temporary tables (only with analyzer). [#58916](https://github.com/ClickHouse/ClickHouse/pull/58916) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -1610,7 +1610,7 @@ doc_type: 'changelog' * Implement comparison operator for `Variant` values and proper Field inserting into `Variant` column. Don't allow creating `Variant` type with similar variant types by default (allow uder a setting `allow_suspicious_variant_types`) Closes [#59996](https://github.com/ClickHouse/ClickHouse/issues/59996). Closes [#59850](https://github.com/ClickHouse/ClickHouse/issues/59850). [#60198](https://github.com/ClickHouse/ClickHouse/pull/60198) ([Kruglov Pavel](https://github.com/Avogar)). * Disable parallel replicas JOIN with CTE (not analyzer) [#59239](https://github.com/ClickHouse/ClickHouse/pull/59239) ([Raúl Marín](https://github.com/Algunenano)). -#### Performance Improvement {#performance-improvement-10} +#### Performance improvement {#performance-improvement-10} * Primary key will use less amount of memory. [#60049](https://github.com/ClickHouse/ClickHouse/pull/60049) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Improve memory usage for primary key and some other operations. [#60050](https://github.com/ClickHouse/ClickHouse/pull/60050) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * The tables' primary keys will be loaded in memory lazily on first access. This is controlled by the new MergeTree setting `primary_key_lazy_load`, which is on by default. This provides several advantages: - it will not be loaded for tables that are not used; - if there is not enough memory, an exception will be thrown on first use instead of at server startup. This provides several disadvantages: - the latency of loading the primary key will be paid on the first query rather than before accepting connections; this theoretically may introduce a thundering-herd problem. This closes [#11188](https://github.com/ClickHouse/ClickHouse/issues/11188). [#60093](https://github.com/ClickHouse/ClickHouse/pull/60093) ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -1669,7 +1669,7 @@ doc_type: 'changelog' * Connections through the MySQL port now automatically run with setting `prefer_column_name_to_alias = 1` to support QuickSight out-of-the-box. Also, settings `mysql_map_string_to_text_in_show_columns` and `mysql_map_fixed_string_to_text_in_show_columns` are now enabled by default, affecting also only MySQL connections. This increases compatibility with more BI tools. [#60365](https://github.com/ClickHouse/ClickHouse/pull/60365) ([Robert Schulze](https://github.com/rschu1ze)). * Fix a race condition in JavaScript code leading to duplicate charts on top of each other. [#60392](https://github.com/ClickHouse/ClickHouse/pull/60392) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-6} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-6} * Added builds and tests with coverage collection with introspection. Continuation of [#56102](https://github.com/ClickHouse/ClickHouse/issues/56102). [#58792](https://github.com/ClickHouse/ClickHouse/pull/58792) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Update the Rust toolchain in `corrosion-cmake` when the CMake cross-compilation toolchain variable is set. [#59309](https://github.com/ClickHouse/ClickHouse/pull/59309) ([Aris Tritas](https://github.com/aris-aiven)). * Add some fuzzing to ASTLiterals. [#59383](https://github.com/ClickHouse/ClickHouse/pull/59383) ([Raúl Marín](https://github.com/Algunenano)). @@ -1678,7 +1678,7 @@ doc_type: 'changelog' * Query fuzzer will fuzz SETTINGS inside queries. [#60087](https://github.com/ClickHouse/ClickHouse/pull/60087) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Add support for building ClickHouse with clang-19 (master). [#60448](https://github.com/ClickHouse/ClickHouse/pull/60448) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-8} +#### Bug fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-8} * Fix a "Non-ready set" error in TTL WHERE. [#57430](https://github.com/ClickHouse/ClickHouse/pull/57430) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix a bug in the `quantilesGK` function [#58216](https://github.com/ClickHouse/ClickHouse/pull/58216) ([李扬](https://github.com/taiyang-li)). * Fix a wrong behavior with `intDiv` for Decimal arguments [#59243](https://github.com/ClickHouse/ClickHouse/pull/59243) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). @@ -1729,9 +1729,9 @@ doc_type: 'changelog' * Fix OptimizeDateOrDateTimeConverterWithPreimageVisitor with null arguments [#60453](https://github.com/ClickHouse/ClickHouse/pull/60453) ([Raúl Marín](https://github.com/Algunenano)). * Fixed a minor bug that prevented distributed table queries sent from either KQL or PRQL dialect clients to be executed on replicas. [#59674](https://github.com/ClickHouse/ClickHouse/issues/59674). [#60470](https://github.com/ClickHouse/ClickHouse/pull/60470) ([Alexey Milovidov](https://github.com/alexey-milovidov)) [#59674](https://github.com/ClickHouse/ClickHouse/pull/59674) ([Austin Kothig](https://github.com/kothiga)). -### ClickHouse release 24.1, 2024-01-30 {#a-id241a-clickhouse-release-241-2024-01-30} +### ClickHouse release 24.1, 2024-01-30 {#a-id241a-clickhouse-release-241-2024-01-30} -#### Backward Incompatible Change {#backward-incompatible-change-9} +#### Backward incompatible change {#backward-incompatible-change-9} * The setting `print_pretty_type_names` is turned on by default. You can turn it off to keep the old behavior or `SET compatibility = '23.12'`. [#57726](https://github.com/ClickHouse/ClickHouse/pull/57726) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * The MergeTree setting `clean_deleted_rows` is deprecated, it has no effect anymore. The `CLEANUP` keyword for `OPTIMIZE` is not allowed by default (unless `allow_experimental_replacing_merge_with_cleanup` is enabled). [#58316](https://github.com/ClickHouse/ClickHouse/pull/58316) ([Alexander Tokmakov](https://github.com/tavplubix)). * The function `reverseDNSQuery` is no longer available. This closes [#58368](https://github.com/ClickHouse/ClickHouse/issues/58368). [#58369](https://github.com/ClickHouse/ClickHouse/pull/58369) ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -1741,7 +1741,7 @@ doc_type: 'changelog' * `Kusto` dialect is disabled until these two bugs will be fixed: [#59037](https://github.com/ClickHouse/ClickHouse/issues/59037) and [#59036](https://github.com/ClickHouse/ClickHouse/issues/59036). [#59305](https://github.com/ClickHouse/ClickHouse/pull/59305) ([Alexey Milovidov](https://github.com/alexey-milovidov)). Any attempt to use `Kusto` will result in exception. * More efficient implementation of the `FINAL` modifier no longer guarantees preserving the order even if `max_threads = 1`. If you counted on the previous behavior, set `enable_vertical_final` to 0 or `compatibility` to `23.12`. -#### New Feature {#new-feature-11} +#### New feature {#new-feature-11} * Implement Variant data type that represents a union of other data types. Type `Variant(T1, T2, ..., TN)` means that each row of this type has a value of either type `T1` or `T2` or ... or `TN` or none of them (`NULL` value). Variant type is available under a setting `allow_experimental_variant_type`. Reference: [#54864](https://github.com/ClickHouse/ClickHouse/issues/54864). [#58047](https://github.com/ClickHouse/ClickHouse/pull/58047) ([Kruglov Pavel](https://github.com/Avogar)). * Certain settings (currently `min_compress_block_size` and `max_compress_block_size`) can now be specified at column-level where they take precedence over the corresponding table-level setting. Example: `CREATE TABLE tab (col String SETTINGS (min_compress_block_size = 81920, max_compress_block_size = 163840)) ENGINE = MergeTree ORDER BY tuple();`. [#55201](https://github.com/ClickHouse/ClickHouse/pull/55201) ([Duc Canh Le](https://github.com/canhld94)). * Add `quantileDD` aggregate function as well as the corresponding `quantilesDD` and `medianDD`. It is based on the DDSketch https://www.vldb.org/pvldb/vol12/p2195-masson.pdf. ### Documentation entry for user-facing changes. [#56342](https://github.com/ClickHouse/ClickHouse/pull/56342) ([Srikanth Chekuri](https://github.com/srikanthccv)). @@ -1768,7 +1768,7 @@ doc_type: 'changelog' * Added setting `update_insert_deduplication_token_in_dependent_materialized_views`. This setting allows to update insert deduplication token with table identifier during insert in dependent materialized views. Closes [#59165](https://github.com/ClickHouse/ClickHouse/issues/59165). [#59238](https://github.com/ClickHouse/ClickHouse/pull/59238) ([Maksim Kita](https://github.com/kitaisreal)). * Added statement `SYSTEM RELOAD ASYNCHRONOUS METRICS` which updates the asynchronous metrics. Mostly useful for testing and development. [#53710](https://github.com/ClickHouse/ClickHouse/pull/53710) ([Robert Schulze](https://github.com/rschu1ze)). -#### Performance Improvement {#performance-improvement-11} +#### Performance improvement {#performance-improvement-11} * Coordination for parallel replicas is rewritten for better parallelism and cache locality. It has been tested for linear scalability on hundreds of replicas. It also got support for reading in order. [#57968](https://github.com/ClickHouse/ClickHouse/pull/57968) ([Nikita Taranov](https://github.com/nickitat)). * Replace HTTP outgoing buffering based with the native ClickHouse buffers. Add bytes counting metrics for interfaces. [#56064](https://github.com/ClickHouse/ClickHouse/pull/56064) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). * Large aggregation states of `uniqExact` will be merged in parallel in distrubuted queries. [#59009](https://github.com/ClickHouse/ClickHouse/pull/59009) ([Nikita Taranov](https://github.com/nickitat)). @@ -1839,7 +1839,7 @@ doc_type: 'changelog' * Support the `FORMAT` clause in `BACKUP` and `RESTORE` queries. [#59338](https://github.com/ClickHouse/ClickHouse/pull/59338) ([Vitaly Baranov](https://github.com/vitlibar)). * Function `concatWithSeparator` now supports arbitrary argument types (instead of only `String` and `FixedString` arguments). For example, `SELECT concatWithSeparator('.', 'number', 1)` now returns `number.1`. [#59341](https://github.com/ClickHouse/ClickHouse/pull/59341) ([Robert Schulze](https://github.com/rschu1ze)). -#### Build/Testing/Packaging Improvement {#buildtestingpackaging-improvement-7} +#### Build/Testing/Packaging improvement {#buildtestingpackaging-improvement-7} * Improve aliases for clickhouse binary (now `ch`/`clickhouse` is `clickhouse-local` or `clickhouse` depends on the arguments) and add bash completion for new aliases. [#58344](https://github.com/ClickHouse/ClickHouse/pull/58344) ([Azat Khuzhin](https://github.com/azat)). * Add settings changes check to CI to check that all settings changes are reflected in settings changes history. [#58555](https://github.com/ClickHouse/ClickHouse/pull/58555) ([Kruglov Pavel](https://github.com/Avogar)). * Use tables directly attached from S3 in stateful tests. [#58791](https://github.com/ClickHouse/ClickHouse/pull/58791) ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -1847,7 +1847,7 @@ doc_type: 'changelog' * Enable Rust on macOS with Aarch64 (this will add fuzzy search in client with skim and the PRQL language, though I don't think that are people who host ClickHouse on darwin, so it is mostly for fuzzy search in client I would say). [#59272](https://github.com/ClickHouse/ClickHouse/pull/59272) ([Azat Khuzhin](https://github.com/azat)). * Fix aggregation issue in mixed x86_64 and ARM clusters [#59132](https://github.com/ClickHouse/ClickHouse/pull/59132) ([Harry Lee](https://github.com/HarryLeeIBM)). -#### Bug Fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-9} +#### Bug fix (user-visible misbehavior in an official stable release) {#bug-fix-user-visible-misbehavior-in-an-official-stable-release-9} * Add join keys conversion for nested LowCardinality [#51550](https://github.com/ClickHouse/ClickHouse/pull/51550) ([vdimir](https://github.com/vdimir)). * Flatten only true Nested type if flatten_nested=1, not all Array(Tuple) [#56132](https://github.com/ClickHouse/ClickHouse/pull/56132) ([Kruglov Pavel](https://github.com/Avogar)). diff --git a/docs/whats-new/changelog/cloud.md b/docs/whats-new/changelog/cloud.md index bafbe8de4c7..2c2698f569f 100644 --- a/docs/whats-new/changelog/cloud.md +++ b/docs/whats-new/changelog/cloud.md @@ -8,7 +8,7 @@ doc_type: 'changelog' keywords: ['ClickHouse Cloud', 'cloud changelog', 'cloud updates', 'cloud features', 'cloud releases'] --- -# Cloud Changelog +# Cloud changelog import CloudChangelog from '@site/docs/cloud/reference/01_changelog/01_changelog.md'; diff --git a/docs/whats-new/changelog/index.md b/docs/whats-new/changelog/index.md index 56577f45157..9cf427cc25e 100644 --- a/docs/whats-new/changelog/index.md +++ b/docs/whats-new/changelog/index.md @@ -8,7 +8,7 @@ title: 'Changelog 2025' doc_type: 'changelog' --- -### Table of Contents +### Table of contents **[ClickHouse release v25.11, 2025-11-27](#2511)**
**[ClickHouse release v25.10, 2025-10-30](#2510)**
**[ClickHouse release v25.9, 2025-09-25](#259)**
@@ -32,7 +32,7 @@ doc_type: 'changelog' ### ClickHouse release 25.11, 2025-11-27 {#2511} -#### Backward Incompatible Change +#### Backward incompatible change * Remove deprecated `Object` type. [#85718](https://github.com/ClickHouse/ClickHouse/pull/85718) ([Pavel Kruglov](https://github.com/Avogar)). * Remove the obsolete `LIVE VIEW` feature. If you use `LIVE VIEW`, upgrading to the new version will not be possible. [#88706](https://github.com/ClickHouse/ClickHouse/pull/88706) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * In previous versions, the `Geometry` type was aliased to `String`, but now it is a full-featured type. [#83344](https://github.com/ClickHouse/ClickHouse/pull/83344) ([scanhex12](https://github.com/scanhex12)). @@ -44,7 +44,7 @@ doc_type: 'changelog' * Parquet no-timezone timestamps (isAdjustedToUTC=false) are now read as DateTime64(..., 'UTC') instead of DateTime64(...). This is less incorrect as converting such UTC timestamp to string produces a representation of the correct local time. Use `input_format_parquet_local_time_as_utc = 0` to get the old behavior. Resolves [#87469](https://github.com/ClickHouse/ClickHouse/issues/87469). [#87872](https://github.com/ClickHouse/ClickHouse/pull/87872) ([Michael Kolupaev](https://github.com/al13n321)). * A small improvement for `T64` codec: it no longer accepts data types unaligned with the compression element size, which otherwise triggers a bug. Resolves [#89282](https://github.com/ClickHouse/ClickHouse/issues/89282). [#89432](https://github.com/ClickHouse/ClickHouse/pull/89432) ([yanglongwei](https://github.com/ylw510)). -#### New Feature +#### New feature * Introduce `Geometry` type. Support reading `WKB` and `WKT` formats for it. In previous versions, the `Geometry` type was aliased to `String`, but now it is a full-featured type. [#83344](https://github.com/ClickHouse/ClickHouse/pull/83344) ([scanhex12](https://github.com/scanhex12)). * Added new SQL statement `EXECUTE AS` to support user impersonation. Resolves [#39048](https://github.com/ClickHouse/ClickHouse/issues/39048). [#70775](https://github.com/ClickHouse/ClickHouse/pull/70775) ([Shankar](https://github.com/shiyer7474)). * Add `naiveBayesClassifier` function to classify text using Naive Bayes based on ngrams. [#88677](https://github.com/ClickHouse/ClickHouse/pull/88677) ([Nihal Z. Miaji](https://github.com/nihalzp)). @@ -68,13 +68,13 @@ doc_type: 'changelog' * Add `h3PolygonToCells` function which can fill geometry with h3 hexagons. Resolves [#33991](https://github.com/ClickHouse/ClickHouse/issues/33991). [#66262](https://github.com/ClickHouse/ClickHouse/pull/66262) ([Zacharias Knudsen](https://github.com/zachasme)). * Add new virtual column `_tags` (`Map(String, String)`) with all the tags associated with the blob in S3 (Note, if blob does not have any tags no extra request will be done). Resolves [#72945](https://github.com/ClickHouse/ClickHouse/issues/72945). [#77773](https://github.com/ClickHouse/ClickHouse/pull/77773) ([Zicong Qu](https://github.com/zicongleoqu)). -#### Experimental Feature +#### Experimental feature * Support TLS certificates retrieval from ACME providers, [RFC 8555](https://datatracker.ietf.org/doc/html/rfc8555), such as Let's Encrypt. This allows the autoconfiguration of TLS on distributed clusters. [#66315](https://github.com/ClickHouse/ClickHouse/pull/66315) ([Konstantin Bogdanov](https://github.com/thevar1able)). * Support part of the Prometheus HTTP Query API. To enable it, add a rule with type `query_api` in the `` section in the configuration file. Supported handlers are `/api/v1/query_range` and `/api/v1/query`. [#86132](https://github.com/ClickHouse/ClickHouse/pull/86132) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Full-text search is now in beta (from a previously experimental stage). [#88928](https://github.com/ClickHouse/ClickHouse/pull/88928) ([Robert Schulze](https://github.com/rschu1ze)). * Move `Alias` to experimental, can be enabled with `allow_experimental_alias_table_engine = 1`. [#89712](https://github.com/ClickHouse/ClickHouse/pull/89712) ([Kai Zhu](https://github.com/nauu)). -#### Performance Improvement +#### Performance improvement * Parquet reader v3 is enabled by default. [#88827](https://github.com/ClickHouse/ClickHouse/pull/88827) ([Michael Kolupaev](https://github.com/al13n321)). * Distributed execution: better split tasks by row groups IDs, not by files. [#87508](https://github.com/ClickHouse/ClickHouse/pull/87508) ([scanhex12](https://github.com/scanhex12)). * `RIGHT` and `FULL` JOINs now use ConcurrentHashJoin; that means that these types of join are now running with a higher degree of parallelism. Improves various cases for RIGHT and FULL JOINs up to two times. Resolves [#78027](https://github.com/ClickHouse/ClickHouse/issues/78027). [#78462](https://github.com/ClickHouse/ClickHouse/pull/78462) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). @@ -169,7 +169,7 @@ doc_type: 'changelog' * Fix incorrect `rows_before_limit_at_least` when there are multiple remote shards involved in a distributed merging aggregation, or there are subqueries INs. This fixes [#63280](https://github.com/ClickHouse/ClickHouse/issues/63280). [#63511](https://github.com/ClickHouse/ClickHouse/pull/63511) ([Amos Bird](https://github.com/amosbird)). * Fixed `0 rows in set` appearance after `INSERT INTO ... SELECT` query. Closes [#47800](https://github.com/ClickHouse/ClickHouse/issues/47800). [#79462](https://github.com/ClickHouse/ClickHouse/pull/79462) ([Engel Danila](https://github.com/aaaengel)). -#### Bug Fix (user-visible misbehavior in an official stable release) +#### Bug fix (user-visible misbehavior in an official stable release) * Fix `multiIf` with constant arguments and short-circuit evaluation. Closes [#72714](https://github.com/ClickHouse/ClickHouse/issues/72714). [#84546](https://github.com/ClickHouse/ClickHouse/pull/84546) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). * Fix logical error when selecting from a table with a subquery constraint. Resolves [#84190](https://github.com/ClickHouse/ClickHouse/issues/84190). [#85575](https://github.com/ClickHouse/ClickHouse/pull/85575) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). * Fixed a bug for special queries using a URI with question marks. [#85663](https://github.com/ClickHouse/ClickHouse/pull/85663) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). @@ -268,7 +268,7 @@ doc_type: 'changelog' * Backported in [#90601](https://github.com/ClickHouse/ClickHouse/issues/90601): Fix logical error caused by a rare case in `equals` function. Closes [#88142](https://github.com/ClickHouse/ClickHouse/issues/88142). [#90557](https://github.com/ClickHouse/ClickHouse/pull/90557) ([Nihal Z. Miaji](https://github.com/nihalzp)). * Fix `CoalescingMergeTree` for `Tuple` types. [#88828](https://github.com/ClickHouse/ClickHouse/pull/88828) ([scanhex12](https://github.com/scanhex12)). -#### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging improvement * Fix Connection refused error for running ClickHouse in Docker with initdb SQL scripts and overridden TCP port. [#88042](https://github.com/ClickHouse/ClickHouse/pull/88042) ([Grigorii](https://github.com/GSokol)). * Experimentally support e2k as a new platform for ClickHouse. [#90159](https://github.com/ClickHouse/ClickHouse/pull/90159) ([Ramil Sattarov](https://github.com/r-a-sattarov)). * Remove the remaining use of `FindPackage` from CMake. Builds should not depend on the system packages. [#89380](https://github.com/ClickHouse/ClickHouse/pull/89380) ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -278,7 +278,7 @@ doc_type: 'changelog' ### ClickHouse release 25.10, 2025-10-31 {#2510} -#### Backward Incompatible Change +#### Backward incompatible change * Changed default `schema_inference_make_columns_nullable` setting to respect column `Nullable`-ness information from Parquet/ORC/Arrow metadata, instead of making everything Nullable. No change for text formats. [#71499](https://github.com/ClickHouse/ClickHouse/pull/71499) ([Michael Kolupaev](https://github.com/al13n321)). * The query result cache will ignore the `log_comment` setting, so that changing only the `log_comment` on a query no longer forces a cache miss. There is a small chance users intentionally segmented their cache by varying `log_comment`. This change alters that behavior and is therefore backward incompatible. Please use setting `query_cache_tag` for this purpose. [#79878](https://github.com/ClickHouse/ClickHouse/pull/79878) ([filimonov](https://github.com/filimonov)). * In previous versions, queries with table functions named the same way as the implementation functions for operators were formatted inconsistently. Closes [#81601](https://github.com/ClickHouse/ClickHouse/issues/81601). Closes [#81977](https://github.com/ClickHouse/ClickHouse/issues/81977). Closes [#82834](https://github.com/ClickHouse/ClickHouse/issues/82834). Closes [#82835](https://github.com/ClickHouse/ClickHouse/issues/82835). EXPLAIN SYNTAX queries will not always format operators - the new behavior better reflects the purpose of explaining syntax. `clickhouse-format`, `formatQuery`, and similar will not format functions as operators if the query contained them in a functional form. [#82825](https://github.com/ClickHouse/ClickHouse/pull/82825) ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -292,7 +292,7 @@ doc_type: 'changelog' * Two slight changes to how `min_free_disk_ratio_to_perform_insert` and `min_free_disk_bytes_to_perform_insert` settings work: - use unreserved instead of available bytes to determine if an insert should be rejected. This is probably not crucial if the reservations for background merges and mutations are small compared to the configured thresholds, but it seems more correct. - Don't apply these settings to system tables. The reasoning for this is that we still want tables like `query_log` to be updated. This helps a lot with debugging. Data written to system tables is usually small compared to actual data, so they should be able to continue for much longer with a reasonable `min_free_disk_ratio_to_perform_insert` threshold. [#88468](https://github.com/ClickHouse/ClickHouse/pull/88468) ([c-end](https://github.com/c-end)). * Enable async mode for Keeper's internal replication. Keeper will preserve the same behavior as before with possible performance improvements. If you are updating from a version older than 23.9, you need to either update first to 23.9+ and than to 25.10+. You can also set `keeper_server.coordination_settings.async_replication` to 0 before update and enable it after update is done. [#88515](https://github.com/ClickHouse/ClickHouse/pull/88515) ([Antonio Andelic](https://github.com/antonio2368)). -#### New Feature +#### New feature * Add support for negative `LIMIT` and negative `OFFSET`. Closes [#28913](https://github.com/ClickHouse/ClickHouse/issues/28913). [#88411](https://github.com/ClickHouse/ClickHouse/pull/88411) ([Nihal Z. Miaji](https://github.com/nihalzp)). * The `Alias` engine creates a proxy to another table. All read and write operations are forwarded to the target table, while the alias itself stores no data and only maintains a reference to the target table. [#87965](https://github.com/ClickHouse/ClickHouse/pull/87965) ([Kai Zhu](https://github.com/nauu)). * Full support of operator `IS NOT DISTINCT FROM` (`<=>`). [#88155](https://github.com/ClickHouse/ClickHouse/pull/88155) ([simonmichal](https://github.com/simonmichal)). @@ -313,11 +313,11 @@ doc_type: 'changelog' * Add recursive variants of `cp`-`cpr` and `mv`-`mvr` commands in Keeper client. [#88570](https://github.com/ClickHouse/ClickHouse/pull/88570) ([Mikhail Artemenko](https://github.com/Michicosun)). * Added session setting to exclude list of skip indexes from materialization on inserts (`exclude_materialize_skip_indexes_on_insert`). Added merge tree table setting to exclude list of skip indexes from materialization during merge (`exclude_materialize_skip_indexes_on_merge`). [#87252](https://github.com/ClickHouse/ClickHouse/pull/87252) ([George Larionov](https://github.com/george-larionov)). -#### Experimental Feature +#### Experimental feature * Implement `QBit` data type that stores vectors in bit-sliced format and `L2DistanceTransposed` function that allows approximate vector search where precision-speed trade-off is controlled by a parameter. [#87922](https://github.com/ClickHouse/ClickHouse/pull/87922) ([Raufs Dunamalijevs](https://github.com/rienath)). * Functions `searchAll` and `searchAny` now work on top of columns without text columns. In those cases, they use the default tokenizer. [#87722](https://github.com/ClickHouse/ClickHouse/pull/87722) ([Jimmy Aguilar Mena](https://github.com/Ergus)). -#### Performance Improvement +#### Performance improvement * Implement lazy columns replication in JOIN and ARRAY JOIN. Avoid converting special columns representation like Sparse and Replicated to full columns in some output formats. This avoids unnecessary data copy in memory. [#88752](https://github.com/ClickHouse/ClickHouse/pull/88752) ([Pavel Kruglov](https://github.com/Avogar)). * Add optional `.size` subcolumn serialization for top-level String columns in MergeTree tables to improve compression and enable efficient subcolumn access. Introduce new MergeTree settings for serialization version control and expression optimization for empty strings. [#82850](https://github.com/ClickHouse/ClickHouse/pull/82850) ([Amos Bird](https://github.com/amosbird)). * Support for read in order for Iceberg. [#88454](https://github.com/ClickHouse/ClickHouse/pull/88454) ([scanhex12](https://github.com/scanhex12)). @@ -417,7 +417,7 @@ doc_type: 'changelog' * Limit exception message length when a constraint is violated. In previous versions, you could get a very long exception message when a very long string was inserted, and it ended up being written in the query_log. Closes [#87032](https://github.com/ClickHouse/ClickHouse/issues/87032). [#88801](https://github.com/ClickHouse/ClickHouse/pull/88801) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Fix requesting the structure of a dataset from the ArrowFlight server when creating a table. [#87542](https://github.com/ClickHouse/ClickHouse/pull/87542) ([Vitaly Baranov](https://github.com/vitlibar)). -#### Bug Fix (user-visible misbehavior in an official stable release) +#### Bug fix (user-visible misbehavior in an official stable release) * Fixed GeoParquet causing client protocol errors. [#84020](https://github.com/ClickHouse/ClickHouse/pull/84020) ([Michael Kolupaev](https://github.com/al13n321)). * Fix resolving host-dependent functions like shardNum() in subqueries on initiator node. [#84409](https://github.com/ClickHouse/ClickHouse/pull/84409) ([Eduard Karacharov](https://github.com/korowa)). * Fixed incorrect handling of pre-epoch dates with fractional seconds in various date time related functions, such as `parseDateTime64BestEffort`, `change{Year,Month,Day}` and `makeDateTime64`. Previously the subsecond part was substracted from seconds instead of adding them. For example `parseDateTime64BestEffort('1969-01-01 00:00:00.468')` was returning `1968-12-31 23:59:59.532` instead of `1969-01-01 00:00:00.468`. [#85396](https://github.com/ClickHouse/ClickHouse/pull/85396) ([xiaohuanlin](https://github.com/xiaohuanlin)). @@ -522,7 +522,7 @@ doc_type: 'changelog' * Backported in [#89167](https://github.com/ClickHouse/ClickHouse/issues/89167): Fix crash in unknown catalog type. Resolves [#88819](https://github.com/ClickHouse/ClickHouse/issues/88819). [#88987](https://github.com/ClickHouse/ClickHouse/pull/88987) ([scanhex12](https://github.com/scanhex12)). * Backported in [#89028](https://github.com/ClickHouse/ClickHouse/issues/89028): Fixed performance degradation in analysis of skipping indexes. [#89004](https://github.com/ClickHouse/ClickHouse/pull/89004) ([Anton Popov](https://github.com/CurtizJ)). -#### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging improvement * Use `postgres` library version 18.0. [#87647](https://github.com/ClickHouse/ClickHouse/pull/87647) ([Konstantin Bogdanov](https://github.com/thevar1able)). * Enable ICU for FreeBSD. [#87891](https://github.com/ClickHouse/ClickHouse/pull/87891) ([Raúl Marín](https://github.com/Algunenano)). * Use SSE 4.2 when we use dynamic dispatch to SSE 4.2 and not SSE 4. [#88029](https://github.com/ClickHouse/ClickHouse/pull/88029) ([Raúl Marín](https://github.com/Algunenano)). @@ -532,13 +532,13 @@ doc_type: 'changelog' ### ClickHouse release 25.9, 2025-09-25 {#259} -#### Backward Incompatible Change +#### Backward incompatible change * Disable nonsensical binary operations with IPv4/IPv6: Plus / minus of a IPv4/IPv6 with a non-integer type is disabled. Before it would allow operations with floating types and throw logical errors with some other types (such as DateTime). [#86336](https://github.com/ClickHouse/ClickHouse/pull/86336) ([Raúl Marín](https://github.com/Algunenano)). * Deprecate setting `allow_dynamic_metadata_for_data_lakes`. Now all iceberg tables try to fetch up-to-date table schema from storage before executing of each query. [#86366](https://github.com/ClickHouse/ClickHouse/pull/86366) ([Daniil Ivanik](https://github.com/divanik)). * Changed resolving of the coalesced column from `OUTER JOIN ... USING` clause to be more consistent: previously, when selecting both the USING column and qualified columns (`a, t1.a, t2.a`) in a OUTER JOIN, the USING column would incorrectly be resolved to `t1.a`, showing 0/NULL for rows from the right table with no left match. Now identifiers from USING clause are always resolved to the coalesced column, while qualified identifiers resolve to the non-coalesced columns, regardless of which other identifiers are present in the query. For example: ```sql SELECT a, t1.a, t2.a FROM (SELECT 1 as a WHERE 0) t1 FULL JOIN (SELECT 2 as a) t2 USING (a) -- Before: a=0, t1.a=0, t2.a=2 (incorrect - 'a' resolved to t1.a) -- After: a=2, t1.a=0, t2.a=2 (correct - 'a' is coalesced). [#80848](https://github.com/ClickHouse/ClickHouse/pull/80848) ([Vladimir Cherkasov](https://github.com/vdimir)). * Increase replicated deduplication window up to 10000. This is fully compatible, but we can imagine scenarios when this change could lead to high resource consumption in the presence of a large number of tables. [#86820](https://github.com/ClickHouse/ClickHouse/pull/86820) ([Sema Checherinda](https://github.com/CheSema)). -#### New Feature +#### New feature * Users can now use NATS JetStream to consume messages by specifying the new settings of `nats_stream` and `nats_consumer` for the NATS engine. [#84799](https://github.com/ClickHouse/ClickHouse/pull/84799) ([Dmitry Novikov](https://github.com/dmitry-sles-novikov)). * Added support for authentication and SSL in the `arrowFlight` table function. [#87120](https://github.com/ClickHouse/ClickHouse/pull/87120) ([Vitaly Baranov](https://github.com/vitlibar)). * Add new parameter to `S3` table engine and `s3` table function named `storage_class_name` which allows to specify intelligent tiring supported by AWS. Supported both in key-value format and in positional (deprecated) format). [#87122](https://github.com/ClickHouse/ClickHouse/pull/87122) ([alesapin](https://github.com/alesapin)). @@ -561,12 +561,12 @@ doc_type: 'changelog' * Improve allocation profiling based on jemalloc's internal tooling. Global jemalloc profiler can now be enabled with config `jemalloc_enable_global_profiler`. Sampled global allocations and deallocations can be stored in `system.trace_log` under `JemallocSample` type by enabling config `jemalloc_collect_global_profile_samples_in_trace_log`. Jemalloc profiling can now be enabled for each query independently using setting `jemalloc_enable_profiler`. Storing samples in `system.trace_log` can be controlled per query using setting `jemalloc_collect_profile_samples_in_trace_log`. Update jemalloc to newer version. [#85438](https://github.com/ClickHouse/ClickHouse/pull/85438) ([Antonio Andelic](https://github.com/antonio2368)). * A new setting to delete files on dropping Iceberg tables. This closes [#86211](https://github.com/ClickHouse/ClickHouse/issues/86211). [#86501](https://github.com/ClickHouse/ClickHouse/pull/86501) ([scanhex12](https://github.com/scanhex12)). -#### Experimental Feature +#### Experimental feature * The inverted text index was reworked from scratch to be scalable for datasets that don't fit into RAM. [#86485](https://github.com/ClickHouse/ClickHouse/pull/86485) ([Anton Popov](https://github.com/CurtizJ)). * Join reordering now uses statistics. The feature can be enabled by setting `allow_statistics_optimize = 1` and `query_plan_optimize_join_order_limit = 10`. [#86822](https://github.com/ClickHouse/ClickHouse/pull/86822) ([Han Fei](https://github.com/hanfei1991)). * Support `alter table ... materialize statistics all` will materialize all the statistics of a table. [#87197](https://github.com/ClickHouse/ClickHouse/pull/87197) ([Han Fei](https://github.com/hanfei1991)). -#### Performance Improvement +#### Performance improvement * Support filtering data parts using skip indexes during reading to reduce unnecessary index reads. Controlled by the new setting `use_skip_indexes_on_data_read` (disabled by default). This addresses [#75774](https://github.com/ClickHouse/ClickHouse/issues/75774). This includes some common groundwork shared with [#81021](https://github.com/ClickHouse/ClickHouse/issues/81021). [#81526](https://github.com/ClickHouse/ClickHouse/pull/81526) ([Amos Bird](https://github.com/amosbird)). * Added JOIN order optimization that can automatically reorder JOINs for better performance (controlled by `query_plan_optimize_join_order_limit` setting). Note that the join order optimization currently has limited statistics support and primarily relies on row count estimates from storage engines - more sophisticated statistics collection and cardinality estimation will be added in future releases. **If you encounter issues with JOIN queries after upgrading**, you can temporarily disable the new implementation by setting `SET query_plan_use_new_logical_join_step = 0` and report the issue for investigation. **Note about resolution of identifiers from USING clause**: Changed resolving of the coalesced column from `OUTER JOIN ... USING` clause to be more consistent: previously, when selecting both the USING column and qualified columns (`a, t1.a, t2.a`) in a OUTER JOIN, the USING column would incorrectly be resolved to `t1.a`, showing 0/NULL for rows from the right table with no left match. Now identifiers from USING clause are always resolved to the coalesced column, while qualified identifiers resolve to the non-coalesced columns, regardless of which other identifiers are present in the query. For example: ```sql SELECT a, t1.a, t2.a FROM (SELECT 1 as a WHERE 0) t1 FULL JOIN (SELECT 2 as a) t2 USING (a) -- Before: a=0, t1.a=0, t2.a=2 (incorrect - 'a' resolved to t1.a) -- After: a=2, t1.a=0, t2.a=2 (correct - 'a' is coalesced). [#80848](https://github.com/ClickHouse/ClickHouse/pull/80848) ([Vladimir Cherkasov](https://github.com/vdimir)). * Distributed `INSERT SELECT` for data lakes. [#86783](https://github.com/ClickHouse/ClickHouse/pull/86783) ([scanhex12](https://github.com/scanhex12)). @@ -622,7 +622,7 @@ doc_type: 'changelog' * Enable short circuit evaluation for `parseDateTime` function. [#87184](https://github.com/ClickHouse/ClickHouse/pull/87184) ([Pavel Kruglov](https://github.com/Avogar)). * Add a new column `statistics` in `system.parts_columns`. [#87259](https://github.com/ClickHouse/ClickHouse/pull/87259) ([Han Fei](https://github.com/hanfei1991)). -#### Bug Fix (user-visible misbehavior in an official stable release) +#### Bug fix (user-visible misbehavior in an official stable release) * The results of alter queries are only validated on the initiator node for replicated databases and internally replicated tables. This will fix situations where an already committed alter query could get stuck on other nodes. [#83849](https://github.com/ClickHouse/ClickHouse/pull/83849) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). * Limit the number of tasks of each type in `BackgroundSchedulePool`. Avoid situations when all slots are occupied by task of one type, while other tasks are starving. Also avoids deadlocks when tasks wait for each other. This is controlled by `background_schedule_pool_max_parallel_tasks_per_type_ratio` server setting. [#84008](https://github.com/ClickHouse/ClickHouse/pull/84008) ([Alexander Tokmakov](https://github.com/tavplubix)). * Shutdown tables properly when recovering database replica. Improper shutdown would lead to LOGICAL_ERROR for some table engines during database replica recovery. [#84744](https://github.com/ClickHouse/ClickHouse/pull/84744) ([Antonio Andelic](https://github.com/antonio2368)). @@ -707,7 +707,7 @@ doc_type: 'changelog' * Fix possible wrong result of non-correlated `EXISTS`. It was broken with `execute_exists_as_scalar_subquery=1` which was introduced in https://github.com/ClickHouse/ClickHouse/pull/85481 and affects `25.8`. Fixes [#86415](https://github.com/ClickHouse/ClickHouse/issues/86415). [#87207](https://github.com/ClickHouse/ClickHouse/pull/87207) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Throws an error if iceberg_metadata_log is not configured, but user tries to get debug iceberg metadata info. Fixes nullptr access. [#87250](https://github.com/ClickHouse/ClickHouse/pull/87250) ([Daniil Ivanik](https://github.com/divanik)). -#### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging improvement * Fix compatibility with abseil-cpp 20250814.0, https://github.com/abseil/abseil-cpp/issues/1923. [#85970](https://github.com/ClickHouse/ClickHouse/pull/85970) ([Yuriy Chernyshov](https://github.com/georgthegreat)). * Put building standalone WASM lexer under a flag. [#86505](https://github.com/ClickHouse/ClickHouse/pull/86505) ([Konstantin Bogdanov](https://github.com/thevar1able)). * Fix crc32c build on older ARM CPUs without support for the `vmull_p64` instruction. [#86521](https://github.com/ClickHouse/ClickHouse/pull/86521) ([Pablo Marcos](https://github.com/pamarcos)). @@ -717,9 +717,9 @@ doc_type: 'changelog' * Fix unpacking Rust crate names in `list-licenses.sh`. [#87305](https://github.com/ClickHouse/ClickHouse/pull/87305) ([Konstantin Bogdanov](https://github.com/thevar1able)). -### ClickHouse release 25.8 LTS, 2025-08-28 {#258} +### ClickHouse release 25.8 lts, 2025-08-28 {#258} -#### Backward Incompatible Change +#### Backward incompatible change * Infer `Array(Dynamic)` instead of unnamed `Tuple` for arrays of values with different types in JSON. To use the previous behaviour, disable setting `input_format_json_infer_array_of_dynamic_from_array_of_different_types`. [#80859](https://github.com/ClickHouse/ClickHouse/pull/80859) ([Pavel Kruglov](https://github.com/Avogar)). * Move S3 latency metrics to histograms for homogeneity and simplicity. [#82305](https://github.com/ClickHouse/ClickHouse/pull/82305) ([Miсhael Stetsyuk](https://github.com/mstetsyuk)). * Require backticks around identifiers with dots in default expressions to prevent them from being parsed as compound identifiers. [#83162](https://github.com/ClickHouse/ClickHouse/pull/83162) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). @@ -732,7 +732,7 @@ doc_type: 'changelog' * Enables `allow_dynamic_metadata_for_data_lakes` setting (Table Engine level setting) by default. [#85044](https://github.com/ClickHouse/ClickHouse/pull/85044) ([Daniil Ivanik](https://github.com/divanik)). * Disable quoting 64 bit integers in JSON formats by default. [#74079](https://github.com/ClickHouse/ClickHouse/pull/74079) ([Pavel Kruglov](https://github.com/Avogar)) -#### New Feature +#### New feature * Basic support for the PromQL dialect is added. To use it, set `dialect='promql'` in clickhouse-client, point it to the TimeSeries table using the setting `promql_table_name='X'` and execute queries like `rate(ClickHouseProfileEvents_ReadCompressedBytes[1m])[5m:1m]`. In addition you can wrap the PromQL query with SQL: `SELECT * FROM prometheusQuery('up', ...);`. So far only functions `rate`, `delta` and `increase` are supported. No unary/binary operators. No HTTP API. [#75036](https://github.com/ClickHouse/ClickHouse/pull/75036) ([Vitaly Baranov](https://github.com/vitlibar)). * AI Powered SQL generation can now infer from env ANTHROPIC_API_KEY and OPENAI_API_KEY if available, this is to make it so that we can have a zero config option to use this feature. [#83787](https://github.com/ClickHouse/ClickHouse/pull/83787) ([Kaushik Iska](https://github.com/iskakaushik)). * Implement support for [ArrowFlight RPC](https://arrow.apache.org/docs/format/Flight.html) protocol by adding: - new table function `arrowflight`. [#74184](https://github.com/ClickHouse/ClickHouse/pull/74184) ([zakr600](https://github.com/zakr600)). @@ -779,7 +779,7 @@ doc_type: 'changelog' * Adds an option `--max-concurrency` for the `clickhouse-benchmark` tool that enables a mode with a gradual increase in the number of parallel queries. [#85623](https://github.com/ClickHouse/ClickHouse/pull/85623) ([Sergei Trifonov](https://github.com/serxa)). * Support partially aggregated metrics. [#85328](https://github.com/ClickHouse/ClickHouse/pull/85328) ([Mikhail Artemenko](https://github.com/Michicosun)). -#### Experimental Feature +#### Experimental feature * Enable correlated subqueries support by default, they are no longer experimental. [#85107](https://github.com/ClickHouse/ClickHouse/pull/85107) ([Dmitry Novik](https://github.com/novikd)). * Unity, Glue, Rest, and Hive Metastore data lake catalogs are promoted from experimental to beta. [#85848](https://github.com/ClickHouse/ClickHouse/pull/85848) ([Melvyn Peignon](https://github.com/melvynator)). * Lightweight updates and deletes are promoted from experimental to beta. @@ -787,7 +787,7 @@ doc_type: 'changelog' * Ytsaurus table engine and table function. [#77606](https://github.com/ClickHouse/ClickHouse/pull/77606) ([MikhailBurdukov](https://github.com/MikhailBurdukov)). * Previously, the text index data would be separated into multiple segments (each segment size by default was 256 MiB). This might reduce the memory consumption while building the text index, however this increases the space requirement on the disk and increase the query response time. [#84590](https://github.com/ClickHouse/ClickHouse/pull/84590) ([Elmi Ahmadov](https://github.com/ahmadov)). -#### Performance Improvement +#### Performance improvement * New parquet reader implementation. It's generally faster and supports page-level filter pushdown and PREWHERE. Currently experimental. Use setting `input_format_parquet_use_native_reader_v3` to enable. [#82789](https://github.com/ClickHouse/ClickHouse/pull/82789) ([Michael Kolupaev](https://github.com/al13n321)). * Replaced the official HTTP transport in Azure library with our own HTTP client implementation for Azure Blob Storage. Introduced multiple settings for this clients which mirror settings from S3. Introduced aggressive connection timeouts for both Azure and S3. Improved introspection into Azure profile events and metrics. New client is enabled by default, provide much better latencies for cold queries on top of Azure Blob Storage. Old `Curl` client can be returned back by setting `azure_sdk_use_native_client=false`. [#83294](https://github.com/ClickHouse/ClickHouse/pull/83294) ([alesapin](https://github.com/alesapin)). The previous, official implementation of Azure client was unsuitable for production due to terrible latency spikes, ranging from five seconds to minutes. We have ditched that terrible implementation and are very proud of that. * Processes indexes in increasing order of file size. The net index ordering prioritizes minmax and vector indexes (due to simplicity and selectivity respectively), and small indexes thereafter. Within the minmax/vector indexes smaller indexes are also preferred. [#84094](https://github.com/ClickHouse/ClickHouse/pull/84094) ([Maruth Goyal](https://github.com/maruthgoyal)). @@ -903,7 +903,7 @@ doc_type: 'changelog' * Support inner arrays for the function `nested`. [#85719](https://github.com/ClickHouse/ClickHouse/pull/85719) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * All the allocations done by external libraries are now visible to ClickHouse's memory tracker and accounted properly. This may result in "increased" reported memory usage for certain queries or failures with `MEMORY_LIMIT_EXCEEDED`. [#84082](https://github.com/ClickHouse/ClickHouse/pull/84082) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). -#### Bug Fix (user-visible misbehavior in an official stable release) +#### Bug fix (user-visible misbehavior in an official stable release) * This pr fixes the metadata resolution when querying iceberg tables through rest catalog. ... [#80562](https://github.com/ClickHouse/ClickHouse/pull/80562) ([Saurabh Kumar Ojha](https://github.com/saurabhojha)). * Fix markReplicasActive in DDLWorker and DatabaseReplicatedDDLWorker. [#81395](https://github.com/ClickHouse/ClickHouse/pull/81395) ([Tuan Pham Anh](https://github.com/tuanpach)). @@ -1025,7 +1025,7 @@ doc_type: 'changelog' * Don't create empty iceberg delete file. [#86061](https://github.com/ClickHouse/ClickHouse/pull/86061) ([Konstantin Vedernikov](https://github.com/scanhex12)). * Fix large setting values breaking S3Queue tables and replica restart. [#86074](https://github.com/ClickHouse/ClickHouse/pull/86074) ([Nikolay Degterinsky](https://github.com/evillique)). -#### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging improvement * Use encrypted disks for tests with S3 by default. [#59898](https://github.com/ClickHouse/ClickHouse/pull/59898) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Use `clickhouse` binary in integrations tests to get unstripped debug symbols. [#83779](https://github.com/ClickHouse/ClickHouse/pull/83779) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). * Bumped the internal libxml2 from 2.14.4 to 2.14.5. [#84230](https://github.com/ClickHouse/ClickHouse/pull/84230) ([Robert Schulze](https://github.com/rschu1ze)). @@ -1035,7 +1035,7 @@ doc_type: 'changelog' ### ClickHouse release 25.7, 2025-07-24 {#257} -#### Backward Incompatible Change +#### Backward incompatible change * Changes to `extractKeyValuePairs` function: introduce a new argument `unexpected_quoting_character_strategy` that controls what happens when a `quoting_character` is unexpectedly found when reading a non quoted key or value. The value can be one of: `invalid`, `accept` or `promote`. Invalid will discard the key and go back to waiting key state. Accept will treat it as part of the key. Promote will discard previous character and start parsing as a quoted key. In addition, after parsing a quoted value, only parse the next key if a pair delimiter is found. [#80657](https://github.com/ClickHouse/ClickHouse/pull/80657) ([Arthur Passos](https://github.com/arthurpassos)). * Support zero-byte match in `countMatches` function. Users who like to retain the old behavior can enable setting `count_matches_stop_at_empty_match`. [#81676](https://github.com/ClickHouse/ClickHouse/pull/81676) ([Elmi Ahmadov](https://github.com/ahmadov)). * Use server-wide throttlers for local (`max_local_read_bandwidth_for_server` and `max_local_write_bandwidth_for_server`) and remote (`max_remote_read_network_bandwidth_for_server` and `max_remote_write_network_bandwidth_for_server`) when generating BACKUPs in addition to their dedicated server settings (`max_backup_bandwidth_for_server`, `max_mutations_bandwidth_for_server` and `max_merges_bandwidth_for_server`). [#81753](https://github.com/ClickHouse/ClickHouse/pull/81753) ([Sergei Trifonov](https://github.com/serxa)). @@ -1043,7 +1043,7 @@ doc_type: 'changelog' * Parallelize cluster functions by files within archives. In previous versions, the whole archive (such as zip, tar, or 7z) was a unit of work. Added a new setting `cluster_function_process_archive_on_multiple_nodes`, by default equal to `true`. If set to `true`, increases performance of processing archives in cluster functions. Should be set to `false` for compatibility and to avoid errors during upgrade to 25.7+ if using cluster functions with archives on earlier versions. [#82355](https://github.com/ClickHouse/ClickHouse/pull/82355) ([Kseniia Sumarokova](https://github.com/kssenii)). * `SYSTEM RESTART REPLICAS` query led to the wakeup of tables in the Lazy database, even without access to that database, and it happened while these tables were being concurrently dropped. Note: Now `SYSTEM RESTART REPLICAS` will only restart replicas in the databases where you have permission to `SHOW TABLES`, which is natural. [#83321](https://github.com/ClickHouse/ClickHouse/pull/83321) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### New Feature +#### New feature * Added support for lightweight updates for `MergeTree`-family tables. Lightweight updates can be used by a new syntax: `UPDATE
SET col1 = val1, col2 = val2, ... WHERE `. Added implementation of lightweight deletes via lightweight updates. It can be enabled by setting `lightweight_delete_mode = 'lightweight_update'`. [#82004](https://github.com/ClickHouse/ClickHouse/pull/82004) ([Anton Popov](https://github.com/CurtizJ)). * Support complex types in Iceberg schema evolution. [#73714](https://github.com/ClickHouse/ClickHouse/pull/73714) ([Konstantin Vedernikov](https://github.com/scanhex12)). * Introduce support for INSERTs into Iceberg tables. [#82692](https://github.com/ClickHouse/ClickHouse/pull/82692) ([Konstantin Vedernikov](https://github.com/scanhex12)). @@ -1062,7 +1062,7 @@ doc_type: 'changelog' * Allow parameters in `CREATE USER` queries for usernames. [#81387](https://github.com/ClickHouse/ClickHouse/pull/81387) ([Diskein](https://github.com/Diskein)). * The `system.formats` table now contains extended information about formats, such as HTTP content type, the capabilities of schema inference, etc. [#81505](https://github.com/ClickHouse/ClickHouse/pull/81505) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### Experimental Feature +#### Experimental feature * Added functions `searchAny` and `searchAll` which are general purpose tools to search text indexes. [#80641](https://github.com/ClickHouse/ClickHouse/pull/80641) ([Elmi Ahmadov](https://github.com/ahmadov)). * The text index now supports the new `split` tokenizer. [#81752](https://github.com/ClickHouse/ClickHouse/pull/81752) ([Elmi Ahmadov](https://github.com/ahmadov)). * Changed the default index granularity value for `text` indexes to 64. This improves the expected performance for the average test query in internal benchmarks. [#82162](https://github.com/ClickHouse/ClickHouse/pull/82162) ([Jimmy Aguilar Mena](https://github.com/Ergus)). @@ -1073,7 +1073,7 @@ doc_type: 'changelog' * Integrate `StorageKafka2` to `system.kafka_consumers`. [#82652](https://github.com/ClickHouse/ClickHouse/pull/82652) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). * Estimate complex CNF/DNF, for example, `(a < 1 and a > 0) or b = 3`, by statistics. [#82663](https://github.com/ClickHouse/ClickHouse/pull/82663) ([Han Fei](https://github.com/hanfei1991)). -#### Performance Improvement +#### Performance improvement * Introduce async logging. When logs are output to a slow device, it no longer delays queries. [#82516](https://github.com/ClickHouse/ClickHouse/pull/82516) ([Raúl Marín](https://github.com/Algunenano)). Limit the max number of entries that are hold in the queue. [#83214](https://github.com/ClickHouse/ClickHouse/pull/83214) ([Raúl Marín](https://github.com/Algunenano)). * Parallel distributed INSERT SELECT is enabled by default in mode where INSERT SELECT executed on each shard independently, see `parallel_distributed_insert_select` setting. [#83040](https://github.com/ClickHouse/ClickHouse/pull/83040) ([Igor Nikonov](https://github.com/devcrafter)). * When the aggregation query contains only a single `count()` function on a not-`Nullable` column, the aggregation logic is fully inlined during hash table probing. This avoids allocating and maintaining any aggregation state, significantly reducing memory usage and CPU overhead. This partially addresses [#81982](https://github.com/ClickHouse/ClickHouse/issues/81982). [#82104](https://github.com/ClickHouse/ClickHouse/pull/82104) ([Amos Bird](https://github.com/amosbird)). @@ -1167,7 +1167,7 @@ doc_type: 'changelog' * Made exception messages for certain situations for loading and adding projections easier to read. [#83728](https://github.com/ClickHouse/ClickHouse/pull/83728) ([Robert Schulze](https://github.com/rschu1ze)). * Introduce a configuration option to skip binary checksum integrity checks for `clickhouse-server`. Resolves [#83637](https://github.com/ClickHouse/ClickHouse/issues/83637). [#83749](https://github.com/ClickHouse/ClickHouse/pull/83749) ([Rafael Roquetto](https://github.com/rafaelroquetto)). -#### Bug Fix (user-visible misbehavior in an official stable release) +#### Bug fix (user-visible misbehavior in an official stable release) * Fix the wrong default value for the `--reconnect` option in `clickhouse-benchmark`. It was changed by mistake in [#79465](https://github.com/ClickHouse/ClickHouse/issues/79465). [#82677](https://github.com/ClickHouse/ClickHouse/pull/82677) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Fix inconsistent formatting of `CREATE DICTIONARY`. Closes [#82105](https://github.com/ClickHouse/ClickHouse/issues/82105). [#82829](https://github.com/ClickHouse/ClickHouse/pull/82829) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Fix inconsistent formatting of TTL when it contains a `materialize` function. Closes [#82828](https://github.com/ClickHouse/ClickHouse/issues/82828). [#82831](https://github.com/ClickHouse/ClickHouse/pull/82831) ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -1267,7 +1267,7 @@ doc_type: 'changelog' * Introduce backward compatibility setting to allow new analyzer to reference outer alias in WITH clause in the case of name clashes. Fixes [#82700](https://github.com/ClickHouse/ClickHouse/issues/82700). [#83797](https://github.com/ClickHouse/ClickHouse/pull/83797) ([Dmitry Novik](https://github.com/novikd)). * Fix deadlock on shutdown due to recursive context locking during library bridge cleanup. [#83824](https://github.com/ClickHouse/ClickHouse/pull/83824) ([Azat Khuzhin](https://github.com/azat)). -#### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging improvement * Build a minimal C library (10 KB) for the ClickHouse lexer. This is needed for [#80977](https://github.com/ClickHouse/ClickHouse/issues/80977). [#81347](https://github.com/ClickHouse/ClickHouse/pull/81347) ([Alexey Milovidov](https://github.com/alexey-milovidov)). Add test for standalone lexer, add test tag `fasttest-only`. [#82472](https://github.com/ClickHouse/ClickHouse/pull/82472) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). * Add a check for Nix submodule inputs. [#81691](https://github.com/ClickHouse/ClickHouse/pull/81691) ([Konstantin Bogdanov](https://github.com/thevar1able)). * Fix a list of problems that can occur when trying to run integration tests on a localhost. [#82135](https://github.com/ClickHouse/ClickHouse/pull/82135) ([Oleg Doronin](https://github.com/dorooleg)). @@ -1282,12 +1282,12 @@ doc_type: 'changelog' ### ClickHouse release 25.6, 2025-06-26 {#256} -#### Backward Incompatible Change +#### Backward incompatible change * Previously, function `countMatches` would stop counting at the first empty match even if the pattern accepts it. To overcome this issue, `countMatches` now continues execution by advancing by a single character when an empty match occurs. Users who like to retain the old behavior can enable setting `count_matches_stop_at_empty_match`. [#81676](https://github.com/ClickHouse/ClickHouse/pull/81676) ([Elmi Ahmadov](https://github.com/ahmadov)). * Minor: Force `backup_threads` and `restore_threads` server settings to be non zero. [#80224](https://github.com/ClickHouse/ClickHouse/pull/80224) ([Raúl Marín](https://github.com/Algunenano)). * Minor: Fix `bitNot` for `String` will return a zero-terminated string in the internal memory representation. This should not affect any user visible behavior, however the author wanted to highlight this change. [#80791](https://github.com/ClickHouse/ClickHouse/pull/80791) ([Azat Khuzhin](https://github.com/azat)). -#### New Feature +#### New feature * New data types: `Time` (\[H\]HH:MM:SS) and `Time64` (\[H\]HH:MM:SS\[.fractional\]), and some basic cast functions and functions to interact with other data types. Added settings for compatibility with the existing function `toTime`. The setting `use_legacy_to_time` is set to keep the old behavior for now. [#81217](https://github.com/ClickHouse/ClickHouse/pull/81217) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). Support comparison between Time/Time64. [#80327](https://github.com/ClickHouse/ClickHouse/pull/80327) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). * A new CLI tool, [`chdig`](https://github.com/azat/chdig/) - TUI interface for ClickHouse (top like) as part of ClickHouse. [#79666](https://github.com/ClickHouse/ClickHouse/pull/79666) ([Azat Khuzhin](https://github.com/azat)). * Support `disk` setting for `Atomic` and `Ordinary` database engines, specifying the disk to store table metadata files. [#80546](https://github.com/ClickHouse/ClickHouse/pull/80546) ([Tuan Pham Anh](https://github.com/tuanpach)). This allows attaching databases from external sources. @@ -1313,13 +1313,13 @@ doc_type: 'changelog' * Function `tokens` now supports a new tokenizer, named `split`, which is good for logs. [#80195](https://github.com/ClickHouse/ClickHouse/pull/80195) ([Robert Schulze](https://github.com/rschu1ze)). * Add support for the `--database` argument in `clickhouse-local`. You can switch to a previously created database. This closes [#44115](https://github.com/ClickHouse/ClickHouse/issues/44115). [#81465](https://github.com/ClickHouse/ClickHouse/pull/81465) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### Experimental Feature +#### Experimental feature * Implement Kafka rebalance like logic for `Kafka2` using ClickHouse Keeper For each replica we support two types of partition locks: permanent locks and temporary locks. The replica tries to hold permanent locks as long as possible, at any given time there are no more than `all_topic_partitions / active_replicas_count` (here `all_topic_partitions` is the number of all partitions, `active_replicas_count` is the number of active replicas) permanent locks on the replica, if there are more, then the replica releases some partitions. Some partitions are temporarily held by the replica. The maximum number of temporary locks on a replica changes dynamically to give other replicas a chance to take some partitions into permanent locks. When updating temporary locks, the replica releases them all and tries to take some others again. [#78726](https://github.com/ClickHouse/ClickHouse/pull/78726) ([Daria Fomina](https://github.com/sinfillo)). * An improvement for the experimental text index: explicit parameters are supported via key-value pairs. Currently, supported parameters are a mandatory `tokenizer` and two optional `max_rows_per_postings_list` and `ngram_size`. [#80262](https://github.com/ClickHouse/ClickHouse/pull/80262) ([Elmi Ahmadov](https://github.com/ahmadov)). * Previously, `packed` storage was not supported for the full-text index, because the segment id was updated on-fly by reading and writing (`.gin_sid`) file on disk. In case of packed storage, reading a value from the uncommited file is not supported and this led to an issue. Now it is alright. [#80852](https://github.com/ClickHouse/ClickHouse/pull/80852) ([Elmi Ahmadov](https://github.com/ahmadov)). * Experimental indexes of type `gin` (which I don't like because it is an inside joke of PostgreSQL hackers) were renamed to `text`. Existing indexes of type `gin` remain loadable but they will throw an exception (suggesting `text` indexes instead) when one tries to use them in searches. [#80855](https://github.com/ClickHouse/ClickHouse/pull/80855) ([Robert Schulze](https://github.com/rschu1ze)). -#### Performance Improvement +#### Performance improvement * Enable multiple-projection filtering support, allowing to use more than one projection for part-level filtering. This addresses [#55525](https://github.com/ClickHouse/ClickHouse/issues/55525). This is the second step to implement projection index, following [#78429](https://github.com/ClickHouse/ClickHouse/issues/78429). [#80343](https://github.com/ClickHouse/ClickHouse/pull/80343) ([Amos Bird](https://github.com/amosbird)). * Use `SLRU` cache policy in filesystem cache by default. [#75072](https://github.com/ClickHouse/ClickHouse/pull/75072) ([Kseniia Sumarokova](https://github.com/kssenii)). * Remove contention in the Resize step in query pipeline. [#77562](https://github.com/ClickHouse/ClickHouse/pull/77562) ([Zhiguo Zhou](https://github.com/ZhiguoZh)). @@ -1399,7 +1399,7 @@ doc_type: 'changelog' * Add profile event `PageCacheReadBytes`. [#81742](https://github.com/ClickHouse/ClickHouse/pull/81742) ([Kseniia Sumarokova](https://github.com/kssenii)). * Fix logical error in filesystem cache: "Having zero bytes but range is not finished". [#81868](https://github.com/ClickHouse/ClickHouse/pull/81868) ([Kseniia Sumarokova](https://github.com/kssenii)). -#### Bug Fix (user-visible misbehavior in an official stable release) +#### Bug fix (user-visible misbehavior in an official stable release) * Fix parameterized view with SELECT EXCEPT query. Closes [#49447](https://github.com/ClickHouse/ClickHouse/issues/49447). [#57380](https://github.com/ClickHouse/ClickHouse/pull/57380) ([Nikolay Degterinsky](https://github.com/evillique)). * Analyzer: Fix column projection name after column type promotion in join. Closes [#63345](https://github.com/ClickHouse/ClickHouse/issues/63345). [#63519](https://github.com/ClickHouse/ClickHouse/pull/63519) ([Dmitry Novik](https://github.com/novikd)). * Fixed a logical error in cases of column name clashes when analyzer_compatibility_join_using_top_level_identifier is enabled. [#75676](https://github.com/ClickHouse/ClickHouse/pull/75676) ([Vladimir Cherkasov](https://github.com/vdimir)). @@ -1506,7 +1506,7 @@ doc_type: 'changelog' * Fixing copy-paste error in `arraySimilarity`, disallowing the use of `UInt32` and `Int32` weights. Update tests and docs. [#82103](https://github.com/ClickHouse/ClickHouse/pull/82103) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). * Fix possible data-race between suggestion thread and main client thread. [#82233](https://github.com/ClickHouse/ClickHouse/pull/82233) ([Azat Khuzhin](https://github.com/azat)). -#### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging improvement * Use `postgres` 16.9. [#81437](https://github.com/ClickHouse/ClickHouse/pull/81437) ([Konstantin Bogdanov](https://github.com/thevar1able)). * Use `openssl` 3.2.4. [#81438](https://github.com/ClickHouse/ClickHouse/pull/81438) ([Konstantin Bogdanov](https://github.com/thevar1able)). * Use `abseil-cpp` 2025-01-27. [#81440](https://github.com/ClickHouse/ClickHouse/pull/81440) ([Konstantin Bogdanov](https://github.com/thevar1able)). @@ -1531,7 +1531,7 @@ doc_type: 'changelog' ### ClickHouse release 25.5, 2025-05-22 {#255} -#### Backward Incompatible Change +#### Backward incompatible change * Function `geoToH3` now accepts the input in the order (lat, lon, res) (which is common for other geometric functions). Users who wish to retain the previous result order (lon, lat, res) can set setting `geotoh3_argument_order = 'lon_lat'`. [#78852](https://github.com/ClickHouse/ClickHouse/pull/78852) ([Pratima Patel](https://github.com/pratimapatel2008)). * Add a filesystem cache setting `allow_dynamic_cache_resize`, by default `false`, to allow dynamic resize of filesystem cache. Why: in certain environments (ClickHouse Cloud) all the scaling events happen through the restart of the process and we would love this feature to be explicitly disabled to have more control over the behaviour + as a safety measure. This PR is marked as backward incompatible, because in older versions dynamic cache resize worked by default without special setting. [#79148](https://github.com/ClickHouse/ClickHouse/pull/79148) ([Kseniia Sumarokova](https://github.com/kssenii)). * Removed support for legacy index types `annoy` and `usearch`. Both have been stubs for a long time, i.e. every attempt to use the legacy indexes returned an error anyways. If you still have `annoy` and `usearch` indexes, please drop them. [#79802](https://github.com/ClickHouse/ClickHouse/pull/79802) ([Robert Schulze](https://github.com/rschu1ze)). @@ -1539,7 +1539,7 @@ doc_type: 'changelog' * Enable `DeltaLake` storage `delta-kernel-rs` implementation by default. [#79541](https://github.com/ClickHouse/ClickHouse/pull/79541) ([Kseniia Sumarokova](https://github.com/kssenii)). * If reading from an `URL` involves multiple redirects, setting `enable_url_encoding` is correctly applied across all redirects in the chain. [#79563](https://github.com/ClickHouse/ClickHouse/pull/79563) ([Shankar Iyer](https://github.com/shankar-iyer)). Setting `enble_url_encoding` default value is now set to `false`. [#80088](https://github.com/ClickHouse/ClickHouse/pull/80088) ([Shankar Iyer](https://github.com/shankar-iyer)). -#### New Feature +#### New feature * Support scalar correlated subqueries in the WHERE clause. Closes [#6697](https://github.com/ClickHouse/ClickHouse/issues/6697). [#79600](https://github.com/ClickHouse/ClickHouse/pull/79600) ([Dmitry Novik](https://github.com/novikd)). Support correlated subqueries in the projection list in simple cases. [#79925](https://github.com/ClickHouse/ClickHouse/pull/79925) ([Dmitry Novik](https://github.com/novikd)). [#76078](https://github.com/ClickHouse/ClickHouse/pull/76078) ([Dmitry Novik](https://github.com/novikd)). Now it covers 100% of TPC-H test suite. * Vector search using the vector similarity index is now beta (from previously experimental). [#80164](https://github.com/ClickHouse/ClickHouse/pull/80164) ([Robert Schulze](https://github.com/rschu1ze)). * Support geo types in `Parquet` format. This closes [#75317](https://github.com/ClickHouse/ClickHouse/issues/75317). [#79777](https://github.com/ClickHouse/ClickHouse/pull/79777) ([scanhex12](https://github.com/scanhex12)). @@ -1555,13 +1555,13 @@ doc_type: 'changelog' * Clickhouse vector search now supports both pre-filtering and post-filtering and provides related settings for finer control. (issue [#78161](https://github.com/ClickHouse/ClickHouse/issues/78161)). [#79854](https://github.com/ClickHouse/ClickHouse/pull/79854) ([Shankar Iyer](https://github.com/shankar-iyer)). * Add [`icebergHash`](https://iceberg.apache.org/spec/#appendix-b-32-bit-hash-requirements) and [`icebergBucket`](https://iceberg.apache.org/spec/#bucket-transform-details) functions. Support data files pruning in `Iceberg` tables partitioned with [`bucket transfom`](https://iceberg.apache.org/spec/#partitioning). [#79262](https://github.com/ClickHouse/ClickHouse/pull/79262) ([Daniil Ivanik](https://github.com/divanik)). -#### Experimental Feature +#### Experimental feature * New `Time`/`Time64` data types: `Time` (HHH:MM:SS) and `Time64` (HHH:MM:SS.``) and some basic cast functions and functions to interact with other data types. Also, changed the existing function's name toTime to toTimeWithFixedDate because the function toTime is required for the cast function. [#75735](https://github.com/ClickHouse/ClickHouse/pull/75735) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). 72459). * Hive metastore catalog for Iceberg datalake. [#77677](https://github.com/ClickHouse/ClickHouse/pull/77677) ([scanhex12](https://github.com/scanhex12)). * Indexes of type `full_text` were renamed to `gin`. This follows the more familiar terminology of PostgreSQL and other databases. Existing indexes of type `full_text` remain loadable but they will throw an exception (suggesting `gin` indexes instead) when one tries to use them in searches. [#79024](https://github.com/ClickHouse/ClickHouse/pull/79024) ([Robert Schulze](https://github.com/rschu1ze)). -#### Performance Improvement +#### Performance improvement * Change the Compact part format to save marks for each substream to be able to read individual subcolumns. Old Compact format is still supported for reads and can be enabled for writes using MergeTree setting `write_marks_for_substreams_in_compact_parts`. It's disabled by default for safer upgrades as it changes the compact parts storage. It will be enabled by default in one of the next releases. [#77940](https://github.com/ClickHouse/ClickHouse/pull/77940) ([Pavel Kruglov](https://github.com/Avogar)). * Allow moving conditions with subcolumns to prewhere. [#79489](https://github.com/ClickHouse/ClickHouse/pull/79489) ([Pavel Kruglov](https://github.com/Avogar)). * Speed up secondary indices by evaluating their expressions on multiple granules at once. [#64109](https://github.com/ClickHouse/ClickHouse/pull/64109) ([Alexey Milovidov](https://github.com/alexey-milovidov)). @@ -1639,7 +1639,7 @@ doc_type: 'changelog' * Add system.parts table visualizer. [#79437](https://github.com/ClickHouse/ClickHouse/pull/79437) ([Sergei Trifonov](https://github.com/serxa)). * Add tool for query latency analyzing. [#79978](https://github.com/ClickHouse/ClickHouse/pull/79978) ([Sergei Trifonov](https://github.com/serxa)). -#### Bug Fix (user-visible misbehavior in an official stable release) +#### Bug fix (user-visible misbehavior in an official stable release) * Fix renames of columns missing in part. [#76346](https://github.com/ClickHouse/ClickHouse/pull/76346) ([Anton Popov](https://github.com/CurtizJ)). * A materialized view can start too late, e.g. after the Kafka table that streams to it. [#72123](https://github.com/ClickHouse/ClickHouse/pull/72123) ([Ilya Golshtein](https://github.com/ilejn)). * Fix `SELECT` query rewriting during `VIEW` creation with enabled analyzer. closes [#75956](https://github.com/ClickHouse/ClickHouse/issues/75956). [#76356](https://github.com/ClickHouse/ClickHouse/pull/76356) ([Dmitry Novik](https://github.com/novikd)). @@ -1705,7 +1705,7 @@ doc_type: 'changelog' * `enable_user_name_access_type` must not affect `DEFINER` access type. [#80026](https://github.com/ClickHouse/ClickHouse/pull/80026) ([pufit](https://github.com/pufit)). * Query to system database can hang if system database metadata located in keeper. [#79304](https://github.com/ClickHouse/ClickHouse/pull/79304) ([Mikhail Artemenko](https://github.com/Michicosun)). -#### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging improvement * Make it possible to reuse the built `chcache` binary instead of always rebuilding it. [#78851](https://github.com/ClickHouse/ClickHouse/pull/78851) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). * Add NATS pause waiting. [#78987](https://github.com/ClickHouse/ClickHouse/pull/78987) ([Dmitry Novikov](https://github.com/dmitry-sles-novikov)). * Fix for incorrectly publishing ARM build as amd64compat. [#79122](https://github.com/ClickHouse/ClickHouse/pull/79122) ([Alexander Gololobov](https://github.com/davenger)). @@ -1717,13 +1717,13 @@ doc_type: 'changelog' ### ClickHouse release 25.4, 2025-04-22 {#254} -#### Backward Incompatible Change +#### Backward incompatible change * Check if all columns in a materialized view match the target table when `allow_materialized_view_with_bad_select` is `false`. [#74481](https://github.com/ClickHouse/ClickHouse/pull/74481) ([Christoph Wurm](https://github.com/cwurm)). * Fix cases where `dateTrunc` is used with negative Date/DateTime arguments. [#77622](https://github.com/ClickHouse/ClickHouse/pull/77622) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). * The legacy `MongoDB` integration has been removed. Server setting `use_legacy_mongodb_integration` became obsolete and now does nothing. [#77895](https://github.com/ClickHouse/ClickHouse/pull/77895) ([Robert Schulze](https://github.com/rschu1ze)). * Enhance `SummingMergeTree` validation to skip aggregation for columns used in partition or sort keys. [#78022](https://github.com/ClickHouse/ClickHouse/pull/78022) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). -#### New Feature +#### New feature * Added CPU slot scheduling for workloads, see [the docs](https://clickhouse.com/docs/operations/workload-scheduling#cpu_scheduling) for details. [#77595](https://github.com/ClickHouse/ClickHouse/pull/77595) ([Sergei Trifonov](https://github.com/serxa)). * `clickhouse-local` will retain its databases after restart if you specify the `--path` command line argument. This closes [#50647](https://github.com/ClickHouse/ClickHouse/issues/50647). This closes [#49947](https://github.com/ClickHouse/ClickHouse/issues/49947). [#71722](https://github.com/ClickHouse/ClickHouse/pull/71722) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Reject queries when the server is overloaded. The decision is made based on the ratio of wait time (`OSCPUWaitMicroseconds`) to busy time (`OSCPUVirtualTimeMicroseconds`). The query is dropped with some probability, when this ratio is between `min_os_cpu_wait_time_ratio_to_throw` and `max_os_cpu_wait_time_ratio_to_throw` (those are query level settings). [#63206](https://github.com/ClickHouse/ClickHouse/pull/63206) ([Alexey Katsman](https://github.com/alexkats)). @@ -1747,11 +1747,11 @@ doc_type: 'changelog' * Add several convenient ways to resolve root `metadata.json` file in an iceberg table function and engine. Closes [#78455](https://github.com/ClickHouse/ClickHouse/issues/78455). [#78475](https://github.com/ClickHouse/ClickHouse/pull/78475) ([Daniil Ivanik](https://github.com/divanik)). * Support password based auth in SSH protocol in ClickHouse. [#78586](https://github.com/ClickHouse/ClickHouse/pull/78586) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). -#### Experimental Feature +#### Experimental feature * Support correlated subqueries as an argument of `EXISTS` expression in the `WHERE` clause. Closes [#72459](https://github.com/ClickHouse/ClickHouse/issues/72459). [#76078](https://github.com/ClickHouse/ClickHouse/pull/76078) ([Dmitry Novik](https://github.com/novikd)). * Functions `sparseGrams` and `sparseGramsHashes` with ASCII and UTF8 versions added. Author: [scanhex12](https://github.com/scanhex12). [#78176](https://github.com/ClickHouse/ClickHouse/pull/78176) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). Do not use it: the implementation will change in the next versions. -#### Performance Improvement +#### Performance improvement * Optimize performance with lazy columns, that read the data after ORDER BY and LIMIT. [#55518](https://github.com/ClickHouse/ClickHouse/pull/55518) ([Xiaozhe Yu](https://github.com/wudidapaopao)). * Enabled the query condition cache by default. [#79080](https://github.com/ClickHouse/ClickHouse/pull/79080) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Speed-up building JOIN result by de-virtualizing calls to `col->insertFrom()`. [#77350](https://github.com/ClickHouse/ClickHouse/pull/77350) ([Alexander Gololobov](https://github.com/davenger)). @@ -1813,7 +1813,7 @@ doc_type: 'changelog' * Cleanup settings during `recoverLostReplica` same as it was done in: [#78637](https://github.com/ClickHouse/ClickHouse/pull/78637). [#79113](https://github.com/ClickHouse/ClickHouse/pull/79113) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Use insertion columns for INFILE schema inference. [#78490](https://github.com/ClickHouse/ClickHouse/pull/78490) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). -#### Bug Fix (user-visible misbehavior in an official stable release) +#### Bug fix (user-visible misbehavior in an official stable release) * Fix incorrect projection analysis when `count(Nullable)` is used in aggregate projections. This fixes [#74495](https://github.com/ClickHouse/ClickHouse/issues/74495) . This PR also adds some logs around projection analysis to clarify why a projection is used or why not. [#74498](https://github.com/ClickHouse/ClickHouse/pull/74498) ([Amos Bird](https://github.com/amosbird)). * Fix `Part <...> does not contain in snapshot of previous virtual parts. (PART_IS_TEMPORARILY_LOCKED)` during `DETACH PART`. [#76039](https://github.com/ClickHouse/ClickHouse/pull/76039) ([Aleksei Filatov](https://github.com/aalexfvk)). * Fix not working skip indexes with expression with literals in analyzer and remove trivial casts during indexes analysis. [#77229](https://github.com/ClickHouse/ClickHouse/pull/77229) ([Pavel Kruglov](https://github.com/Avogar)). @@ -1873,17 +1873,17 @@ doc_type: 'changelog' * Keeper fix: Avoid triggering watches on failed multi requests. [#79247](https://github.com/ClickHouse/ClickHouse/pull/79247) ([Antonio Andelic](https://github.com/antonio2368)). * Fix reading Iceberg table failed when min-max value is specified explicitly but is `NULL`. The Go Iceberg library was noted for generating such an atrocious files. Closes [#78740](https://github.com/ClickHouse/ClickHouse/issues/78740). [#78764](https://github.com/ClickHouse/ClickHouse/pull/78764) ([flynn](https://github.com/ucasfl)). -#### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging improvement * Respect CPU target features in Rust and enable LTO in all crates. [#78590](https://github.com/ClickHouse/ClickHouse/pull/78590) ([Raúl Marín](https://github.com/Algunenano)). -### ClickHouse release 25.3 LTS, 2025-03-20 {#253} +### ClickHouse release 25.3 lts, 2025-03-20 {#253} -#### Backward Incompatible Change +#### Backward incompatible change * Disallow truncating replicated databases. [#76651](https://github.com/ClickHouse/ClickHouse/pull/76651) ([Bharat Nallan](https://github.com/bharatnc)). * Skipping index cache is reverted. [#77447](https://github.com/ClickHouse/ClickHouse/pull/77447) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). -#### New Feature +#### New feature * `JSON` data type is production-ready. See https://jsonbench.com/. `Dynamic` and `Variant` data types are production-ready. [#77785](https://github.com/ClickHouse/ClickHouse/pull/77785) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Introduce the SSH protocol for clickhouse-server. Now, it is possible to connect to ClickHouse using any SSH client. This closes: [#74340](https://github.com/ClickHouse/ClickHouse/issues/74340). [#74989](https://github.com/ClickHouse/ClickHouse/pull/74989) ([George Gamezardashvili](https://github.com/Infjoker)). * Replace table functions with their -Cluster alternatives if parallel replicas are enabled. Fixes [#65024](https://github.com/ClickHouse/ClickHouse/issues/65024). [#70659](https://github.com/ClickHouse/ClickHouse/pull/70659) ([Konstantin Bogdanov](https://github.com/thevar1able)). @@ -1899,13 +1899,13 @@ doc_type: 'changelog' * Support for the histogram metric type. The interface closely mirrors the Prometheus client, where you simply call `observe(value)` to increment the counter in the bucket corresponding to the value. The histogram metrics are exposed via `system.histogram_metrics`. [#75736](https://github.com/ClickHouse/ClickHouse/pull/75736) ([Miсhael Stetsyuk](https://github.com/mstetsyuk)). * Non-constant CASE support for switching on explicit values. [#77399](https://github.com/ClickHouse/ClickHouse/pull/77399) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). -#### Experimental Feature +#### Experimental feature * Add support [for Unity Catalog](https://www.databricks.com/product/unity-catalog) for DeltaLake tables on top of AWS S3 and local filesystem. [#76988](https://github.com/ClickHouse/ClickHouse/pull/76988) ([alesapin](https://github.com/alesapin)). * Introduce experimental integration with AWS Glue service catalog for Iceberg tables. [#77257](https://github.com/ClickHouse/ClickHouse/pull/77257) ([alesapin](https://github.com/alesapin)). * Added support for dynamic cluster autodiscovery. This extends the existing _node_ autodiscovery feature. ClickHouse can now automatically detect and register new _clusters_ under a common ZooKeeper path using ``. [#76001](https://github.com/ClickHouse/ClickHouse/pull/76001) ([Anton Ivashkin](https://github.com/ianton-ru)). * Allow automatic cleanup merges of entire partitions after a configurable timeout with a new setting `enable_replacing_merge_with_cleanup_for_min_age_to_force_merge`. [#76440](https://github.com/ClickHouse/ClickHouse/pull/76440) ([Christoph Wurm](https://github.com/cwurm)). -#### Performance Improvement +#### Performance improvement * Implement query condition cache to improve query performance using repeated conditions. The range of the portion of data that does not meet the condition is remembered as a temporary index in memory. Subsequent queries will use this index. Close [#67768](https://github.com/ClickHouse/ClickHouse/issues/67768) [#69236](https://github.com/ClickHouse/ClickHouse/pull/69236) ([zhongyuankai](https://github.com/zhongyuankai)). * Actively evict data from the cache on parts removal. Do not let the cache grow to the maximum size if the amount of data is less. [#76641](https://github.com/ClickHouse/ClickHouse/pull/76641) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Replace Int256 and UInt256 with clang builtin i256 in arithmetic calculation, and it gives a performance improvement [#70502](https://github.com/ClickHouse/ClickHouse/issues/70502). [#73658](https://github.com/ClickHouse/ClickHouse/pull/73658) ([李扬](https://github.com/taiyang-li)). @@ -1943,7 +1943,7 @@ doc_type: 'changelog' * Backup/restore setting `allow_s3_native_copy` now supports value three possible values: - `False` - s3 native copy will not be used; - `True` (old default) - ClickHouse will try s3 native copy first, if it fails then fallback to the reading+writing approach; - `'auto'` (new default) - ClickHouse will compare the source and destination credentials first. If they are same, ClickHouse will try s3 native copy and then may fallback to the reading+writing approach. If they are different, ClickHouse will go directly to the reading+writing approach. [#77401](https://github.com/ClickHouse/ClickHouse/pull/77401) ([Vitaly Baranov](https://github.com/vitlibar)). * Support aws session token and environment credentials usage in delta kernel for DeltaLake table engine. [#77661](https://github.com/ClickHouse/ClickHouse/pull/77661) ([Kseniia Sumarokova](https://github.com/kssenii)). -#### Bug Fix (user-visible misbehavior in an official stable release) +#### Bug fix (user-visible misbehavior in an official stable release) * Fix stuck while processing pending batch for async distributed INSERT (due to i.e. `No such file or directory`). [#72939](https://github.com/ClickHouse/ClickHouse/pull/72939) ([Azat Khuzhin](https://github.com/azat)). * Improved datetime conversion during index analysis by enforcing saturating behavior for implicit Date to DateTime conversions. This resolves potential index analysis inaccuracies caused by datetime range limitations. This fixes [#73307](https://github.com/ClickHouse/ClickHouse/issues/73307). It also fixes explicit `toDateTime` conversion when `date_time_overflow_behavior = 'ignore'` which is the default value. [#73326](https://github.com/ClickHouse/ClickHouse/pull/73326) ([Amos Bird](https://github.com/amosbird)). * Fix all sort of bugs due to race between UUID and table names (for instance it will fix the race between `RENAME` and `RESTART REPLICA`, in case of concurrent `RENAME` with `SYSTEM RESTART REPLICA` you may get end up restarting wrong replica, or/and leaving one of the tables in a `Table X is being restarted` state). [#76308](https://github.com/ClickHouse/ClickHouse/pull/76308) ([Azat Khuzhin](https://github.com/azat)). @@ -1993,14 +1993,14 @@ doc_type: 'changelog' * Second arguments of type `Nullable` for function `arrayResize` are now disallowed. Previously, anything from errors to wrong results could happen with `Nullable` as second argument. (issue [#48398](https://github.com/ClickHouse/ClickHouse/issues/48398)). [#77724](https://github.com/ClickHouse/ClickHouse/pull/77724) ([Manish Gill](https://github.com/mgill25)). * Regularly check if merges and mutations were cancelled even in case when the operation doesn't produce any blocks to write. [#77766](https://github.com/ClickHouse/ClickHouse/pull/77766) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). -#### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging improvement * `clickhouse-odbc-bridge` and `clickhouse-library-bridge` are moved to a separate repository, https://github.com/ClickHouse/odbc-bridge/. [#76225](https://github.com/ClickHouse/ClickHouse/pull/76225) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Fix Rust cross-compilation and allow disabling Rust completely. [#76921](https://github.com/ClickHouse/ClickHouse/pull/76921) ([Raúl Marín](https://github.com/Algunenano)). ### ClickHouse release 25.2, 2025-02-27 {#252} -#### Backward Incompatible Change +#### Backward incompatible change * Completely enable `async_load_databases` by default (even for those installations that do not upgrade `config.xml`). [#74772](https://github.com/ClickHouse/ClickHouse/pull/74772) ([Azat Khuzhin](https://github.com/azat)). * Add `JSONCompactEachRowWithProgress` and `JSONCompactStringsEachRowWithProgress` formats. Continuation of [#69989](https://github.com/ClickHouse/ClickHouse/issues/69989). The `JSONCompactWithNames` and `JSONCompactWithNamesAndTypes` no longer output "totals" - apparently, it was a mistake in the implementation. [#75037](https://github.com/ClickHouse/ClickHouse/pull/75037) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Change the `format_alter_operations_with_parentheses` default to true to disambiguate the ALTER commands list (see https://github.com/ClickHouse/ClickHouse/pull/59532). This breaks replication with clusters prior to 24.3. If you are upgrading a cluster using older releases turn off the setting in the server config or upgrade to 24.3 first. [#75302](https://github.com/ClickHouse/ClickHouse/pull/75302) ([Raúl Marín](https://github.com/Algunenano)). @@ -2008,7 +2008,7 @@ doc_type: 'changelog' * The setting `min_chunk_bytes_for_parallel_parsing` cannot be zero anymore. This fixes: [#71110](https://github.com/ClickHouse/ClickHouse/issues/71110). [#75239](https://github.com/ClickHouse/ClickHouse/pull/75239) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Validate settings in the cache configuration. Non existing settings were ignored before, now they will throw an error and they should be removed. [#75452](https://github.com/ClickHouse/ClickHouse/pull/75452) ([Kseniia Sumarokova](https://github.com/kssenii)). -#### New Feature +#### New feature * Support type `Nullable(JSON)`. [#73556](https://github.com/ClickHouse/ClickHouse/pull/73556) ([Pavel Kruglov](https://github.com/Avogar)). * Support subcolumns in the DEFAULT and MATERIALIZED expressions. [#74403](https://github.com/ClickHouse/ClickHouse/pull/74403) ([Pavel Kruglov](https://github.com/Avogar)). * Support writing Parquet bloom filters using the `output_format_parquet_write_bloom_filter` setting (enabled by default). [#71681](https://github.com/ClickHouse/ClickHouse/pull/71681) ([Michael Kolupaev](https://github.com/al13n321)). @@ -2021,13 +2021,13 @@ doc_type: 'changelog' * A new function, `initialQueryStartTime` is added. It returns the start time of the current query. The value is the same across all shards during a distributed query. [#75087](https://github.com/ClickHouse/ClickHouse/pull/75087) ([Roman Lomonosov](https://github.com/lomik)). * Add support for SSL authentication with named collections for MySQL. Closes [#59111](https://github.com/ClickHouse/ClickHouse/issues/59111). [#59452](https://github.com/ClickHouse/ClickHouse/pull/59452) ([Nikolay Degterinsky](https://github.com/evillique)). -#### Experimental Features +#### Experimental features * Added a new setting, `enable_adaptive_memory_spill_scheduler` that allows multiple Grace JOINs in the same query to monitor their combined memory footprint and trigger spilling into external storage adaptively to prevent MEMORY_LIMIT_EXCEEDED. [#72728](https://github.com/ClickHouse/ClickHouse/pull/72728) ([lgbo](https://github.com/lgbo-ustc)). * Make the new, experimental `Kafka` table engine fully respect Keeper feature flags. [#76004](https://github.com/ClickHouse/ClickHouse/pull/76004) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). * Restore the (Intel) QPL codec, which was removed in v24.10 due to licensing issues. [#76021](https://github.com/ClickHouse/ClickHouse/pull/76021) ([Konstantin Bogdanov](https://github.com/thevar1able)). * For the integration with HDFS added support for the `dfs.client.use.datanode.hostname` configuration option. [#74635](https://github.com/ClickHouse/ClickHouse/pull/74635) ([Mikhail Tiukavkin](https://github.com/freshertm)). -#### Performance Improvement +#### Performance improvement * Improve performance of the whole JSON column reading in Wide parts from S3. It's done by adding prefetches for subcolumn prefixes deserialization, cache of deserialized prefixes, and parallel deserialization of subcolumn prefixes. It improves the reading of the JSON column from S3 4 times in queries like `SELECT data FROM table` and about 10 times in queries like `SELECT data FROM table LIMIT 10`. [#74827](https://github.com/ClickHouse/ClickHouse/pull/74827) ([Pavel Kruglov](https://github.com/Avogar)). * Fixed unnecessary contention in `parallel_hash` when `max_rows_in_join = max_bytes_in_join = 0`. [#75155](https://github.com/ClickHouse/ClickHouse/pull/75155) ([Nikita Taranov](https://github.com/nickitat)). * Fixed double preallocation in `ConcurrentHashJoin` in case join sides are swapped by the optimizer. [#75149](https://github.com/ClickHouse/ClickHouse/pull/75149) ([Nikita Taranov](https://github.com/nickitat)). @@ -2080,7 +2080,7 @@ doc_type: 'changelog' * Add the `CPUOverload` asynchronous metric, which calculates the server's relative CPU deficit. [#76404](https://github.com/ClickHouse/ClickHouse/pull/76404) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Changed the default value of `output_format_pretty_max_rows` from 10000 to 1000. I think it is better for usability. [#76407](https://github.com/ClickHouse/ClickHouse/pull/76407) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### Bug Fix (user-visible misbehavior in an official stable release) +#### Bug fix (user-visible misbehavior in an official stable release) * Fix formatting of exceptions using a custom format if they appear during query interpretation. In previous versions, exceptions were formatted using the default format rather than the format specified in the query. This closes [#55422](https://github.com/ClickHouse/ClickHouse/issues/55422). [#74994](https://github.com/ClickHouse/ClickHouse/pull/74994) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Fix type mapping for SQLite (integer types into `int64`, floating points into `float64`). [#73853](https://github.com/ClickHouse/ClickHouse/pull/73853) ([Joanna Hulboj](https://github.com/jh0x)). * Fix identifier resolution from parent scopes. Allow the use of aliases to expressions in the WITH clause. Fixes [#58994](https://github.com/ClickHouse/ClickHouse/issues/58994). Fixes [#62946](https://github.com/ClickHouse/ClickHouse/issues/62946). Fixes [#63239](https://github.com/ClickHouse/ClickHouse/issues/63239). Fixes [#65233](https://github.com/ClickHouse/ClickHouse/issues/65233). Fixes [#71659](https://github.com/ClickHouse/ClickHouse/issues/71659). Fixes [#71828](https://github.com/ClickHouse/ClickHouse/issues/71828). Fixes [#68749](https://github.com/ClickHouse/ClickHouse/issues/68749). [#66143](https://github.com/ClickHouse/ClickHouse/pull/66143) ([Dmitry Novik](https://github.com/novikd)). @@ -2154,7 +2154,7 @@ doc_type: 'changelog' * Fix potentially unsafe call in signal handler. [#76549](https://github.com/ClickHouse/ClickHouse/pull/76549) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). * Fix reverse key support in PartsSplitter. This fixes [#73400](https://github.com/ClickHouse/ClickHouse/issues/73400). [#73418](https://github.com/ClickHouse/ClickHouse/pull/73418) ([Amos Bird](https://github.com/amosbird)). -#### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging improvement * Support build HDFS on both ARM and Intel Mac. [#74244](https://github.com/ClickHouse/ClickHouse/pull/74244) ([Yan Xin](https://github.com/yxheartipp)). * Enable ICU and GRPC when cross-compiling for Darwin. [#75922](https://github.com/ClickHouse/ClickHouse/pull/75922) ([Raúl Marín](https://github.com/Algunenano)). * Update to embedded LLVM 19. [#75148](https://github.com/ClickHouse/ClickHouse/pull/75148) ([Konstantin Bogdanov](https://github.com/thevar1able)). @@ -2163,7 +2163,7 @@ doc_type: 'changelog' ### ClickHouse release 25.1, 2025-01-28 {#251} -#### Backward Incompatible Change +#### Backward incompatible change * `JSONEachRowWithProgress` will write the progress whenever the progress happens. In previous versions, the progress was shown only after each block of the result, which made it useless. Change the way how the progress is displayed: it will not show zero values. This closes [#70800](https://github.com/ClickHouse/ClickHouse/issues/70800). [#73834](https://github.com/ClickHouse/ClickHouse/pull/73834) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * `Merge` tables will unify the structure of underlying tables by using a union of their columns and deriving common types. This closes [#64864](https://github.com/ClickHouse/ClickHouse/issues/64864). In certain cases, this change could be backward incompatible. One example is when there is no common type between tables, but conversion to the type of the first table is still possible, such as in the case of UInt64 and Int64 or any numeric type and String. If you want to return to the old behavior, set `merge_table_max_tables_to_look_for_schema_inference` to `1` or set `compatibility` to `24.12` or earlier. [#73956](https://github.com/ClickHouse/ClickHouse/pull/73956) ([Alexey Milovidov](https://github.com/alexey-milovidov)). * Parquet output format converts Date and DateTime columns to date/time types supported by Parquet, instead of writing them as raw numbers. `DateTime` becomes `DateTime64(3)` (was: `UInt32`); setting `output_format_parquet_datetime_as_uint32` brings back the old behavior. `Date` becomes `Date32` (was: `UInt16`). [#70950](https://github.com/ClickHouse/ClickHouse/pull/70950) ([Michael Kolupaev](https://github.com/al13n321)). @@ -2174,7 +2174,7 @@ doc_type: 'changelog' * Function `h3ToGeo()` now returns the results in the order `(lat, lon)` (which is the standard order for geometric functions). Users who wish to retain the legacy result order `(lon, lat)` can set setting `h3togeo_lon_lat_result_order = true`. [#74719](https://github.com/ClickHouse/ClickHouse/pull/74719) ([Manish Gill](https://github.com/mgill25)). * A new MongoDB driver is now the default. Users who like to continue using the legacy driver can set server setting `use_legacy_mongodb_integration` to true. [#73359](https://github.com/ClickHouse/ClickHouse/pull/73359) ([Robert Schulze](https://github.com/rschu1ze)). -#### New Feature +#### New feature * Added an ability to apply non-finished (not materialized by background process) mutations during the execution of `SELECT` queries immediately after submitting. It can be enabled by setting `apply_mutations_on_fly`. [#74877](https://github.com/ClickHouse/ClickHouse/pull/74877) ([Anton Popov](https://github.com/CurtizJ)). * Implement `Iceberg` tables partition pruning for time-related transform partition operations in Iceberg. [#72044](https://github.com/ClickHouse/ClickHouse/pull/72044) ([Daniil Ivanik](https://github.com/divanik)). * Support subcolumns in MergeTree sorting key and skip indexes. [#72644](https://github.com/ClickHouse/ClickHouse/pull/72644) ([Pavel Kruglov](https://github.com/Avogar)). @@ -2190,10 +2190,10 @@ doc_type: 'changelog' * Added function `arrayNormalizedGini`. [#72823](https://github.com/ClickHouse/ClickHouse/pull/72823) ([flynn](https://github.com/ucasfl)). * Add minus operator support for `DateTime64`, to allow subtraction between `DateTime64` values, as well as `DateTime`. [#74482](https://github.com/ClickHouse/ClickHouse/pull/74482) ([Li Yin](https://github.com/liyinsg)). -#### Experimental Features +#### Experimental features * The `BFloat16` data type is production-ready. [#73840](https://github.com/ClickHouse/ClickHouse/pull/73840) ([Alexey Milovidov](https://github.com/alexey-milovidov)). -#### Performance Improvement +#### Performance improvement * Optimized function `indexHint`. Now, columns that are only used as arguments of function `indexHint` are not read from the table. [#74314](https://github.com/ClickHouse/ClickHouse/pull/74314) ([Anton Popov](https://github.com/CurtizJ)). If the `indexHint` function is a central piece of your enterprise data architecture, this optimization will save your life. * More accurate accounting for `max_joined_block_size_rows` setting for `parallel_hash` JOIN algorithm. Helps to avoid increased memory consumption compared to `hash` algorithm. [#74630](https://github.com/ClickHouse/ClickHouse/pull/74630) ([Nikita Taranov](https://github.com/nickitat)). * Support predicate push down optimization on the query plan level for the `MergingAggregated` step. It improves performance for some queries with the analyzer. [#74073](https://github.com/ClickHouse/ClickHouse/pull/74073) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -2311,7 +2311,7 @@ doc_type: 'changelog' * Now users can be specified inside the startup scripts. [#74894](https://github.com/ClickHouse/ClickHouse/pull/74894) ([pufit](https://github.com/pufit)). * Add support for Azure SAS Tokens. [#72959](https://github.com/ClickHouse/ClickHouse/pull/72959) ([Azat Khuzhin](https://github.com/azat)). -#### Bug Fix (user-visible misbehavior in an official stable release) +#### Bug fix (user-visible misbehavior in an official stable release) * Set parquet compression level only if compression codec supports it. [#74659](https://github.com/ClickHouse/ClickHouse/pull/74659) ([Arthur Passos](https://github.com/arthurpassos)). * Fixed a regression that using collation locales with modifiers throws an error. As an example, `SELECT arrayJoin(['kk 50', 'KK 01', ' KK 2', ' KK 3', 'kk 1', 'x9y99', 'x9y100']) item ORDER BY item ASC COLLATE 'tr-u-kn-true-ka-shifted` now works. [#73544](https://github.com/ClickHouse/ClickHouse/pull/73544) ([Robert Schulze](https://github.com/rschu1ze)). * Fix cannot create SEQUENTIAL node with keeper-client. [#64177](https://github.com/ClickHouse/ClickHouse/pull/64177) ([Duc Canh Le](https://github.com/canhld94)). @@ -2390,5 +2390,5 @@ doc_type: 'changelog' * Fix FILE_DOESNT_EXIST error occurring during data parts merge for a table with an empty column in Azure Blob Storage. [#74892](https://github.com/ClickHouse/ClickHouse/pull/74892) ([Julia Kartseva](https://github.com/jkartseva)). * Fix projection column name when joining temporary tables, close [#68872](https://github.com/ClickHouse/ClickHouse/issues/68872). [#74897](https://github.com/ClickHouse/ClickHouse/pull/74897) ([Vladimir Cherkasov](https://github.com/vdimir)). -#### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging improvement * The universal installation script will propose installation even on macOS. [#74339](https://github.com/ClickHouse/ClickHouse/pull/74339) ([Alexey Milovidov](https://github.com/alexey-milovidov)). diff --git a/docs/whats-new/security-changelog.md b/docs/whats-new/security-changelog.md index 69bf6f1867b..1de88cbdcf1 100644 --- a/docs/whats-new/security-changelog.md +++ b/docs/whats-new/security-changelog.md @@ -80,7 +80,7 @@ Credits: malacupa (Independent researcher) ## Fixed in ClickHouse 22.9.1.2603, 2022-09-22 {#fixed-in-clickhouse-release-22-9-1-2603-2022-9-22} -### CVE-2022-44011 {#CVE-2022-44011} +### Cve-2022-44011 {#CVE-2022-44011} A heap buffer overflow issue was discovered in ClickHouse server. A malicious user with ability to load data into ClickHouse server could crash the ClickHouse server by inserting a malformed CapnProto object. @@ -88,7 +88,7 @@ Fix has been pushed to version 22.9.1.2603, 22.8.2.11, 22.7.4.16, 22.6.6.16, 22. Credits: Kiojj (independent researcher) -### CVE-2022-44010 {#CVE-2022-44010} +### Cve-2022-44010 {#CVE-2022-44010} A heap buffer overflow issue was discovered in ClickHouse server. An attacker could send a specially crafted HTTP request to the HTTP Endpoint (listening on port 8123 by default), causing a heap-based buffer overflow that crashes the ClickHouse server process. This attack does not require authentication. @@ -98,43 +98,43 @@ Credits: Kiojj (independent researcher) ## Fixed in ClickHouse 21.10.2.15, 2021-10-18 {#fixed-in-clickhouse-release-21-10-2-215-2021-10-18} -### CVE-2021-43304 {#cve-2021-43304} +### Cve-2021-43304 {#cve-2021-43304} Heap buffer overflow in ClickHouse's LZ4 compression codec when parsing a malicious query. There is no verification that the copy operations in the LZ4::decompressImpl loop and especially the arbitrary copy operation `wildCopy(op, ip, copy_end)`, don't exceed the destination buffer's limits. Credits: JFrog Security Research Team -### CVE-2021-43305 {#cve-2021-43305} +### Cve-2021-43305 {#cve-2021-43305} Heap buffer overflow in ClickHouse's LZ4 compression codec when parsing a malicious query. There is no verification that the copy operations in the LZ4::decompressImpl loop and especially the arbitrary copy operation `wildCopy(op, ip, copy_end)`, don't exceed the destination buffer's limits. This issue is very similar to CVE-2021-43304, but the vulnerable copy operation is in a different wildCopy call. Credits: JFrog Security Research Team -### CVE-2021-42387 {#cve-2021-42387} +### Cve-2021-42387 {#cve-2021-42387} Heap out-of-bounds read in ClickHouse's LZ4 compression codec when parsing a malicious query. As part of the LZ4::decompressImpl() loop, a 16-bit unsigned user-supplied value ('offset') is read from the compressed data. The offset is later used in the length of a copy operation, without checking the upper bounds of the source of the copy operation. Credits: JFrog Security Research Team -### CVE-2021-42388 {#cve-2021-42388} +### Cve-2021-42388 {#cve-2021-42388} Heap out-of-bounds read in ClickHouse's LZ4 compression codec when parsing a malicious query. As part of the LZ4::decompressImpl() loop, a 16-bit unsigned user-supplied value ('offset') is read from the compressed data. The offset is later used in the length of a copy operation, without checking the lower bounds of the source of the copy operation. Credits: JFrog Security Research Team -### CVE-2021-42389 {#cve-2021-42389} +### Cve-2021-42389 {#cve-2021-42389} Divide-by-zero in ClickHouse's Delta compression codec when parsing a malicious query. The first byte of the compressed buffer is used in a modulo operation without being checked for 0. Credits: JFrog Security Research Team -### CVE-2021-42390 {#cve-2021-42390} +### Cve-2021-42390 {#cve-2021-42390} Divide-by-zero in ClickHouse's DeltaDouble compression codec when parsing a malicious query. The first byte of the compressed buffer is used in a modulo operation without being checked for 0. Credits: JFrog Security Research Team -### CVE-2021-42391 {#cve-2021-42391} +### Cve-2021-42391 {#cve-2021-42391} Divide-by-zero in ClickHouse's Gorilla compression codec when parsing a malicious query. The first byte of the compressed buffer is used in a modulo operation without being checked for 0. @@ -142,7 +142,7 @@ Credits: JFrog Security Research Team ## Fixed in ClickHouse 21.4.3.21, 2021-04-12 {#fixed-in-clickhouse-release-21-4-3-21-2021-04-12} -### CVE-2021-25263 {#cve-2021-25263} +### Cve-2021-25263 {#cve-2021-25263} An attacker that has CREATE DICTIONARY privilege, can read arbitary file outside permitted directory. @@ -150,69 +150,69 @@ Fix has been pushed to versions 20.8.18.32-lts, 21.1.9.41-stable, 21.2.9.41-stab Credits: [Vyacheslav Egoshin](https://twitter.com/vegoshin) -## Fixed in ClickHouse Release 19.14.3.3, 2019-09-10 {#fixed-in-clickhouse-release-19-14-3-3-2019-09-10} +## Fixed in ClickHouse release 19.14.3.3, 2019-09-10 {#fixed-in-clickhouse-release-19-14-3-3-2019-09-10} -### CVE-2019-15024 {#cve-2019-15024} +### Cve-2019-15024 {#cve-2019-15024} Аn attacker that has write access to ZooKeeper and who can run a custom server available from the network where ClickHouse runs, can create a custom-built malicious server that will act as a ClickHouse replica and register it in ZooKeeper. When another replica will fetch data part from the malicious replica, it can force clickhouse-server to write to arbitrary path on filesystem. Credits: Eldar Zaitov of Yandex Information Security Team -### CVE-2019-16535 {#cve-2019-16535} +### Cve-2019-16535 {#cve-2019-16535} Аn OOB read, OOB write and integer underflow in decompression algorithms can be used to achieve RCE or DoS via native protocol. Credits: Eldar Zaitov of Yandex Information Security Team -### CVE-2019-16536 {#cve-2019-16536} +### Cve-2019-16536 {#cve-2019-16536} Stack overflow leading to DoS can be triggered by a malicious authenticated client. Credits: Eldar Zaitov of Yandex Information Security Team -## Fixed in ClickHouse Release 19.13.6.1, 2019-09-20 {#fixed-in-clickhouse-release-19-13-6-1-2019-09-20} +## Fixed in ClickHouse release 19.13.6.1, 2019-09-20 {#fixed-in-clickhouse-release-19-13-6-1-2019-09-20} -### CVE-2019-18657 {#cve-2019-18657} +### Cve-2019-18657 {#cve-2019-18657} Table function `url` had the vulnerability allowed the attacker to inject arbitrary HTTP headers in the request. Credits: [Nikita Tikhomirov](https://github.com/NSTikhomirov) -## Fixed in ClickHouse Release 18.12.13, 2018-09-10 {#fixed-in-clickhouse-release-18-12-13-2018-09-10} +## Fixed in ClickHouse release 18.12.13, 2018-09-10 {#fixed-in-clickhouse-release-18-12-13-2018-09-10} -### CVE-2018-14672 {#cve-2018-14672} +### Cve-2018-14672 {#cve-2018-14672} Functions for loading CatBoost models allowed path traversal and reading arbitrary files through error messages. Credits: Andrey Krasichkov of Yandex Information Security Team -## Fixed in ClickHouse Release 18.10.3, 2018-08-13 {#fixed-in-clickhouse-release-18-10-3-2018-08-13} +## Fixed in ClickHouse release 18.10.3, 2018-08-13 {#fixed-in-clickhouse-release-18-10-3-2018-08-13} -### CVE-2018-14671 {#cve-2018-14671} +### Cve-2018-14671 {#cve-2018-14671} unixODBC allowed loading arbitrary shared objects from the file system which led to a Remote Code Execution vulnerability. Credits: Andrey Krasichkov and Evgeny Sidorov of Yandex Information Security Team -## Fixed in ClickHouse Release 1.1.54388, 2018-06-28 {#fixed-in-clickhouse-release-1-1-54388-2018-06-28} +## Fixed in ClickHouse release 1.1.54388, 2018-06-28 {#fixed-in-clickhouse-release-1-1-54388-2018-06-28} -### CVE-2018-14668 {#cve-2018-14668} +### Cve-2018-14668 {#cve-2018-14668} "remote" table function allowed arbitrary symbols in "user", "password" and "default_database" fields which led to Cross Protocol Request Forgery Attacks. Credits: Andrey Krasichkov of Yandex Information Security Team -## Fixed in ClickHouse Release 1.1.54390, 2018-07-06 {#fixed-in-clickhouse-release-1-1-54390-2018-07-06} +## Fixed in ClickHouse release 1.1.54390, 2018-07-06 {#fixed-in-clickhouse-release-1-1-54390-2018-07-06} -### CVE-2018-14669 {#cve-2018-14669} +### Cve-2018-14669 {#cve-2018-14669} ClickHouse MySQL client had "LOAD DATA LOCAL INFILE" functionality enabled that allowed a malicious MySQL database read arbitrary files from the connected ClickHouse server. Credits: Andrey Krasichkov and Evgeny Sidorov of Yandex Information Security Team -## Fixed in ClickHouse Release 1.1.54131, 2017-01-10 {#fixed-in-clickhouse-release-1-1-54131-2017-01-10} +## Fixed in ClickHouse release 1.1.54131, 2017-01-10 {#fixed-in-clickhouse-release-1-1-54131-2017-01-10} -### CVE-2018-14670 {#cve-2018-14670} +### Cve-2018-14670 {#cve-2018-14670} Incorrect configuration in deb package could lead to the unauthorized use of the database.