Skip to content

Commit 40a9f35

Browse files
BlargianBlargian
authored
Merge branch 'main' of https://github.com/ClickHouse/clickhouse-docs into ip-filter-update
2 parents c8ab4d8 + 36458bf commit 40a9f35

File tree

13 files changed

+356
-47
lines changed

13 files changed

+356
-47
lines changed

‎docs/cloud/reference/changelog.md

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,14 @@ In addition to this ClickHouse Cloud changelog, please see the [Cloud Compatibil
3333

3434
## May 30, 2025 {#may-30-2025}
3535

36+
- We’re excited to announce general availability of **ClickPipes for Postgres CDC **
37+
in ClickHouse Cloud. With just a few clicks, you can now replicate your Postgres
38+
databases and unlock blazing-fast, real-time analytics. The connector delivers
39+
faster data synchronization, latency as low as a few seconds, automatic schema changes,
40+
fully secure connectivity, and more. Refer to the
41+
[blog](https://clickhouse.com/blog/postgres-cdc-connector-clickpipes-ga) for
42+
more information. To get started, refer to the instructions [here](https://clickhouse.com/docs/integrations/clickpipes/postgres).
43+
3644
- Introduced new improvements to the SQL console dashboards:
3745
- Sharing: You can share your dashboard with your team members. Four levels of access are supported, that can be adjusted both globally and on a per-user basis:
3846
- _Write access_: Add/edit visualizations, refresh settings, interact with dashboards via filters.
@@ -44,13 +52,7 @@ In addition to this ClickHouse Cloud changelog, please see the [Cloud Compatibil
4452

4553
<Image img={dashboards} size="md" alt="Dashboards improvements" border />
4654

47-
- We’re excited to announce general availability of **ClickPipes for Postgres CDC GA**
48-
in ClickHouse Cloud. With just a few clicks, you can now replicate your Postgres
49-
databases and unlock blazing-fast, real-time analytics. The connector delivers
50-
faster data synchronization, latency as low as a few seconds, automatic schema changes,
51-
fully secure connectivity, and more. Refer to the
52-
[blog](https://clickhouse.com/blog/postgres-cdc-connector-clickpipes-ga) for
53-
more information. To get started, refer to the instructions [here](https://clickhouse.com/docs/integrations/clickpipes/postgres).
55+
5456
- We are enlisting preview participants for [Distributed cache](https://clickhouse.com/cloud/distributed-cache-waitlist)
5557
for AWS and GCP. Read more in the [blog](https://clickhouse.com/blog/building-a-distributed-cache-for-s3).
5658

‎docs/integrations/data-ingestion/clickpipes/mysql/index.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,11 @@ To get started, you first need to make sure that your MySQL database is set up c
3636

3737
3. [Cloud SQL for MySQL](./mysql/source/gcp)
3838

39-
4. [Amazon RDS MariaDB](./mysql/source/rds_maria)
39+
4. [Generic MySQL](./mysql/source/generic)
40+
41+
5. [Amazon RDS MariaDB](./mysql/source/rds_maria)
42+
43+
6. [Generic MariaDB](./mysql/source/generic_maria)
4044

4145
Once your source MySQL database is set up, you can continue creating your ClickPipe.
4246

‎docs/integrations/data-ingestion/clickpipes/mysql/source/generic.md

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
---
2+
sidebar_label: 'Generic MySQL'
3+
description: 'Set up any MySQL instance as a source for ClickPipes'
4+
slug: /integrations/clickpipes/mysql/source/generic
5+
title: 'Generic MySQL source setup guide'
6+
---
7+
8+
# Generic MySQL source setup guide
9+
10+
:::info
11+
12+
If you use one of the supported providers (in the sidebar), please refer to the specific guide for that provider.
13+
14+
:::
15+
16+
## Enable binary log retention {#enable-binlog-retention}
17+
18+
Binary logs contain information about data modifications made to a MySQL server instance and are required for replication.
19+
20+
### MySQL 8.x and newer {#binlog-v8-x}
21+
22+
To enable binary logging on your MySQL instance, ensure that the following settings are configured:
23+
24+
```sql
25+
log_bin = ON -- default value
26+
binlog_format = ROW -- default value
27+
binlog_row_image = FULL -- default value
28+
binlog_row_metadata = FULL
29+
binlog_expire_logs_seconds = 86400 -- 1 day or higher; default is 30 days
30+
```
31+
32+
To check these settings, run the following SQL commands:
33+
```sql
34+
SHOW VARIABLES LIKE 'log_bin';
35+
SHOW VARIABLES LIKE 'binlog_format';
36+
SHOW VARIABLES LIKE 'binlog_row_image';
37+
SHOW VARIABLES LIKE 'binlog_row_metadata';
38+
SHOW VARIABLES LIKE 'binlog_expire_logs_seconds';
39+
```
40+
41+
If the values don't match, you can run the following SQL commands to set them:
42+
```sql
43+
SET PERSIST log_bin = ON;
44+
SET PERSIST binlog_format = ROW;
45+
SET PERSIST binlog_row_image = FULL;
46+
SET PERSIST binlog_row_metadata = FULL;
47+
SET PERSIST binlog_expire_logs_seconds = 86400;
48+
```
49+
50+
If you have changed the `log_bin` setting, you NEED to RESTART the MySQL instance for the changes to take effect.
51+
52+
After changing the settings, continue on with [configuring a database user](#configure-database-user).
53+
54+
### MySQL 5.7 {#binlog-v5-x}
55+
56+
To enable binary logging on your MySQL 5.7 instance, ensure that the following settings are configured:
57+
58+
```sql
59+
server_id = 1 -- or greater; anything but 0
60+
log_bin = ON
61+
binlog_format = ROW -- default value
62+
binlog_row_image = FULL -- default value
63+
expire_logs_days = 1 -- or higher; 0 would mean logs are preserved forever
64+
```
65+
66+
To check these settings, run the following SQL commands:
67+
```sql
68+
SHOW VARIABLES LIKE 'server_id';
69+
SHOW VARIABLES LIKE 'log_bin';
70+
SHOW VARIABLES LIKE 'binlog_format';
71+
SHOW VARIABLES LIKE 'binlog_row_image';
72+
SHOW VARIABLES LIKE 'expire_logs_days';
73+
```
74+
75+
If the values don't match, you can set them in the config file (typically at `/etc/my.cnf` or `/etc/mysql/my.cnf`):
76+
```ini
77+
[mysqld]
78+
server_id = 1
79+
log_bin = ON
80+
binlog_format = ROW
81+
binlog_row_image = FULL
82+
expire_logs_days = 1
83+
```
84+
85+
You NEED to RESTART the MySQL instance for the changes to take effect.
86+
87+
:::note
88+
89+
Column exclusion is not supported for MySQL 5.7 because the `binlog_row_metadata` setting wasn't yet introduced.
90+
91+
:::
92+
93+
## Configure a database user {#configure-database-user}
94+
95+
Connect to your MySQL instance as the root user and execute the following commands:
96+
97+
1. Create a dedicated user for ClickPipes:
98+
99+
```sql
100+
CREATE USER 'clickpipes_user'@'%' IDENTIFIED BY 'some_secure_password';
101+
```
102+
103+
2. Grant schema permissions. The following example shows permissions for the `clickpipes` database. Repeat these commands for each database and host you want to replicate:
104+
105+
```sql
106+
GRANT SELECT ON `clickpipes`.* TO 'clickpipes_user'@'%';
107+
```
108+
109+
3. Grant replication permissions to the user:
110+
111+
```sql
112+
GRANT REPLICATION CLIENT ON *.* TO 'clickpipes_user'@'%';
113+
GRANT REPLICATION SLAVE ON *.* TO 'clickpipes_user'@'%';
114+
```
115+
116+
:::note
117+
118+
Make sure to replace `clickpipes_user` and `some_secure_password` with your desired username and password.
119+
120+
:::
121+
122+
## SSL/TLS configuration (recommended) {#ssl-tls-configuration}
123+
124+
SSL certificates ensure secure connections to your MySQL database. Configuration depends on your certificate type:
125+
126+
**Trusted Certificate Authority (DigiCert, Let's Encrypt, etc.)** - no additional configuration needed.
127+
128+
**Internal Certificate Authority** - Obtain the root CA certificate file from your IT team. In the ClickPipes UI, upload it when creating a new MySQL ClickPipe.
129+
130+
**Self-hosted MySQL** - Copy the CA certificate from your MySQL server (typically at `/var/lib/mysql/ca.pem`) and upload it in the UI when creating a new MySQL ClickPipe. Use the IP address of the server as the host.
131+
132+
**Self-hosted MySQL without server access** - Contact your IT team for the certificate. As a last resort, use the "Skip Certificate Verification" toggle in ClickPipes UI (not recommended for security reasons).
133+
134+
For more information on SSL/TLS options, check out our [FAQ](https://clickhouse.com/docs/integrations/clickpipes/mysql/faq#tls-certificate-validation-error).
135+
136+
## What's next? {#whats-next}
137+
138+
You can now [create your ClickPipe](../index.md) and start ingesting data from your MySQL instance into ClickHouse Cloud.
139+
Make sure to note down the connection details you used while setting up your MySQL instance as you will need them during the ClickPipe creation process.

‎docs/integrations/data-ingestion/clickpipes/mysql/source/generic_maria.md

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
---
2+
sidebar_label: 'Generic MariaDB'
3+
description: 'Set up any MariaDB instance as a source for ClickPipes'
4+
slug: /integrations/clickpipes/mysql/source/generic_maria
5+
title: 'Generic MariaDB source setup guide'
6+
---
7+
8+
# Generic MariaDB source setup guide
9+
10+
:::info
11+
12+
If you use one of the supported providers (in the sidebar), please refer to the specific guide for that provider.
13+
14+
:::
15+
16+
## Enable binary log retention {#enable-binlog-retention}
17+
18+
Binary logs contain information about data modifications made to a MariaDB server instance and are required for replication.
19+
20+
To enable binary logging on your MariaDB instance, ensure that the following settings are configured:
21+
22+
```sql
23+
server_id = 1 -- or greater; anything but 0
24+
log_bin = ON
25+
binlog_format = ROW
26+
binlog_row_image = FULL
27+
binlog_row_metadata = FULL -- introduced in 10.5.0
28+
expire_logs_days = 1 -- or higher; 0 would mean logs are preserved forever
29+
```
30+
31+
To check these settings, run the following SQL commands:
32+
```sql
33+
SHOW VARIABLES LIKE 'server_id';
34+
SHOW VARIABLES LIKE 'log_bin';
35+
SHOW VARIABLES LIKE 'binlog_format';
36+
SHOW VARIABLES LIKE 'binlog_row_image';
37+
SHOW VARIABLES LIKE 'binlog_row_metadata';
38+
SHOW VARIABLES LIKE 'expire_logs_days';
39+
```
40+
41+
If the values don't match, you can set them in the config file (typically at `/etc/my.cnf` or `/etc/my.cnf.d/mariadb-server.cnf`):
42+
```ini
43+
[mysqld]
44+
server_id = 1
45+
log_bin = ON
46+
binlog_format = ROW
47+
binlog_row_image = FULL
48+
binlog_row_metadata = FULL ; only in 10.5.0 and newer
49+
expire_logs_days = 1
50+
```
51+
52+
You NEED to RESTART the MariaDB instance for the changes to take effect.
53+
54+
:::note
55+
56+
Column exclusion is not supported for MariaDB \<= 10.4 because the `binlog_row_metadata` setting wasn't yet introduced.
57+
58+
:::
59+
60+
## Configure a database user {#configure-database-user}
61+
62+
Connect to your MariaDB instance as the root user and execute the following commands:
63+
64+
1. Create a dedicated user for ClickPipes:
65+
66+
```sql
67+
CREATE USER 'clickpipes_user'@'%' IDENTIFIED BY 'some_secure_password';
68+
```
69+
70+
2. Grant schema permissions. The following example shows permissions for the `clickpipes` database. Repeat these commands for each database and host you want to replicate:
71+
72+
```sql
73+
GRANT SELECT ON `clickpipes`.* TO 'clickpipes_user'@'%';
74+
```
75+
76+
3. Grant replication permissions to the user:
77+
78+
```sql
79+
GRANT REPLICATION CLIENT ON *.* TO 'clickpipes_user'@'%';
80+
GRANT REPLICATION SLAVE ON *.* TO 'clickpipes_user'@'%';
81+
```
82+
83+
:::note
84+
85+
Make sure to replace `clickpipes_user` and `some_secure_password` with your desired username and password.
86+
87+
:::
88+
89+
## SSL/TLS configuration (recommended) {#ssl-tls-configuration}
90+
91+
SSL certificates ensure secure connections to your MariaDB database. Configuration depends on your certificate type:
92+
93+
**Trusted Certificate Authority (DigiCert, Let's Encrypt, etc.)** - no additional configuration needed.
94+
95+
**Internal Certificate Authority** - Obtain the root CA certificate file from your IT team. In the ClickPipes UI, upload it when creating a new MariaDB ClickPipe.
96+
97+
**Self-hosted MariaDB** - Copy the CA certificate from your MariaDB server (look up the path via the `ssl_ca` setting in your `my.cnf`). In the ClickPipes UI, upload it when creating a new MariaDB ClickPipe. Use the IP address of the server as the host.
98+
99+
**Self-hosted MariaDB starting with 11.4** - If your server has `ssl_ca` set up, follow the option above. Otherwise, consult with your IT team to provision a proper certificate. As a last resort, use the "Skip Certificate Verification" toggle in ClickPipes UI (not recommended for security reasons).
100+
101+
For more information on SSL/TLS options, check out our [FAQ](https://clickhouse.com/docs/integrations/clickpipes/mysql/faq#tls-certificate-validation-error).
102+
103+
## What's next? {#whats-next}
104+
105+
You can now [create your ClickPipe](../index.md) and start ingesting data from your MariaDB instance into ClickHouse Cloud.
106+
Make sure to note down the connection details you used while setting up your MariaDB instance as you will need them during the ClickPipe creation process.

‎knowledgebase/aws-privatelink-setup-for-clickpipes.mdx

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,11 @@ RDS instances that require endpoint services (OR you may have different listener
5858
- If using Terraform, set the `enforce_security_group_inbound_rules_on_private_link_traffic` attribute to `off` for the NLB
5959
- This setting is **required** to allow traffic from the ClickPipes VPC to the NLB.
6060

61+
7. **Optional**: Enable Cross-region support in case your database region is different from the ClickPipes region
62+
- Under `Endpoint Services` in the VPC console, select your endpoint service.
63+
- Click on the `Actions` button and select `Modify supported Regions`.
64+
- Add the regions where you want to allow access to the endpoint service, [refer here](/integrations/clickpipes#list-of-static-ips) for the list of ClickPipes regions.
65+
6166
## Initiating connection {#initiating-connection}
6267

6368
When it's done you can follow the guide to [initiate the connection](/integrations/clickpipes/aws-privatelink#creating-clickpipe)

‎scripts/aspell-ignore/en/aspell-dict.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -296,6 +296,7 @@ DeprecatedBadge
296296
DestroyAggregatesThreads
297297
DestroyAggregatesThreadsActive
298298
DictCacheRequests
299+
DigiCert
299300
DiskAvailable
300301
DiskObjectStorage
301302
DiskObjectStorageAsyncThreads

‎scripts/settings/autogenerate-settings.sh

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -315,11 +315,13 @@ insert_src_files=(
315315
"experimental-beta-settings.md"
316316
"arithmetic-functions.md"
317317
"array-functions.md"
318+
"bit-functions.md"
318319
)
319320
insert_dest_files=(
320321
"docs/about-us/beta-and-experimental-features.md"
321322
"docs/sql-reference/functions/arithmetic-functions.md"
322323
"docs/sql-reference/functions/array-functions.md"
324+
"docs/sql-reference/functions/bit-functions.md"
323325
)
324326
echo "[$SCRIPT_NAME] Inserting generated markdown content between <!-- AUTOGENERATED_START --> <!-- AUTOGENERATED_END --> tags"
325327
for i in "${!insert_src_files[@]}"; do

‎scripts/settings/bit-functions.sql

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
WITH bit_functions AS (
2+
SELECT
3+
name,
4+
introduced_in,
5+
syntax,
6+
arguments,
7+
returned_value,
8+
examples
9+
FROM system.functions WHERE categories='Bit' ORDER BY name ASC
10+
)
11+
SELECT
12+
format(
13+
'{}{}{}{}{}{}',
14+
'## ' || name || ' ' || printf('{#%s}', name) || '\n\n',
15+
'Introduced in: v'||introduced_in||'\n\n',
16+
'**Syntax**\n\n'||printf('```sql\n%s\n```', syntax)||'\n\n',
17+
if(empty(arguments), '**Arguments**\n\n- None.\n', '**Arguments**\n\n'||arguments||'\n'),
18+
'**Returned value**\n\n'||trim(returned_value)||'\n\n',
19+
'**Examples**\n\n'||examples||'\n'
20+
)
21+
FROM bit_functions
22+
INTO OUTFILE 'bit-functions.md' TRUNCATE FORMAT LineAsString

‎sidebars.js

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -747,8 +747,10 @@ const sidebars = {
747747
items: [
748748
"integrations/data-ingestion/clickpipes/mysql/source/rds",
749749
"integrations/data-ingestion/clickpipes/mysql/source/aurora",
750+
"integrations/data-ingestion/clickpipes/mysql/source/gcp",
751+
"integrations/data-ingestion/clickpipes/mysql/source/generic",
750752
"integrations/data-ingestion/clickpipes/mysql/source/rds_maria",
751-
"integrations/data-ingestion/clickpipes/mysql/source/gcp"
753+
"integrations/data-ingestion/clickpipes/mysql/source/generic_maria",
752754
],
753755
},
754756
"integrations/data-ingestion/clickpipes/mysql/datatypes"

‎src/components/Feedback/index.jsx

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -191,6 +191,7 @@ export default function Feedback({side}) {
191191

192192

193193
return (
194+
<div className={styles.displayFeedback}>
194195
<Panel hasBorder alignItems='start'>
195196
<Popover open={open} >
196197
<Popover.Trigger>
@@ -212,6 +213,7 @@ export default function Feedback({side}) {
212213
selected === 'neg' && negative_feedback
213214
}
214215
</Popover>
215-
</Panel>
216+
</Panel>
217+
</div>
216218
);
217219
}

0 commit comments

Comments
 (0)