Skip to content

Commit 36458bf

Browse files
ilidemiserprexBlargian
authored
MySQL ClickPipe: Document generic MySQL and MariaDB (#3867)
Co-authored-by: Philip Dubé <[email protected]> Co-authored-by: Shaun Struwig <[email protected]>
1 parent 0f9eda8 commit 36458bf

File tree

5 files changed

+254
-2
lines changed

5 files changed

+254
-2
lines changed

‎docs/integrations/data-ingestion/clickpipes/mysql/index.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,11 @@ To get started, you first need to make sure that your MySQL database is set up c
3636

3737
3. [Cloud SQL for MySQL](./mysql/source/gcp)
3838

39-
4. [Amazon RDS MariaDB](./mysql/source/rds_maria)
39+
4. [Generic MySQL](./mysql/source/generic)
40+
41+
5. [Amazon RDS MariaDB](./mysql/source/rds_maria)
42+
43+
6. [Generic MariaDB](./mysql/source/generic_maria)
4044

4145
Once your source MySQL database is set up, you can continue creating your ClickPipe.
4246

‎docs/integrations/data-ingestion/clickpipes/mysql/source/generic.md

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
---
2+
sidebar_label: 'Generic MySQL'
3+
description: 'Set up any MySQL instance as a source for ClickPipes'
4+
slug: /integrations/clickpipes/mysql/source/generic
5+
title: 'Generic MySQL source setup guide'
6+
---
7+
8+
# Generic MySQL source setup guide
9+
10+
:::info
11+
12+
If you use one of the supported providers (in the sidebar), please refer to the specific guide for that provider.
13+
14+
:::
15+
16+
## Enable binary log retention {#enable-binlog-retention}
17+
18+
Binary logs contain information about data modifications made to a MySQL server instance and are required for replication.
19+
20+
### MySQL 8.x and newer {#binlog-v8-x}
21+
22+
To enable binary logging on your MySQL instance, ensure that the following settings are configured:
23+
24+
```sql
25+
log_bin = ON -- default value
26+
binlog_format = ROW -- default value
27+
binlog_row_image = FULL -- default value
28+
binlog_row_metadata = FULL
29+
binlog_expire_logs_seconds = 86400 -- 1 day or higher; default is 30 days
30+
```
31+
32+
To check these settings, run the following SQL commands:
33+
```sql
34+
SHOW VARIABLES LIKE 'log_bin';
35+
SHOW VARIABLES LIKE 'binlog_format';
36+
SHOW VARIABLES LIKE 'binlog_row_image';
37+
SHOW VARIABLES LIKE 'binlog_row_metadata';
38+
SHOW VARIABLES LIKE 'binlog_expire_logs_seconds';
39+
```
40+
41+
If the values don't match, you can run the following SQL commands to set them:
42+
```sql
43+
SET PERSIST log_bin = ON;
44+
SET PERSIST binlog_format = ROW;
45+
SET PERSIST binlog_row_image = FULL;
46+
SET PERSIST binlog_row_metadata = FULL;
47+
SET PERSIST binlog_expire_logs_seconds = 86400;
48+
```
49+
50+
If you have changed the `log_bin` setting, you NEED to RESTART the MySQL instance for the changes to take effect.
51+
52+
After changing the settings, continue on with [configuring a database user](#configure-database-user).
53+
54+
### MySQL 5.7 {#binlog-v5-x}
55+
56+
To enable binary logging on your MySQL 5.7 instance, ensure that the following settings are configured:
57+
58+
```sql
59+
server_id = 1 -- or greater; anything but 0
60+
log_bin = ON
61+
binlog_format = ROW -- default value
62+
binlog_row_image = FULL -- default value
63+
expire_logs_days = 1 -- or higher; 0 would mean logs are preserved forever
64+
```
65+
66+
To check these settings, run the following SQL commands:
67+
```sql
68+
SHOW VARIABLES LIKE 'server_id';
69+
SHOW VARIABLES LIKE 'log_bin';
70+
SHOW VARIABLES LIKE 'binlog_format';
71+
SHOW VARIABLES LIKE 'binlog_row_image';
72+
SHOW VARIABLES LIKE 'expire_logs_days';
73+
```
74+
75+
If the values don't match, you can set them in the config file (typically at `/etc/my.cnf` or `/etc/mysql/my.cnf`):
76+
```ini
77+
[mysqld]
78+
server_id = 1
79+
log_bin = ON
80+
binlog_format = ROW
81+
binlog_row_image = FULL
82+
expire_logs_days = 1
83+
```
84+
85+
You NEED to RESTART the MySQL instance for the changes to take effect.
86+
87+
:::note
88+
89+
Column exclusion is not supported for MySQL 5.7 because the `binlog_row_metadata` setting wasn't yet introduced.
90+
91+
:::
92+
93+
## Configure a database user {#configure-database-user}
94+
95+
Connect to your MySQL instance as the root user and execute the following commands:
96+
97+
1. Create a dedicated user for ClickPipes:
98+
99+
```sql
100+
CREATE USER 'clickpipes_user'@'%' IDENTIFIED BY 'some_secure_password';
101+
```
102+
103+
2. Grant schema permissions. The following example shows permissions for the `clickpipes` database. Repeat these commands for each database and host you want to replicate:
104+
105+
```sql
106+
GRANT SELECT ON `clickpipes`.* TO 'clickpipes_user'@'%';
107+
```
108+
109+
3. Grant replication permissions to the user:
110+
111+
```sql
112+
GRANT REPLICATION CLIENT ON *.* TO 'clickpipes_user'@'%';
113+
GRANT REPLICATION SLAVE ON *.* TO 'clickpipes_user'@'%';
114+
```
115+
116+
:::note
117+
118+
Make sure to replace `clickpipes_user` and `some_secure_password` with your desired username and password.
119+
120+
:::
121+
122+
## SSL/TLS configuration (recommended) {#ssl-tls-configuration}
123+
124+
SSL certificates ensure secure connections to your MySQL database. Configuration depends on your certificate type:
125+
126+
**Trusted Certificate Authority (DigiCert, Let's Encrypt, etc.)** - no additional configuration needed.
127+
128+
**Internal Certificate Authority** - Obtain the root CA certificate file from your IT team. In the ClickPipes UI, upload it when creating a new MySQL ClickPipe.
129+
130+
**Self-hosted MySQL** - Copy the CA certificate from your MySQL server (typically at `/var/lib/mysql/ca.pem`) and upload it in the UI when creating a new MySQL ClickPipe. Use the IP address of the server as the host.
131+
132+
**Self-hosted MySQL without server access** - Contact your IT team for the certificate. As a last resort, use the "Skip Certificate Verification" toggle in ClickPipes UI (not recommended for security reasons).
133+
134+
For more information on SSL/TLS options, check out our [FAQ](https://clickhouse.com/docs/integrations/clickpipes/mysql/faq#tls-certificate-validation-error).
135+
136+
## What's next? {#whats-next}
137+
138+
You can now [create your ClickPipe](../index.md) and start ingesting data from your MySQL instance into ClickHouse Cloud.
139+
Make sure to note down the connection details you used while setting up your MySQL instance as you will need them during the ClickPipe creation process.

‎docs/integrations/data-ingestion/clickpipes/mysql/source/generic_maria.md

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
---
2+
sidebar_label: 'Generic MariaDB'
3+
description: 'Set up any MariaDB instance as a source for ClickPipes'
4+
slug: /integrations/clickpipes/mysql/source/generic_maria
5+
title: 'Generic MariaDB source setup guide'
6+
---
7+
8+
# Generic MariaDB source setup guide
9+
10+
:::info
11+
12+
If you use one of the supported providers (in the sidebar), please refer to the specific guide for that provider.
13+
14+
:::
15+
16+
## Enable binary log retention {#enable-binlog-retention}
17+
18+
Binary logs contain information about data modifications made to a MariaDB server instance and are required for replication.
19+
20+
To enable binary logging on your MariaDB instance, ensure that the following settings are configured:
21+
22+
```sql
23+
server_id = 1 -- or greater; anything but 0
24+
log_bin = ON
25+
binlog_format = ROW
26+
binlog_row_image = FULL
27+
binlog_row_metadata = FULL -- introduced in 10.5.0
28+
expire_logs_days = 1 -- or higher; 0 would mean logs are preserved forever
29+
```
30+
31+
To check these settings, run the following SQL commands:
32+
```sql
33+
SHOW VARIABLES LIKE 'server_id';
34+
SHOW VARIABLES LIKE 'log_bin';
35+
SHOW VARIABLES LIKE 'binlog_format';
36+
SHOW VARIABLES LIKE 'binlog_row_image';
37+
SHOW VARIABLES LIKE 'binlog_row_metadata';
38+
SHOW VARIABLES LIKE 'expire_logs_days';
39+
```
40+
41+
If the values don't match, you can set them in the config file (typically at `/etc/my.cnf` or `/etc/my.cnf.d/mariadb-server.cnf`):
42+
```ini
43+
[mysqld]
44+
server_id = 1
45+
log_bin = ON
46+
binlog_format = ROW
47+
binlog_row_image = FULL
48+
binlog_row_metadata = FULL ; only in 10.5.0 and newer
49+
expire_logs_days = 1
50+
```
51+
52+
You NEED to RESTART the MariaDB instance for the changes to take effect.
53+
54+
:::note
55+
56+
Column exclusion is not supported for MariaDB \<= 10.4 because the `binlog_row_metadata` setting wasn't yet introduced.
57+
58+
:::
59+
60+
## Configure a database user {#configure-database-user}
61+
62+
Connect to your MariaDB instance as the root user and execute the following commands:
63+
64+
1. Create a dedicated user for ClickPipes:
65+
66+
```sql
67+
CREATE USER 'clickpipes_user'@'%' IDENTIFIED BY 'some_secure_password';
68+
```
69+
70+
2. Grant schema permissions. The following example shows permissions for the `clickpipes` database. Repeat these commands for each database and host you want to replicate:
71+
72+
```sql
73+
GRANT SELECT ON `clickpipes`.* TO 'clickpipes_user'@'%';
74+
```
75+
76+
3. Grant replication permissions to the user:
77+
78+
```sql
79+
GRANT REPLICATION CLIENT ON *.* TO 'clickpipes_user'@'%';
80+
GRANT REPLICATION SLAVE ON *.* TO 'clickpipes_user'@'%';
81+
```
82+
83+
:::note
84+
85+
Make sure to replace `clickpipes_user` and `some_secure_password` with your desired username and password.
86+
87+
:::
88+
89+
## SSL/TLS configuration (recommended) {#ssl-tls-configuration}
90+
91+
SSL certificates ensure secure connections to your MariaDB database. Configuration depends on your certificate type:
92+
93+
**Trusted Certificate Authority (DigiCert, Let's Encrypt, etc.)** - no additional configuration needed.
94+
95+
**Internal Certificate Authority** - Obtain the root CA certificate file from your IT team. In the ClickPipes UI, upload it when creating a new MariaDB ClickPipe.
96+
97+
**Self-hosted MariaDB** - Copy the CA certificate from your MariaDB server (look up the path via the `ssl_ca` setting in your `my.cnf`). In the ClickPipes UI, upload it when creating a new MariaDB ClickPipe. Use the IP address of the server as the host.
98+
99+
**Self-hosted MariaDB starting with 11.4** - If your server has `ssl_ca` set up, follow the option above. Otherwise, consult with your IT team to provision a proper certificate. As a last resort, use the "Skip Certificate Verification" toggle in ClickPipes UI (not recommended for security reasons).
100+
101+
For more information on SSL/TLS options, check out our [FAQ](https://clickhouse.com/docs/integrations/clickpipes/mysql/faq#tls-certificate-validation-error).
102+
103+
## What's next? {#whats-next}
104+
105+
You can now [create your ClickPipe](../index.md) and start ingesting data from your MariaDB instance into ClickHouse Cloud.
106+
Make sure to note down the connection details you used while setting up your MariaDB instance as you will need them during the ClickPipe creation process.

‎scripts/aspell-ignore/en/aspell-dict.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -296,6 +296,7 @@ DeprecatedBadge
296296
DestroyAggregatesThreads
297297
DestroyAggregatesThreadsActive
298298
DictCacheRequests
299+
DigiCert
299300
DiskAvailable
300301
DiskObjectStorage
301302
DiskObjectStorageAsyncThreads

‎sidebars.js

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -747,8 +747,10 @@ const sidebars = {
747747
items: [
748748
"integrations/data-ingestion/clickpipes/mysql/source/rds",
749749
"integrations/data-ingestion/clickpipes/mysql/source/aurora",
750+
"integrations/data-ingestion/clickpipes/mysql/source/gcp",
751+
"integrations/data-ingestion/clickpipes/mysql/source/generic",
750752
"integrations/data-ingestion/clickpipes/mysql/source/rds_maria",
751-
"integrations/data-ingestion/clickpipes/mysql/source/gcp"
753+
"integrations/data-ingestion/clickpipes/mysql/source/generic_maria",
752754
],
753755
},
754756
"integrations/data-ingestion/clickpipes/mysql/datatypes"

0 commit comments

Comments
 (0)