Skip to content

Commit f46d680

Browse files
soyeric128soyeric128
committed
Migrating from MySQL with Debezium
1 parent 4871cc7 commit f46d680

File tree

2 files changed

+93
-88
lines changed

2 files changed

+93
-88
lines changed

‎docs/en/guides/40-load-data/02-load-db/debezium.md‎

Lines changed: 2 additions & 88 deletions
Original file line numberDiff line numberDiff line change
@@ -89,92 +89,6 @@ services:
8989
docker-compose up -d
9090
```
9191

92-
## Usage Example
92+
## Tutorials
9393

94-
This section demonstrates the general steps to load data from MySQL into Databend and assumes that you already have a local MySQL instance running.
95-
96-
### Step 1. Prepare Data in MySQL
97-
98-
Create a database and a table in MySQL, and insert sample data into the table.
99-
100-
```sql
101-
CREATE DATABASE mydb;
102-
USE mydb;
103-
104-
CREATE TABLE products (id INTEGER NOT NULL AUTO_INCREMENT PRIMARY KEY,name VARCHAR(255) NOT NULL,description VARCHAR(512));
105-
ALTER TABLE products AUTO_INCREMENT = 10;
106-
107-
INSERT INTO products VALUES (default,"scooter","Small 2-wheel scooter"),
108-
(default,"car battery","12V car battery"),
109-
(default,"12-pack drill bits","12-pack of drill bits with sizes ranging from #40 to #3"),
110-
(default,"hammer","12oz carpenter's hammer"),
111-
(default,"hammer","14oz carpenter's hammer"),
112-
(default,"hammer","16oz carpenter's hammer"),
113-
(default,"rocks","box of assorted rocks"),
114-
(default,"jacket","water-proof black wind breaker"),
115-
(default,"cloud","test for databend"),
116-
(default,"spare tire","24 inch spare tire");
117-
```
118-
119-
### Step 2. Create database in Databend
120-
121-
Create the corresponding database in Databend. Please note that you don't need to create a table that corresponds to the one in MySQL.
122-
123-
```sql
124-
CREATE DATABASE debezium;
125-
```
126-
127-
### Step 3. Create application.properties
128-
129-
Create the file _application.properties_, then start debezium-server-databend. For how to install and start the tool, see [Installing debezium-server-databend](#installing-debezium-server-databend).
130-
131-
When started for the first time, the tool performs a full synchronization of data from MySQL to Databend using the specified Batch Size. As a result, the data from MySQL is now visible in Databend after successful replication.
132-
133-
```text title='application.properties'
134-
debezium.sink.type=databend
135-
debezium.sink.databend.upsert=true
136-
debezium.sink.databend.upsert-keep-deletes=false
137-
debezium.sink.databend.database.databaseName=debezium
138-
debezium.sink.databend.database.url=jdbc:databend://<your-databend-host>:<port>
139-
debezium.sink.databend.database.username=<your-username>
140-
debezium.sink.databend.database.password=<your-password>
141-
debezium.sink.databend.database.primaryKey=id
142-
debezium.sink.databend.database.tableName=products
143-
debezium.sink.databend.database.param.ssl=true
144-
145-
# enable event schemas
146-
debezium.format.value.schemas.enable=true
147-
debezium.format.key.schemas.enable=true
148-
debezium.format.value=json
149-
debezium.format.key=json
150-
151-
# mysql source
152-
debezium.source.connector.class=io.debezium.connector.mysql.MySqlConnector
153-
debezium.source.offset.storage.file.filename=data/offsets.dat
154-
debezium.source.offset.flush.interval.ms=60000
155-
156-
debezium.source.database.hostname=127.0.0.1
157-
debezium.source.database.port=3306
158-
debezium.source.database.user=root
159-
debezium.source.database.password=123456
160-
debezium.source.database.dbname=mydb
161-
debezium.source.database.server.name=from_mysql
162-
debezium.source.include.schema.changes=false
163-
debezium.source.table.include.list=mydb.products
164-
# debezium.source.database.ssl.mode=required
165-
# Run without Kafka, use local file to store checkpoints
166-
debezium.source.database.history=io.debezium.relational.history.FileDatabaseHistory
167-
debezium.source.database.history.file.filename=data/status.dat
168-
# do event flattening. unwrap message!
169-
debezium.transforms=unwrap
170-
debezium.transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
171-
debezium.transforms.unwrap.delete.handling.mode=rewrite
172-
debezium.transforms.unwrap.drop.tombstones=true
173-
174-
# ############ SET LOG LEVELS ############
175-
quarkus.log.level=INFO
176-
# Ignore messages below warning level from Jetty, because it's a bit verbose
177-
quarkus.log.category."org.eclipse.jetty".level=WARN
178-
```
179-
180-
You're all set! If you query the products table in Databend, you will see that the data from MySQL has been successfully synchronized. Feel free to perform insertions, updates, or deletions in MySQL, and you will observe the corresponding changes reflected in Databend as well.
94+
- [Migrating from MySQL with Debezium](/tutorials/migrate/migrating-from-mysql-with-debezium)

‎docs/en/tutorials/migrate/migrating-from-mysql-with-debezium.md‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
---
2+
title: Migrating from MySQL with Debezium
3+
---
4+
5+
In this tutorial, you will load data from MySQL to Databend with Debezium. Before you start, make sure you have successfully set up Databend, MySQL, and Debezium in your environment.
6+
7+
## Step 1. Prepare Data in MySQL
8+
9+
Create a database and a table in MySQL, and insert sample data into the table.
10+
11+
```sql
12+
CREATE DATABASE mydb;
13+
USE mydb;
14+
15+
CREATE TABLE products (id INTEGER NOT NULL AUTO_INCREMENT PRIMARY KEY,name VARCHAR(255) NOT NULL,description VARCHAR(512));
16+
ALTER TABLE products AUTO_INCREMENT = 10;
17+
18+
INSERT INTO products VALUES (default,"scooter","Small 2-wheel scooter"),
19+
(default,"car battery","12V car battery"),
20+
(default,"12-pack drill bits","12-pack of drill bits with sizes ranging from #40 to #3"),
21+
(default,"hammer","12oz carpenter's hammer"),
22+
(default,"hammer","14oz carpenter's hammer"),
23+
(default,"hammer","16oz carpenter's hammer"),
24+
(default,"rocks","box of assorted rocks"),
25+
(default,"jacket","water-proof black wind breaker"),
26+
(default,"cloud","test for databend"),
27+
(default,"spare tire","24 inch spare tire");
28+
```
29+
30+
## Step 2. Create database in Databend
31+
32+
Create the corresponding database in Databend. Please note that you don't need to create a table that corresponds to the one in MySQL.
33+
34+
```sql
35+
CREATE DATABASE debezium;
36+
```
37+
38+
## Step 3. Create application.properties
39+
40+
Create the file _application.properties_, then start debezium-server-databend. For how to install and start the tool, see [Installing debezium-server-databend](#installing-debezium-server-databend).
41+
42+
When started for the first time, the tool performs a full synchronization of data from MySQL to Databend using the specified Batch Size. As a result, the data from MySQL is now visible in Databend after successful replication.
43+
44+
```text title='application.properties'
45+
debezium.sink.type=databend
46+
debezium.sink.databend.upsert=true
47+
debezium.sink.databend.upsert-keep-deletes=false
48+
debezium.sink.databend.database.databaseName=debezium
49+
debezium.sink.databend.database.url=jdbc:databend://<your-databend-host>:<port>
50+
debezium.sink.databend.database.username=<your-username>
51+
debezium.sink.databend.database.password=<your-password>
52+
debezium.sink.databend.database.primaryKey=id
53+
debezium.sink.databend.database.tableName=products
54+
debezium.sink.databend.database.param.ssl=true
55+
56+
# enable event schemas
57+
debezium.format.value.schemas.enable=true
58+
debezium.format.key.schemas.enable=true
59+
debezium.format.value=json
60+
debezium.format.key=json
61+
62+
# mysql source
63+
debezium.source.connector.class=io.debezium.connector.mysql.MySqlConnector
64+
debezium.source.offset.storage.file.filename=data/offsets.dat
65+
debezium.source.offset.flush.interval.ms=60000
66+
67+
debezium.source.database.hostname=127.0.0.1
68+
debezium.source.database.port=3306
69+
debezium.source.database.user=root
70+
debezium.source.database.password=123456
71+
debezium.source.database.dbname=mydb
72+
debezium.source.database.server.name=from_mysql
73+
debezium.source.include.schema.changes=false
74+
debezium.source.table.include.list=mydb.products
75+
# debezium.source.database.ssl.mode=required
76+
# Run without Kafka, use local file to store checkpoints
77+
debezium.source.database.history=io.debezium.relational.history.FileDatabaseHistory
78+
debezium.source.database.history.file.filename=data/status.dat
79+
# do event flattening. unwrap message!
80+
debezium.transforms=unwrap
81+
debezium.transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
82+
debezium.transforms.unwrap.delete.handling.mode=rewrite
83+
debezium.transforms.unwrap.drop.tombstones=true
84+
85+
# ############ SET LOG LEVELS ############
86+
quarkus.log.level=INFO
87+
# Ignore messages below warning level from Jetty, because it's a bit verbose
88+
quarkus.log.category."org.eclipse.jetty".level=WARN
89+
```
90+
91+
You're all set! If you query the products table in Databend, you will see that the data from MySQL has been successfully synchronized. Feel free to perform insertions, updates, or deletions in MySQL, and you will observe the corresponding changes reflected in Databend as well.

0 commit comments

Comments
 (0)