Skip to content

Commit fc64882

Browse files
soyeric128soyeric128
committed
Create migrating-from-mysql-with-db-archiver.md
1 parent 6d3038b commit fc64882

File tree

1 file changed

+216
-0
lines changed

1 file changed

+216
-0
lines changed

‎docs/en/tutorials/migrate/migrating-from-mysql-with-db-archiver.md‎

Lines changed: 216 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,216 @@
1+
---
2+
title: Migrating from MySQL with db-archiver
3+
---
4+
5+
In this tutorial, we'll walk you through the process of migrating from MySQL to Databend Cloud using db-archiver.
6+
7+
## Before You Start
8+
9+
Before you start, ensure you have the following prerequisites in place:
10+
11+
- [Docker](https://www.docker.com/) is installed on your local machine, as it will be used to launch MySQL.
12+
- [Go](https://go.dev/dl/) is installed on your local machine, as it is required to install db-archiver.
13+
- BendSQL is installed on your local machine. See [Installing BendSQL](/guides/sql-clients/bendsql/#installing-bendsql) for instructions on how to install BendSQL using various package managers.
14+
15+
## Step 1: Launch MySQL in Docker
16+
17+
1. Start a MySQL container on your local machine. The command below launches a MySQL container named **mysql-server**, creates a database named **mydb**, and sets the root password to `root`:
18+
19+
```bash
20+
docker run \
21+
--name mysql-server \
22+
-e MYSQL_ROOT_PASSWORD=root \
23+
-e MYSQL_DATABASE=mydb \
24+
-p 3306:3306 \
25+
-d mysql:8
26+
```
27+
28+
2. Verify MySQL is running:
29+
30+
```bash
31+
docker ps
32+
```
33+
34+
Check the output for a container named **mysql-server**:
35+
36+
```bash
37+
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
38+
1a8f8d7d0e1a mysql:8 "docker-entrypoint.s…" 10 hours ago Up About an hour 0.0.0.0:3306->3306/tcp, 33060/tcp mysql-server
39+
```
40+
41+
## Step 2: Populate MySQL with Sample Data
42+
43+
1. Log in to the MySQL container and enter the password `root` when prompted:
44+
45+
```bash
46+
docker exec -it mysql-server mysql -u root -p
47+
```
48+
49+
```
50+
Enter password:
51+
Welcome to the MySQL monitor. Commands end with ; or \g.
52+
Your MySQL connection id is 8
53+
Server version: 8.4.4 MySQL Community Server - GPL
54+
55+
Copyright (c) 2000, 2025, Oracle and/or its affiliates.
56+
57+
Oracle is a registered trademark of Oracle Corporation and/or its
58+
affiliates. Other names may be trademarks of their respective
59+
owners.
60+
61+
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
62+
```
63+
64+
2. Switch to the **mydb** database:
65+
66+
```bash
67+
mysql> USE mydb;
68+
Database changed
69+
```
70+
71+
3. Copy and paste the following SQL to create a table named **my_table** and insert data:
72+
73+
```sql
74+
CREATE TABLE my_table (
75+
id INT AUTO_INCREMENT PRIMARY KEY,
76+
name VARCHAR(255),
77+
value INT
78+
);
79+
INSERT INTO my_table (name, value) VALUES
80+
('Alice', 10),
81+
('Bob', 20),
82+
('Charlie', 30);
83+
```
84+
85+
4. Verify the data:
86+
87+
```bash
88+
mysql> SELECT * FROM my_table;
89+
+----+---------+-------+
90+
| id | name | value |
91+
+----+---------+-------+
92+
| 1 | Alice | 10 |
93+
| 2 | Bob | 20 |
94+
| 3 | Charlie | 30 |
95+
+----+---------+-------+
96+
3 rows in set (0.00 sec)
97+
```
98+
99+
5. Quit the MySQL container:
100+
101+
```bash
102+
mysql> quit
103+
Bye
104+
```
105+
106+
## Step 3: Set Up Target in Databend Cloud
107+
108+
1. Connect to Databend Cloud using BendSQL. If you're unfamiliar with BendSQL, refer to this tutorial: [Connecting to Databend Cloud using BendSQL](../connect/connect-to-databendcloud-bendsql.md).
109+
2. Copy and paste the following SQL to create a target table named **my_table**:
110+
111+
```sql
112+
CREATE TABLE my_table (
113+
id INT NOT NULL,
114+
name VARCHAR(255),
115+
value INT
116+
);
117+
```
118+
119+
## Step 4: Install db-archiver
120+
121+
Install db-archiver using the `go install` command:
122+
123+
```bash
124+
go install github.com/databend/db-archiver@latest
125+
```
126+
127+
## Step 5: Configure & Run db-archiver
128+
129+
1. Create a file named **conf.json** on your local machine with the following content:
130+
131+
```json
132+
{
133+
// Replace the placeholders with your actual values:
134+
"sourceHost": "127.0.0.1",
135+
"sourcePort": 3306,
136+
"sourceUser": "root",
137+
"sourcePass": "root",
138+
"sourceDB": "mydb",
139+
"sourceTable": "my_table",
140+
"sourceQuery": "select * from mydb.my_table",
141+
"sourceSplitKey": "id",
142+
"sourceWhereCondition": "id < 100",
143+
"databendDSN": "https://cloudapp:{password}@{host}:443?warehouse={warehouse_name}",
144+
"databendTable": "{database}.my_table",
145+
"batchSize": 2,
146+
"batchMaxInterval": 30,
147+
"workers": 1,
148+
"copyPurge": false,
149+
"copyForce": false,
150+
"disableVariantCheck": false,
151+
"deleteAfterSync": false,
152+
"maxThread": 10
153+
}
154+
```
155+
156+
2. Run the following command in the directory where your **conf.json** file is located to start the migration:
157+
158+
```bash
159+
db-archiver -f conf.json
160+
```
161+
162+
Migration will begin as follows:
163+
164+
```bash
165+
start time: 2025-01-22 21:45:33
166+
sourcedatabase pattern ^mydb$
167+
not match db: information_schema
168+
sourcedatabase pattern ^mydb$
169+
match db: mydb
170+
sourcedatabase pattern ^mydb$
171+
not match db: mysql
172+
sourcedatabase pattern ^mydb$
173+
not match db: performance_schema
174+
sourcedatabase pattern ^mydb$
175+
not match db: sys
176+
INFO[0000] Start worker mydb.my_table
177+
INFO[0000] Worker mydb.my_table checking before start
178+
INFO[0000] Starting worker mydb.my_table
179+
INFO[0000] db.table is mydb.my_table, minSplitKey: 1, maxSplitKey : 6
180+
2025/01/22 21:45:33 thread-1: extract 2 rows (1.997771 rows/s)
181+
2025/01/22 21:45:33 thread-1: extract 0 rows (1.999639 rows/s)
182+
2025/01/22 21:45:33 thread-1: extract 2 rows (1.999887 rows/s)
183+
2025/01/22 21:45:33 thread-1: extract 2 rows (1.999786 rows/s)
184+
INFO[0000] get presigned url cost: 126 ms
185+
INFO[0000] get presigned url cost: 140 ms
186+
INFO[0000] get presigned url cost: 159 ms
187+
INFO[0000] upload by presigned url cost: 194 ms
188+
INFO[0000] upload by presigned url cost: 218 ms
189+
INFO[0000] upload by presigned url cost: 230 ms
190+
INFO[0000] thread-1: copy into cost: 364 ms ingest_databend=IngestData
191+
2025/01/22 21:45:34 thread-1: ingest 2 rows (2.777579 rows/s), 68 bytes (94.437695 bytes/s)
192+
2025/01/22 21:45:34 Globla speed: total ingested 2 rows (2.777143 rows/s), 29 bytes (40.268568 bytes/s)
193+
INFO[0001] thread-1: copy into cost: 407 ms ingest_databend=IngestData
194+
2025/01/22 21:45:34 thread-1: ingest 2 rows (2.603310 rows/s), 72 bytes (88.512532 bytes/s)
195+
2025/01/22 21:45:34 Globla speed: total ingested 4 rows (2.603103 rows/s), 62 bytes (37.744993 bytes/s)
196+
INFO[0001] thread-1: copy into cost: 475 ms ingest_databend=IngestData
197+
2025/01/22 21:45:34 thread-1: ingest 2 rows (2.401148 rows/s), 70 bytes (81.639015 bytes/s)
198+
2025/01/22 21:45:34 Globla speed: total ingested 6 rows (2.400957 rows/s), 93 bytes (34.813873 bytes/s)
199+
INFO[0001] Worker dbarchiver finished and data correct, source data count is 6, target data count is 6
200+
end time: 2025-01-22 21:45:34
201+
total time: 1.269478875s
202+
```
203+
204+
3. Return to your BendSQL session and verify the migration:
205+
206+
```sql
207+
SELECT * FROM my_table;
208+
209+
┌────────────────────────────────────────────┐
210+
│ id │ name │ value │
211+
├───────┼──────────────────┼─────────────────┤
212+
│ 3 │ Charlie │ 30 │
213+
│ 1 │ Alice │ 10 │
214+
│ 2 │ Bob │ 20 │
215+
└────────────────────────────────────────────┘
216+
```

0 commit comments

Comments
 (0)