Skip to content

Commit 4871cc7

Browse files
soyeric128soyeric128
committed
Migrating from MySQL with DataX
1 parent 10196ab commit 4871cc7

File tree

2 files changed

+123
-118
lines changed

2 files changed

+123
-118
lines changed

‎docs/en/guides/40-load-data/02-load-db/datax.md‎

Lines changed: 2 additions & 118 deletions
Original file line numberDiff line numberDiff line change
@@ -24,122 +24,6 @@ DatabendWriter supports two operational modes: INSERT (default) and REPLACE. In
2424

2525
If you need more information about DatabendWriter and its functionalities, you can refer to the documentation available at https://github.com/alibaba/DataX/blob/master/databendwriter/doc/databendwriter.md
2626

27-
## Tutorial: Data Loading from MySQL
27+
## Tutorials
2828

29-
In this tutorial, you will load data from MySQL to Databend with DataX. Before you start, make sure you have successfully set up Databend, MySQL, and DataX in your environment.
30-
31-
1. In MySQL, create a SQL user that you will use for data loading and then create a table and populate it with sample data.
32-
33-
```sql title='In MySQL:'
34-
mysql> create user 'mysqlu1'@'%' identified by 'databend';
35-
mysql> grant all on *.* to 'mysqlu1'@'%';
36-
mysql> create database db;
37-
mysql> create table db.tb01(id int, d double, t TIMESTAMP, col1 varchar(10));
38-
mysql> insert into db.tb01 values(1, 3.1,now(), 'test1'), (1, 4.1,now(), 'test2'), (1, 4.1,now(), 'test2');
39-
```
40-
41-
2. In Databend, create a corresponding target table.
42-
43-
:::note
44-
DataX data types can be converted to Databend's data types when loaded into Databend. For the specific correspondences between DataX data types and Databend's data types, refer to the documentation provided at the following link: https://github.com/alibaba/DataX/blob/master/databendwriter/doc/databendwriter.md#33-type-convert
45-
:::
46-
47-
```sql title='In Databend:'
48-
databend> create database migrated_db;
49-
databend> create table migrated_db.tb01(id int null, d double null, t TIMESTAMP null, col1 varchar(10) null);
50-
```
51-
52-
3. Copy and paste the following code to a file, and name the file as *mysql_demo.json*. For the available parameters and their descriptions, refer to the documentation provided at the following link: https://github.com/alibaba/DataX/blob/master/databendwriter/doc/databendwriter.md#32-configuration-description
53-
54-
```json title='mysql_demo.json'
55-
{
56-
"job": {
57-
"content": [
58-
{
59-
"reader": {
60-
"name": "mysqlreader",
61-
"parameter": {
62-
"username": "mysqlu1",
63-
"password": "databend",
64-
"column": [
65-
"id", "d", "t", "col1"
66-
],
67-
"connection": [
68-
{
69-
"jdbcUrl": [
70-
"jdbc:mysql://127.0.0.1:3307/db"
71-
],
72-
"driver": "com.mysql.jdbc.Driver",
73-
"table": [
74-
"tb01"
75-
]
76-
}
77-
]
78-
}
79-
},
80-
"writer": {
81-
"name": "databendwriter",
82-
"parameter": {
83-
"username": "databend",
84-
"password": "databend",
85-
"column": [
86-
"id", "d", "t", "col1"
87-
],
88-
"preSql": [
89-
],
90-
"postSql": [
91-
],
92-
"connection": [
93-
{
94-
"jdbcUrl": "jdbc:databend://localhost:8000/migrated_db",
95-
"table": [
96-
"tb01"
97-
]
98-
}
99-
]
100-
}
101-
}
102-
}
103-
],
104-
"setting": {
105-
"speed": {
106-
"channel": 1
107-
}
108-
}
109-
}
110-
}
111-
```
112-
113-
:::tip
114-
The provided code above configures DatabendWriter to operate in the INSERT mode. To switch to the REPLACE mode, you must include the writeMode and onConflictColumn parameters. For example:
115-
116-
```json title='mysql_demo.json'
117-
...
118-
"writer": {
119-
"name": "databendwriter",
120-
"parameter": {
121-
"writeMode": "replace",
122-
"onConflictColumn":["id"],
123-
"username": ...
124-
```
125-
:::
126-
127-
4. Run DataX:
128-
129-
```shell
130-
cd {YOUR_DATAX_DIR_BIN}
131-
python datax.py ./mysql_demo.json
132-
```
133-
134-
You're all set! To verify the data loading, execute the query in Databend:
135-
136-
```sql
137-
databend> select * from migrated_db.tb01;
138-
+------+------+----------------------------+-------+
139-
| id | d | t | col1 |
140-
+------+------+----------------------------+-------+
141-
| 1 | 3.1 | 2023-02-01 07:11:08.500000 | test1 |
142-
| 1 | 4.1 | 2023-02-01 07:11:08.501000 | test2 |
143-
| 1 | 4.1 | 2023-02-01 07:11:08.501000 | test2 |
144-
+------+------+----------------------------+-------+
145-
```
29+
- [Migrating from MySQL with DataX](/tutorials/migrate/migrating-from-mysql-with-datax)

‎docs/en/tutorials/migrate/migrating-from-mysql-with-datax.md‎

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
---
2+
title: Migrating from MySQL with DataX
3+
---
4+
5+
In this tutorial, you will load data from MySQL to Databend with DataX. Before you start, make sure you have successfully set up Databend, MySQL, and DataX in your environment.
6+
7+
1. In MySQL, create a SQL user that you will use for data loading and then create a table and populate it with sample data.
8+
9+
```sql title='In MySQL:'
10+
mysql> create user 'mysqlu1'@'%' identified by 'databend';
11+
mysql> grant all on *.* to 'mysqlu1'@'%';
12+
mysql> create database db;
13+
mysql> create table db.tb01(id int, d double, t TIMESTAMP, col1 varchar(10));
14+
mysql> insert into db.tb01 values(1, 3.1,now(), 'test1'), (1, 4.1,now(), 'test2'), (1, 4.1,now(), 'test2');
15+
```
16+
17+
2. In Databend, create a corresponding target table.
18+
19+
:::note
20+
DataX data types can be converted to Databend's data types when loaded into Databend. For the specific correspondences between DataX data types and Databend's data types, refer to the documentation provided at the following link: https://github.com/alibaba/DataX/blob/master/databendwriter/doc/databendwriter.md#33-type-convert
21+
:::
22+
23+
```sql title='In Databend:'
24+
databend> create database migrated_db;
25+
databend> create table migrated_db.tb01(id int null, d double null, t TIMESTAMP null, col1 varchar(10) null);
26+
```
27+
28+
3. Copy and paste the following code to a file, and name the file as *mysql_demo.json*. For the available parameters and their descriptions, refer to the documentation provided at the following link: https://github.com/alibaba/DataX/blob/master/databendwriter/doc/databendwriter.md#32-configuration-description
29+
30+
```json title='mysql_demo.json'
31+
{
32+
"job": {
33+
"content": [
34+
{
35+
"reader": {
36+
"name": "mysqlreader",
37+
"parameter": {
38+
"username": "mysqlu1",
39+
"password": "databend",
40+
"column": [
41+
"id", "d", "t", "col1"
42+
],
43+
"connection": [
44+
{
45+
"jdbcUrl": [
46+
"jdbc:mysql://127.0.0.1:3307/db"
47+
],
48+
"driver": "com.mysql.jdbc.Driver",
49+
"table": [
50+
"tb01"
51+
]
52+
}
53+
]
54+
}
55+
},
56+
"writer": {
57+
"name": "databendwriter",
58+
"parameter": {
59+
"username": "databend",
60+
"password": "databend",
61+
"column": [
62+
"id", "d", "t", "col1"
63+
],
64+
"preSql": [
65+
],
66+
"postSql": [
67+
],
68+
"connection": [
69+
{
70+
"jdbcUrl": "jdbc:databend://localhost:8000/migrated_db",
71+
"table": [
72+
"tb01"
73+
]
74+
}
75+
]
76+
}
77+
}
78+
}
79+
],
80+
"setting": {
81+
"speed": {
82+
"channel": 1
83+
}
84+
}
85+
}
86+
}
87+
```
88+
89+
:::tip
90+
The provided code above configures DatabendWriter to operate in the INSERT mode. To switch to the REPLACE mode, you must include the writeMode and onConflictColumn parameters. For example:
91+
92+
```json title='mysql_demo.json'
93+
...
94+
"writer": {
95+
"name": "databendwriter",
96+
"parameter": {
97+
"writeMode": "replace",
98+
"onConflictColumn":["id"],
99+
"username": ...
100+
```
101+
:::
102+
103+
4. Run DataX:
104+
105+
```shell
106+
cd {YOUR_DATAX_DIR_BIN}
107+
python datax.py ./mysql_demo.json
108+
```
109+
110+
You're all set! To verify the data loading, execute the query in Databend:
111+
112+
```sql
113+
databend> select * from migrated_db.tb01;
114+
+------+------+----------------------------+-------+
115+
| id | d | t | col1 |
116+
+------+------+----------------------------+-------+
117+
| 1 | 3.1 | 2023-02-01 07:11:08.500000 | test1 |
118+
| 1 | 4.1 | 2023-02-01 07:11:08.501000 | test2 |
119+
| 1 | 4.1 | 2023-02-01 07:11:08.501000 | test2 |
120+
+------+------+----------------------------+-------+
121+
```

0 commit comments

Comments
 (0)