Skip to content

Commit 472d4f4

Browse files
TanZiYenElliezza
andauthored
docs: change_for_rocketmq_connector_of_integration_folder (#3475)
* Change for rocketmq_connector of integration folder * Update rocketmq_connector.md * Update rocketmq_connector.md * Update rocketmq_connector.md zh --------- Co-authored-by: Siqi Wang <[email protected]>
1 parent 656aa78 commit 472d4f4

File tree

6 files changed

+256
-4
lines changed

6 files changed

+256
-4
lines changed
41.4 KB
Loading
13.2 KB
Loading
39.5 KB
Loading
33.5 KB
Loading
Lines changed: 252 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,252 @@
1+
# RocketMQ
2+
3+
## Why Choose RocketMQ OpenMLDB Connector
4+
5+
To enable efficient and stable transmission channels between OpenMLDB and RocketMQ, the RocketMQ OpenMLDB Connector offers a range of outstanding features, including but not limited to:
6+
- **User-friendly setup**: No coding is required; a **straightforward configuration** allows for seamless flow of RocketMQ messages into OpenMLDB through the RocketMQ OpenMLDB Connector. This simplified data import process **significantly enhances** the effective utilization of enterprise data.
7+
- **Flexible deployment**: Depending on the specific business requirements, the RocketMQ OpenMLDB Connector can be deployed on a **single machine or as a cluster**, enabling enterprises to build real-time data pipelines.
8+
- **Robust reliability**: The cluster deployment method of the RocketMQ OpenMLDB Connector incorporates Failover capability, ensuring smooth task scheduling from problematic nodes to normal nodes and maintaining cluster load balancing. This enhances the **focus and efficiency** of enterprises in exploring the commercial value of their data.
9+
- **Low latency**: With a capacity to meet real-time data and feature development scenarios with just a **second-level delay**, the RocketMQ OpenMLDB Connector facilitates timely and efficient data processing.
10+
11+
## RocketMQ OpenMLDB Connector
12+
13+
### Overview
14+
15+
**Target**
16+
17+
RocketMQ Connect plays a crucial role in facilitating data integration within the RocketMQ ecosystem. It boasts several key features, including low latency, reliability, high performance, low code requirements, and strong scalability. This versatile tool enables seamless connections between various heterogeneous data systems, enabling the construction of data pipelines, ETL (Extract, Transform, Load) processes, CDC (Change Data Capture) operations, data lakes, and more.
18+
19+
The RocketMQ OpenMLDB Connector serves as a dependable and scalable solution for streaming data between RocketMQ and OpenMLDB. Its primary purpose is to simplify the importation of data from RocketMQ and other RocketMQ Connect components into OpenMLDB.
20+
21+
**Function**
22+
23+
It can enable RocketMQ messages to flow into OpenMLDB online storage.
24+
25+
![img](images/rocketmq_overview.png)
26+
27+
**Connector Plugin Compilation**
28+
29+
RocketMQ OpenMLDB Connector
30+
31+
```bash
32+
$ git clone [email protected]:apache/rocketmq-connect.git
33+
$ cd rocketmq-connect/connectors/rocketmq-connect-jdbc/
34+
$ mvn clean package -Dmaven.test.skip=true
35+
```
36+
37+
Last but not least, place the compiled plugin package at the loading address specified by RocketMQ connect.
38+
39+
### Connector Usage Demonstration
40+
41+
**Process Introduction**
42+
43+
Specifically designed for real-time data stream access in OpenMLDB's online mode, the RocketMQ OpenMLDB Connector follows a straightforward four-step usage process:
44+
45+
- Start OpenMLDB and create a database, such as "rocketmq_test," for testing purposes. The RocketMQ connector automatically handles table creation, eliminating the need for manual intervention.
46+
- Start RocketMQ and create a topic to serve as the communication channel for data transfer.
47+
- Launch the RocketMQ OpenMLDB Connector to establish the connection and enable data streaming between RocketMQ and OpenMLDB.
48+
- Proceed with testing or commence normal use to ensure the smooth and efficient transfer of data.
49+
50+
![img](images/rocketmq_workflow.png)
51+
**The Key Steps**
52+
53+
The key steps for using this connector is stated below:
54+
55+
**Step 1 | Start OpenMLDB**
56+
57+
By adhering to these steps, users can leverage the power of RocketMQ OpenMLDB Connector to facilitate real-time data streaming and integration in their online OpenMLDB environment.
58+
59+
```bash
60+
cd /work
61+
./init.sh
62+
echo "create database rocketmq_test;" | /work/openmldb/bin/openmldb --zk_cluster=127.0.0.1:2181 --zk_root_path=/openmldb --role=sql_client
63+
```
64+
65+
**Step 2 | Start RocketMQ**
66+
67+
```bash
68+
Build RocketMQ and Start RocketMQ
69+
1. Downlaod RocketMQ
70+
$ wget https://dlcdn.apache.org/rocketmq/4.9.3/rocketmq-all-4.9.3-source-release.zip
71+
72+
73+
2. Compile RocketMQ
74+
If it has already been compiled, please directly execute Part 3 to start RocketMQ
75+
76+
$ unzip rocketmq-all-4.9.3-source-release.zip
77+
$ cd rocketmq-all-4.9.3/
78+
$ mvn -Prelease-all -DskipTests clean install -U
79+
$ cd distribution/target/rocketmq-4.9.3/rocketmq-4.9.3
80+
81+
82+
3. Start RocketMQ
83+
Start namesrv
84+
$ nohup sh bin/mqnamesrv &
85+
Check if namesrv has been successfully started
86+
$ tail -f ~/logs/rocketmqlogs/namesrv.log
87+
The Name Server boot success...
88+
89+
90+
Start broker
91+
$ nohup sh bin/mqbroker -n localhost:9876 &
92+
Check if broker has been successfully started
93+
$ tail -f ~/logs/rocketmqlogs/broker.log
94+
The broker[%s, 172.30.30.233:10911] boot success...
95+
```
96+
97+
**Step 3 | Start RocketMQ OpenMLDB Connector**
98+
99+
First, built the RocketMQ connect runtime environment.
100+
101+
Downloading the project
102+
103+
```bash
104+
$ git clone [email protected]:apache/rocketmq-connect.git
105+
```
106+
107+
Build the project
108+
109+
```bash
110+
$ cd rocketmq-connect
111+
$ mvn -Prelease-connect -DskipTests clean install -U
112+
```
113+
114+
Edit the configuration `connect-standalone.conf` , and the key configurations are as follows:
115+
116+
```bash
117+
$ cd distribution/target/rocketmq-connect-0.0.1-SNAPSHOT/rocketmq-connect-0.0.1-SNAPSHOT
118+
$ vim conf/connect-standalone.conf
119+
```
120+
121+
```yaml
122+
# Unique Id of the current node
123+
workerId=DEFAULT_WORKER_1
124+
125+
# The port address of the REST API
126+
httpPort=8081
127+
128+
# Local storage path
129+
storePathRootDir=~/storeRoot
130+
131+
# Need to modify to the port address of your own Rocketmq NameServer
132+
# Rocketmq namesrvAddr
133+
namesrvAddr=127.0.0.1:9876
134+
135+
# Need to modify to the location of the connector plugins folder
136+
# Source or sink connector jar file dir
137+
pluginPaths=/usr/local/connector-plugins/
138+
```
139+
140+
We need to put the compiled package of OpenMLDB RocketMQ Connector into this directory. The command is as follows:
141+
142+
```bash
143+
mkdir -p /usr/local/connector-plugins/rocketmq-connect-jdbc
144+
cd ../../../../
145+
cp connectors/rocketmq-connect-jdbc/target/rocketmq-connect-jdbc-0.0.1-SNAPSHOT-jar-with-dependencies.jar /usr/local/connector-plugins/rocketmq-connect-jdbc
146+
```
147+
148+
Use the standalone mode to start the RocketMQ Connect Runtime environment.
149+
150+
```bash
151+
$ cd distribution/target/rocketmq-connect-0.0.1-SNAPSHOT/rocketmq-connect-0.0.1-SNAPSHOT
152+
$ sh bin/connect-standalone.sh -c conf/connect-standalone.conf &
153+
```
154+
155+
The following indicates that RocketMQ connect runtime has run successfully:
156+
157+
![img](images/rocketmq_runtime.png)
158+
159+
**Step 4 | Test**
160+
161+
![img](images/rocketmq_test.png)
162+
163+
- Create a Mysql data table and initialize test data
164+
165+
- Create a mysql source and pull data from the test table
166+
167+
- Create an OpenMLDB sink and write the data pulled from the source to OpenMLDB
168+
169+
Initialize MySQL test data;
170+
171+
```sql
172+
SET NAMES utf8mb4;
173+
SET FOREIGN_KEY_CHECKS = 0;
174+
175+
176+
-- ----------------------------
177+
-- Table structure for employee_test
178+
-- ----------------------------
179+
DROP TABLE IF EXISTS `employee_test`;
180+
CREATE TABLE `employee_test` (
181+
`id` bigint NOT NULL AUTO_INCREMENT,
182+
`name` varchar(128) DEFAULT NULL,
183+
`howold` int DEFAULT NULL,
184+
`male` int DEFAULT NULL,
185+
`company` varchar(128) DEFAULT NULL,
186+
`money` double DEFAULT NULL,
187+
`begin_time` datetime DEFAULT NULL,
188+
PRIMARY KEY (`id`)
189+
) ENGINE=InnoDB AUTO_INCREMENT=15 DEFAULT CHARSET=utf8;
190+
191+
192+
-- ----------------------------
193+
-- Records of employee_test
194+
-- ----------------------------
195+
BEGIN;
196+
INSERT INTO `employee_test` VALUES (2, 'name-02', 19, 7, 'company', 32232, '2021-12-29 08:00:00');
197+
INSERT INTO `employee_test` VALUES (4, 'gjk', 25, 8, 'company', 3232, '2021-12-24 20:43:36');
198+
INSERT INTO `employee_test` VALUES (12, 'name-06', 19, 3, NULL, NULL, NULL);
199+
INSERT INTO `employee_test` VALUES (14, 'name-08', 25, 15, 'company', 32255, '2022-02-08 19:06:39');
200+
COMMIT;
201+
202+
203+
SET FOREIGN_KEY_CHECKS = 1;
204+
```
205+
206+
Create and start RocketMQ conect mysql source connector, the examples is as follows:
207+
208+
```bash
209+
curl -X POST -H "Content-Type: application/json" http://127.0.0.1:8082/connectors/jdbc-mysql-source-test
210+
-d '{
211+
"connector-class":"org.apache.rocketmq.connect.jdbc.connector.JdbcSourceConnector",
212+
"max-task":"1",
213+
"connection.url":"jdbc:mysql://127.0.0.1:3306",
214+
"connection.user":"*****",
215+
"connection.password":"*****",
216+
"table.whitelist":"test_database.employee_test",
217+
"mode": "incrementing", // Incremental pull method
218+
"incrementing.column.name":"id", // Specify fields for incremental pull
219+
"source-record-converter":"org.apache.rocketmq.connect.runtime.converter.JsonConverter"
220+
}'st
221+
```
222+
223+
Confirm task initiation and start pulling data:
224+
![img1](https://pic3.zhimg.com/80/v2-80395baf4060f32bb0e86d959ad6ecae_1440w.webp)
225+
226+
![img2](https://pic2.zhimg.com/80/v2-2e76f61a818c5bc31cdd18eca2a8b4ed_1440w.webp)
227+
228+
To create an OpenMLDB RocketMQ sink connector for writing data to the OpenMLDB table, please refer to the following information. (Note: The listening topic corresponds to the table name of the source pull table.)
229+
230+
```bash
231+
curl -X POST -H "Content-Type: application/json" http://127.0.0.1:8082/connectors/jdbc-openmldb-sink-test
232+
-d '{
233+
"connector-class":"org.apache.rocketmq.connect.jdbc.connector.JdbcSinkConnector",
234+
"max-task":"1",
235+
"connect-topicname":"employee_test",
236+
"connection.url":"jdbc:openmldb:///rocketmq_test?zk=127.0.0.1:2181&zkPath=/openmldb_cluster",
237+
"insert.mode":"INSERT",
238+
"db.timezone":"UTC",
239+
"table.types":"TABLE",
240+
"auto.create":"true",
241+
"source-record-converter":"org.apache.rocketmq.connect.runtime.converter.JsonConverter"
242+
}'
243+
```
244+
245+
By monitoring the data entry, we can verify the successful insertion into OpenMLDB. The details are as follows:
246+
247+
```sql
248+
set @@execute_mode='online';
249+
use rocketmq_test;
250+
select * from employee_test;
251+
```
252+

docs/zh/integration/online_datasources/rocketmq_connector.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# RocketMQ
22

3-
## 1. 为什么选择 RocketMQ OpenMLDB Connector
3+
## 为什么选择 RocketMQ OpenMLDB Connector
44

55
为了使 OpenMLDB 与 RocketMQ 拥有高效稳定的传输通道,RocketMQ OpenMLDB connector 具有诸多优秀特性,包括但不限于:
66

@@ -9,9 +9,9 @@
99
- **高可靠。**RocketMQ OpenMLDB Connector 集群部署的方式具备 Failover 能力,可以将有问题节点的任务调度到正常节点并保证集群负载均衡,使企业能**更专注和更高效**地探索数据的商业价值。
1010
- **低延时。秒级延迟**,满足实时数据及特征开发场景。
1111

12-
## 2. RocketMQ OpenMLDB Connector
12+
## RocketMQ OpenMLDB Connector
1313

14-
### 2.1. Connector 概述
14+
### Connector 概述
1515

1616
**定位**
1717

@@ -37,7 +37,7 @@ $ mvn clean package -Dmaven.test.skip=true
3737

3838
最终将编译好的插件包放在 RocketMQ connect 指定的加载地址。
3939

40-
### 2.2. Connector 使用演示
40+
### Connector 使用演示
4141

4242
**流程介绍**
4343

0 commit comments

Comments
 (0)