Skip to content

Commit 20075b3

Browse files
Davidhua1996Davidhua1996
authored
Merge pull request #26 from WeDataSphere/dev-0.2.0-docs-appconn
[Docs]Streamis document
2 parents 8ee1503 + 9f8f58d commit 20075b3

24 files changed

+534
-58
lines changed

‎README-ZH.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ Dimension节点、Transform节点、Sink节点 和 [Visualis](https://github.com
4545

4646
![流式应用配置](docs/images/流式应用配置.png)
4747

48-
       更多功能，请参考：[Streamis 用户手册](docs/zh_CN/0.1.0/使用文档/Streamis用户手册.md)。
48+
       更多功能，请参考：[Streamis 用户手册](docs/zh_CN/0.2.0/使用文档/Streamis用户手册.md)。
4949

5050
----
5151

@@ -72,20 +72,20 @@ Dimension节点、Transform节点、Sink节点 和 [Visualis](https://github.com
7272

7373
## 编译和安装部署
7474

75-
       请参照 [编译指引](docs/zh_CN/0.1.0/development/Streamis编译文档.md) ，用于编译 Streamis。
75+
       请参照 [编译指引](docs/zh_CN/0.2.0/development/Streamis编译文档.md) ，用于编译 Streamis。
7676

7777
       在安装Streamis之前，请先安装 Linkis1.0.3 和 DSS1.0.1，可参照：[DSS + Linkis 安装部署文档](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/zh_CN/%E5%AE%89%E8%A3%85%E9%83%A8%E7%BD%B2/DSS%E5%8D%95%E6%9C%BA%E9%83%A8%E7%BD%B2%E6%96%87%E6%A1%A3.md)
7878

7979
       同时，请确保 Linkis Flink 引擎已可正常使用，可参照：[Linkis Flink 引擎插件安装文档](https://github.com/WeBankFinTech/Linkis-Doc/blob/master/zh_CN/Deployment_Documents/EngineConnPlugin%E5%BC%95%E6%93%8E%E6%8F%92%E4%BB%B6%E5%AE%89%E8%A3%85%E6%96%87%E6%A1%A3.md)
8080

81-
       最后，请参考 [Streamis 安装部署文档](docs/zh_CN/0.1.0/Streamis安装文档.md) ，用于安装部署 Streamis。
81+
       最后，请参考 [Streamis 安装部署文档](docs/zh_CN/0.2.0/Streamis安装文档.md) ，用于安装部署 Streamis。
8282

8383

8484
----
8585

8686
## 示例和使用指引
8787

88-
       请到 [用户使用文档](docs/zh_CN/0.1.0/使用文档/Streamis用户手册.md) ，了解如何快速使用 Streamis。
88+
       请到 [用户使用文档](docs/zh_CN/0.2.0/使用文档/Streamis用户手册.md) ，了解如何快速使用 Streamis。
8989

9090
----
9191

‎docs/en_US/0.1.0/StreamisAppConnDesignDocument.md‎

Lines changed: 209 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,209 @@
1+
# Streamis access AppConn
2+
3+
## Overall flow chart
4+
![Streamis access DSS](../../images/zh_CN/streamis_appconn.png)
5+
6+
## DSS project APPCONN plug-in streamis-appconn
7+
8+
### The configuration table
9+
Configure the following three tables:dss_appconn、dss_workspace_menu_appconn、dss_appconn_instance，appconn_name for realTimeJobCenter is appconn accessed by the graphical interface，appconn_name for streamis is appconn accessed by the API，The StreamisAppConn object is instantiated based on the configuration information in the table when DSS is started。
10+
```roomsql
11+
select @old_dss_appconn_id:=id from `dss_appconn` where `appconn_name` = 'streamis';
12+
13+
delete from `dss_workspace_menu_appconn` WHERE `appconn_id` = @old_dss_appconn_id;
14+
delete from `dss_appconn_instance` where `appconn_id` = @old_dss_appconn_id;
15+
delete from `dss_appconn` where `appconn_name`='streamis';
16+
17+
select @old_jobcenter_dss_appconn_id:=id from `dss_appconn` where `appconn_name` = 'realTimeJobCenter';
18+
19+
delete from `dss_workspace_menu_appconn` WHERE `appconn_id` = @old_jobcenter_dss_appconn_id;
20+
delete from `dss_appconn_instance` where `appconn_id` = @old_jobcenter_dss_appconn_id;
21+
delete from `dss_appconn` where `appconn_name`='realTimeJobCenter';
22+
23+
INSERT INTO dss_appconn
24+
(appconn_name, is_user_need_init, `level`, if_iframe, is_external, reference, class_name, appconn_class_path, resource)
25+
VALUES('streamis', 0, 1, 1, 1, NULL, 'com.webank.wedatasphere.streamis.dss.appconn.StreamisAppConn', NULL, NULL);
26+
INSERT INTO dss_appconn
27+
(appconn_name, is_user_need_init, `level`, if_iframe, is_external, reference, class_name, appconn_class_path, resource)
28+
VALUES('realTimeJobCenter', 0, 1, 1, 1, 'sso', '', NULL, NULL);
29+
30+
select @dss_appconn_id:=id from `dss_appconn` where `appconn_name` = 'streamis';
31+
select @jobcenter_dss_appconn_id:=id from `dss_appconn` where `appconn_name` = 'realTimeJobCenter';
32+
33+
INSERT INTO dss_workspace_menu_appconn
34+
(appconn_id, menu_id, title_en, title_cn, desc_en, desc_cn, labels_en, labels_cn, is_active, access_button_en, access_button_cn, manual_button_en, manual_button_cn, manual_button_url, icon, `order`, create_by, create_time, last_update_time, last_update_user, image)
35+
VALUES(@jobcenter_dss_appconn_id, 1, 'StreamSQL development', 'StreamSQL开发', 'Real-time application development is a streaming solution jointly built by WeDataSphere, Boss big data team and China Telecom ctcloud Big data team.', '实时应用开发是微众银行微数域(WeDataSphere)、Boss直聘大数据团队 和 中国电信天翼云大数据团队 社区联合共建的流式解决方案，以 Linkis 做为内核，基于 Flink Engine 构建的批流统一的 Flink SQL，助力实时化转型。',
36+
'streaming, realtime', '流式,实时', 0, 'under union construction', '联合共建中', 'related information', '相关资讯', 'http://127.0.0.1:8088/wiki/scriptis/manual/workspace_cn.html', 'shujukaifa-logo', NULL, NULL, NULL, NULL, NULL, 'shujukaifa-icon');
37+
38+
INSERT INTO dss_appconn_instance
39+
(appconn_id, label, url, enhance_json, homepage_uri)
40+
VALUES(@dss_appconn_id, 'DEV', 'http://Streamis_INSTALL_IP:Streamis_INSTALL_PORT/', '', 'http://Streamis_INSTALL_IP:Streamis_INSTALL_PORT/#/realTimeJobCenter');
41+
42+
INSERT INTO dss_appconn_instance
43+
(appconn_id, label, url, enhance_json, homepage_uri)
44+
VALUES(@jobcenter_dss_appconn_id, 'DEV', 'http://Streamis_INSTALL_IP:Streamis_INSTALL_PORT/#/realTimeJobCenter', NULL, NULL);
45+
```
46+
47+
### Concrete implementation description
48+
StreamisAppConn extends AbstractOnlySSOAppConn implements SecondlyAppConn，Override method to create StreamisStructureIntegrationStandard.StreamisAppConn currently only implements section-free login and organizational structure specification capabilities。
49+
50+
To create a StreamisProjectService by rewriting the methods in StreamisStructureIntegrationStandard, four methods need to be rewritten internally, and four operation classes will be created: StreamisProjectSearchOperation, StreamisProjectCreationOperation, StreamisProjectUpdateOperation, and StreamisPrejectDeleteOperation. The operation class calls the streamis application through HTTP to query, create, modify and delete project respectively, and synchronize the DSS project information to Streamis.
51+
52+
- When DSS creates a project, it will first call the query operation to query whether the same project name already exists in streamis. If it does, a prompt will pop up. If it does not exist, it will continue to call the create operation to create a new project in streamis;
53+
- When DSS modifies a project, it will call the modify operation to update the project information in streamis;
54+
- When DSS deletes an item, it will call the delete operation to delete the item information in streamis and change the deletion mark.
55+
56+
## API
57+
1 API name: query project
58+
- API path：GET/streamis/project/searchProject
59+
- Request parameters
60+
61+
|Parameter name |Whether it is necessary |Example |remarks |
62+
|-------------|---------|-------|--------|
63+
|projectName |yes | | |
64+
65+
- Return data
66+
67+
|name |type |Whether it is necessary |Default |remarks |
68+
|-------------|--------|---------|---------|--------|
69+
|method |string |no |
70+
|status |number |yes |
71+
|message |string |no |
72+
|data |object |yes |
73+
|- projectId |number |yes |
74+
75+
2 API name：create project
76+
- API path：GET/streamis/project/createProject
77+
- Request parameters
78+
79+
|Parameter name |Whether it is necessary |Example |remarks |
80+
|-------------|---------|-------|--------|
81+
|projectName |yes | | |
82+
|workspaceId |no | | |
83+
|releaseUsers |no | | |
84+
|editUsers |no | | |
85+
|accessUsers |no | | |
86+
87+
- Return data
88+
89+
|name |type |Whether it is necessary |Default |remarks |
90+
|----------|--------|---------|---------|--------|
91+
|method |string |no |
92+
|status |number |yes |
93+
|message |string |no |
94+
|data |object |yes |
95+
|- projectId |number |yes |
96+
|- projectName |string |no |
97+
98+
3 API name：update project
99+
- API path：GET/streamis/project/updateProject
100+
- Request parameters
101+
102+
|Parameter name |Whether it is necessary |Example |remarks |
103+
|-------------|---------|-------|--------|
104+
|projectId |yes | | |
105+
|projectName |yes | | |
106+
|workspaceId |no | | |
107+
|releaseUsers |no | | |
108+
|editUsers |no | | |
109+
|accessUsers |no | | |
110+
111+
- Return data
112+
113+
|name |type |Whether it is necessary |Default |remarks |
114+
|----------|--------|---------|---------|--------|
115+
|method |string |no |
116+
|status |number |yes |
117+
|message |string |no |
118+
|data |object |no |
119+
120+
4 API name：delete project
121+
- API path：GET/streamis/project/deleteProject
122+
- Request parameters
123+
124+
|Parameter name |Whether it is necessary |Example |remarks |
125+
|-------------|---------|-------|--------|
126+
|projectId |yes | | |
127+
|projectName |no | | |
128+
129+
- Return data
130+
131+
|name |type |Whether it is necessary |Default |remarks |
132+
|----------|--------|---------|---------|--------|
133+
|method |string |no |
134+
|status |number |yes |
135+
|message |string |no |
136+
|data |object |no |
137+
138+
## Streamis project streamis-project-server
139+
140+
### Related table operation
141+
There are 2 tables involved in streamis, including linkis_ stream_ Project and linkis_ stream_ project_ privilege.
142+
```roomsql
143+
--table already exists
144+
CREATE TABLE `linkis_stream_project` (
145+
`id` bigint(20) NOT NULL AUTO_INCREMENT,
146+
`workspace_id` bigint(20) DEFAULT NULL,
147+
`name` varchar(100) DEFAULT NULL,
148+
`create_by` varchar(50) DEFAULT NULL,
149+
PRIMARY KEY (`id`) USING BTREE
150+
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 ROW_FORMAT=COMPACT COMMENT='project table';
151+
152+
--newly added table
153+
CREATE TABLE `linkis_stream_project_privilege` (
154+
`id` bigint(20) NOT NULL AUTO_INCREMENT,
155+
`project_id` bigint(20) NOT NULL,
156+
`user_name` varchar(100) NOT NULL,
157+
`privilege` tinyint(1) DEFAULT '0' NOT NULL COMMENT '1:RELEASE ，2:EDIT ，3:ACCESS',
158+
PRIMARY KEY (`id`) USING BTREE
159+
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COMMENT='project privilege table';
160+
161+
--newly added field
162+
ALTER TABLE `linkis_stream_project` ADD create_time datetime DEFAULT NULL;
163+
ALTER TABLE `linkis_stream_project` ADD last_update_by varchar(50) DEFAULT NULL;
164+
ALTER TABLE `linkis_stream_project` ADD last_update_time datetime DEFAULT NULL;
165+
ALTER TABLE `linkis_stream_project` ADD is_deleted tinyint unsigned DEFAULT 0;
166+
```
167+
168+
### Concrete implementation description
169+
- The query operation will obtain the request parameter projectName in the table links_ stream_project Query the corresponding ID. If the query operation is successful, the returned status is 0, and the projectid is the queried ID. if the query result is empty, the projectid is null;
170+
- The creation operation will insert the project information (projectName、workspaceId) in the request parameters into the project table linkis_stream_project and auto increment the generated ID, associate the user in the permission information data (releaseUsers、editUsers、accessUsers) in the request parameters with the generated ID of the project table, and insert the table linkis_stream_project_privilege, the status value returned from the successful creation operation is 0, and the project table generation ID will be returned as the value of projectId;
171+
- The modification operation will update the request information data to the table linkis_stream_project and linkis_stream_project_privilege, the status value returned successfully is 0;
172+
- In the delete operation, the is_deleted field of the table linkis_stream_project will be marked as 1 according to the projectId. The relevant data in the table linkis_stream_project_privilege will be deleted, the status value returned successfully is 0.
173+
174+
# Authentication
175+
176+
## Authentication flow chart
177+
![Streamis project authentication operation](../../images/zh_CN/streamis_project_privilege.png)
178+
179+
### edit privilege API：
180+
181+
|RequestMethod |API path |name |
182+
|------|----------------------------------------------------------|-----------------|
183+
|POST |/streamis/streamProjectManager/project/files/upload |Project resource file - Import |
184+
|GET |/streamis/streamProjectManager/project/files/delete |Delete all versions of the file under the project |
185+
|GET |/streamis/streamProjectManager/project/files/version/delete |Delete version file |
186+
|GET |/streamis/streamProjectManager/project/files/download |Task details - Download |
187+
|POST |/streamis/streamJobManager/job/upload |Upload file |
188+
|POST |/streamis/streamJobManager/job/execute |start-up |
189+
|GET |/streamis/streamJobManager/job/stop |stop |
190+
|PUT |/streamis/streamJobManager/job//snapshot/{jobId:\w+} |Snapshot generation |
191+
|GET |/streamis/streamJobManager/config/json/{jobId:\w+} |Configuration - save |
192+
|POST |/streamis/streamJobManager/job/bulk/execution |Batch start |
193+
|POST |/streamis/streamJobManager/job/bulk/pause |Batch stop |
194+
195+
196+
### access privilege API：
197+
198+
|RequestMethod |API path |name |
199+
|------|----------------------------------------------------------|-------------|
200+
|GET |streamis/streamJobManager/job/list |Query the jobs that the current user can view |
201+
|GET |/streamis/streamProjectManager/project/files/list |prokect resource document |
202+
|GET |/streamis/streamProjectManager/project/files/version/list |Obtain all versions of the file under the project |
203+
|GET |/streamis/streamJobManager/job/version |Query job version |
204+
|GET |/streamis/streamJobManager/job/execute/history |Job execution history |
205+
|GET |/streamis/streamJobManager/job/progress |Get the latest task status of the current version of the job |
206+
|GET |/streamis/streamJobManager/job/jobContent |Task details |
207+
|GET |/streamis/streamJobManager/job/logs |Get log |
208+
|POST |/streamis/streamJobManager/config/json/{jobId:\w+} |Get task configuration |
209+
|GET |/streamis/streamJobManager/config/view |Query the current job configuration information |

‎docs/images/zh_CN/dss_streamis_project.png‎

312 KB
Loading

‎docs/images/zh_CN/eureka_streamis.png‎

-118 KB
Loading

‎docs/images/zh_CN/streamis_appconn.png‎

76.7 KB
Loading

‎docs/images/zh_CN/streamis_project_privilege.png‎

108 KB
Loading

docs/zh_CN/0.1.0/AppConn的使用.md renamed to docs/zh_CN/0.2.0/AppConn的使用.md

File renamed without changes.

0 commit comments

Comments
 (0)