Documentation translated into English

Author: yuankang134

en_US/Design_Documentation/FlowExecution/images/flowexecution.drawio.png

@@ -0,0 +1,51 @@
+# DSS-UserGuide module design
+
+### Introduce
+
+The DSS user manual module is a newly added function module of DSS1.0, which is used to provide user guidance for DSS users. It contains many problems and solutions encountered during the use of DSS, and also includes some function points. Users can self-service search for solutions to problems encountered. It can also be used to associate error codes in the later stage, which supports directly locating the solutions that have been entered in the knowledge base after the pop-up error codes. The guide module stores the files in the fields of the table in the form of html. It needs to parse the md file and convert it into html. Since some files have links and need to be jumped, it is necessary to build a gitbook to display and manage these documents. In order to be efficient To synchronize the dss-guide-user module, package the files on gitLab, upload and decompress them to the specified directory of the server where gitbook is located, and scan the specified directory regularly through guide-user to achieve the purpose of synchronization.
+
+## Introduction to the main modules of dss_guide
+
+The DSS_Guide module mainly contains the definitions of Restful, Service, Dao, and Entity.
+
+### GuideGroupService
+
+It is used to solve GuideGroup's ability to add, query, modify, save, delete, etc. It also has the ability to synchronize Summary.md. The guide module can parse this file, and then locate the file that needs to be read and write it to the database regularly according to the configuration paths of the parsed directories at all levels in the file, so as to complete the synchronization. When the service is running on other servers, In order to avoid repeated installation of gitbook, the guide module needs to pass the ip of the server where the configuration file is located, and then automatically synchronize the file to the server where the guide module is located for display.
+
+### GuideContentService
+
+It is used to handle the save, query, update and delete operations of GuideContent.
+
+### GuideChapterService
+
+It is used to deal with the specific content of manual chapters, including chapter search, ID-based query, deletion, saving, etc.
+
+### GuideCatalogService
+
+It is used to synchronize the knowledge base, support batch insertion of directory content, and implement operations such as saving, deleting, and querying the directory structure classification.
+
+
+### Core flow chart
+
+![](./images/16559707626688.png)
+
+
+### data structure
+
+![](./images/1653309930194.png)
+
+### dss_guide_group
+
+Divide the grouping of dss_guide, including group_id, path (access path), title, etc.
+
+### dss_guide_chapter
+
+Used to store the detailed content of the dss_guide chapter, including catalog_id, title, content, content_html. Associate with the content of dss_guide_catalog.
+
+### dss_guide_content
+
+It is used to store the description content after grouping, and will be planned under the corresponding group. Contains title, type, content, content_html, etc.
+
+### dss_guide_catalog
+
+It is used to classify the content of dss_guide, which is equivalent to the directory structure of the knowledge base and has a hierarchical directory relationship.

en_US/Design_Documentation/Orchestrator/images/Create_an_orchestration_sequence_diagram.png

@@ -0,0 +1,81 @@
+# DSS access scheduling system
+
+## background
+
+There are many batch timing scheduling systems currently used in the field of big data, such as Azkaban, Dophinscheduler, Airflow, etc. DataSphere Studio (DSS) supports publishing workflows designed by users to different scheduling systems for scheduling. Currently, the default support for publishing to Azkaban. The user has completed the DAG design of the workflow in DSS, including workflow resource file loading, workflow parameter setting, data import, data quality check, node code writing, visual report design, email output, node resource file upload and running parameter setting After the operation, it can be debugged and executed in DSS. After verifying that the execution of all nodes is correct, it will be released to the scheduling system, and the scheduling system will schedule and execute regularly according to the configuration of the scheduled task.
+
+### release method
+
+The DSS integration new scheduling system needs to be accessed by AppConn. Users need to define the corresponding XXXSchedulerAppConn according to different scheduling systems, and the transformation integration specification and the structured integration specification are defined in the SchedulerAppConn. The transformation integration specification contains the transformation of DSS project-level content and DSS workflow-level content to third-party scheduling systems. The access of DSS to the dispatching system can be divided into the following two types:
+
+1. Engineering level release
+
+It refers to converting all the workflows in the project, and uniformly packaging and uploading the converted content to the scheduling system. There are mainly the ProjectPreConversionRel interface, which defines the workflow that needs to be converted in the project.
+
+2. Workflow level release
+
+It refers to the conversion according to the granularity of the workflow, and only the content of the workflow is packaged and uploaded to the scheduling system. The current workflow definition of DSS is stored in the BML file in the form of Json, and the metadata information of the workflow is stored in the database.
+
+## The main steps
+
+### Parser
+
+JsonToFlowParser is used to convert the Json of the workflow into Workflow. Workflow is the standard format for operating workflow in DSS. It includes the node information of the workflow, the edge information of the workflow, the parent workflow, the child workflow, the workflow resource file, Workflow property file, workflow creation time, update time, workflow user, workflow proxy user, workflow metadata information such as name, ID, description, type, whether it is a root workflow, etc. These are parsed according to the content of Json and converted into DSS operable Workflow objects, such as AzkabanWorkflow and DolphinSchedulerWorkflow.
+
+### Converter
+
+Turn the Workflow of DSS into a workflow that can be recognized by the access scheduling system. Each scheduling system has its own definition for the workflow. If you convert the nodes of the DSS workflow into Azkaban's job format files, or into DolphinScheduler's tasks, you can also convert them in reverse, and convert the workflow of the scheduling system into a workflow that can be loaded and displayed by DSS, and the dependencies of the workflow. , the connection of the node is converted into the dependency of the corresponding scheduling system. You can also check whether the workflow nodes under the project have nodes with the same name in Converter. For example, the use of nodes with the same name is not allowed in Azkaban's scheduling system.
+
+WorkflowConVerter defines the workflow conversion output directory structure, including workflow storage directory, workflow resource file storage, and establishment of sub-workflow storage directory. For example, Azkaban also includes creating a project conversion directory in the project-level conversion operation, and establishing a workflow conversion directory according to the workflow situation in the project. Convert Workflow to dolphinSchedulerWorkflow or ScheduleWorkFlow in convertToRel
+
+NodeConverter defines the output content of node conversion: such as Azkaban's ConvertNode, it will convert the node content of the workflow into the corresponding Job file content. Including the name, type, dependency of the conversion node, the execution command of the node (depending on linkis-jobtype parsing), the configuration parameters of the node, the label of the node, etc. Finally, it is stored in the format defined by the Job file. The Converter of DolphinScheduler converts the nodes in DSS into tasks in DolphinScheduler, and builds the execution script of Shell type Task, and converts the node content of DSS into the parameters required for the execution of the custom dss-dolphinscheduler-client.sh script.
+
+```--java
+            addLine.accept("LINKIS_TYPE", dssNode.getNodeType());   //Workflow Node Type
+            addLine.accept("PROXY_USER", dssNode.getUserProxy());   //proxy user
+            addObjectLine.accept("JOB_COMMAND", dssNode.getJobContent()); //Excuting an order
+            addObjectLine.accept("JOB_PARAMS", dssNode.getParams());      //Node execution parameters
+            addObjectLine.accept("JOB_RESOURCES", dssNode.getResources()); //Node execution resource file
+            addObjectLine.accept("JOB_SOURCE", sourceMap);                 //Node's source information
+            addLine.accept("CONTEXT_ID", workflow.getContextID());         //context ID
+            addLine.accept("LINKIS_GATEWAY_URL", Configuration.getGateWayURL());  //linkis gateway address
+            addLine.accept("RUN_DATE", "${system.biz.date}");                    //run date variable
+```
+
+### Tunning
+
+It is used to complete the overall adjustment operation before the project is released. In the implementation of Azkaban, the path setting of the project and the storage path setting of the workflow are mainly completed.Because at this time it is possible to operate the project=》workflow=》sub-workflow，It is convenient for setting operation from outside to inside. For example, the storage of workflow depends on the storage location of the project, and the storage of child workflow depends on the location of the parent workflow. The child node calculation is completed in FlowTuning, and the end node is automatically added.
+
+## Scheduling the AppConn implementation
+
+### AbstractSchedulerAppConn
+
+The abstract class of scheduling AppConn, the new scheduling system AppConn access can directly inherit this abstract class, it implements the SchedulerAppConn interface, and inherits AbstractOnlySSOAppConn, to get through the SSO login between DSS and the scheduling system. For example, the already integrated DolphinSchedulerAppConn and SchedulisAppConn both inherit this abstract class.
+
+This abstract class contains two types of Standard
+
+The first is ConversionIntegrationStandard to support workflows that transform DSS orchestration into a scheduling system
+
+The second is SchedulerStructureIntegrationStandard, a structured integration specification for DSS and scheduling systems
+
+### ConversionIntegrationStandard
+
+Conversion integration specification for scheduling systems, including DSSToRelConversionService for converting DSS orchestration into scheduling system workflows. An interface is also reserved to support converting the workflow of the scheduling system into the orchestration of DSS
+
+### AbstractSchedulerStructureIntegrationStandard
+
+The scheduling system organizational structure integration specification is specially used for the organizational structure management of the scheduling system, mainly including engineering services and orchestration services.
+
+### ProjectService
+
+* The unified creation, update, deletion and duplicate checking operations of the project are realized.
+* It is used to open up the engineering system of DSS engineering and access third-party application tools, and realize the collaborative management of engineering.
+* If the scheduling system needs to open up the engineering system with DSS, it needs to implement all the interfaces of engineering services in the structured integration specification.
+
+### OrchestrationService
+
+The orchestration service is used for the unified orchestration specification of the scheduling system, and has the following functions:
+
+* Unified orchestration specification, specially used to open up the orchestration system of DSS and SchedulerAppConn (scheduling system).
+* For example: connect DSS workflow and Schedules workflow.
+* Please note that if the docking SchedulerAppConn system does not support management workflow itself, you do not need to implement this interface.

en_US/Design_Documentation/UserGuide/DSS-UserGuide_Module_Design.md

@@ -0,0 +1,29 @@
+DSS-AppConn Design Documentation
+------
+## Introduce
+    The principle of the original AppJoint is to define a top-level interface AppJoint. The third party implements this interface and stores its own connection information in the DSS table, and implements a "proxy service" in DSS that communicates with the third-party system. At the initial stage of initialization , creating an instance of the service through the reflection mechanism, and using the connection information in the table, DSS can use the "proxy service" to establish HTTP communication with the third-party system, thereby invoking the third-party system. However, there are shortcomings in the design of AppJoint. Each connected application instance needs to generate an AppJoint instance. Different instances of the same application are not logically associated. The application instance AppConn of each system is DSS1.0. The top-level interface, in DSS1.0, its own orchestration mode, workflow, single-task node, etc., are all instances of AppConn. In addition, third-party systems that access DSS need to implement the AppConn interface to implement DSS. Integrate with third-party systems to call third-party applications. Logically, AppConn has a higher abstraction logic than AppJoint. AppConn is similar to a class instance, while AppJoint is similar to an instance.
+
+### Introduction to related modules
+|Level 1 Module | Level 2 Module | Function Introduction|
+|-------------|-----------|----------------|
+|dss-appconn|appconns|Access DSS to implement AppConn related specification implementation code|
+|           |dss-appconn-core|Appconn interface and basic class definition|
+|           |dss-appconn-loader|Instantiation, loading and assembly of the AppConn compiled package of the connected application|
+|           |dss-appconn-manager|Interact with framework modules and manage related AppConn instance information|
+|           |dss-scheduler-appconn|Abstract AppConn Definition for Scheduling System Implementation|
+|           |linkis-appconn-engineplugin|Implement the relevant specifications of linkis appconn and open up the interaction between DSS AppConn and Linkis|  
+
+
+
+| Core interface/class            | Core functions                           |
+|---------------------------|------------------------------|
+| DSSDictionaryRestful、DSSDictionaryServiceImpl           |  Provide dictionary information acquisition interface, query corresponding records from dictionary table through parameter key or parentKey     |
+| DSSWorkspacePrivRestful、DSSWorkspacePrivServiceImpl    | Provides viewing and editing functions for the permission information of the menu component of the workspace role                 |
+| DSSWorkspaceRestful、DSSWorkspaceServiceImpl            | Provides basic functional interfaces of workspace, such as creating workspace, obtaining workspace list, obtaining permission information of menu components, etc.               |
+| DSSWorkspaceRoleRestful、DSSWorkspaceRoleServiceImpl   | Provides query and creation interfaces for workspace roles              |
+| DSSWorkspaceUserRestful、DSSWorkspaceUserServiceImpl   | Provides an interface for adding, deleting, modifying, and querying workspace users          |
+
+### AppConn Architecture Diagram
+![](./images/appconn_class_uml.png)
+![](./images/appconn_structure.png)  
+![](./images/appconn_load_process.png)