[OSPP 2023] DolphinScheduler idea discussion thread

[zhongjiajie]

DolphinScheduler has participated in the OSPP 2023 event [1] and can publish 8 projects.

If you have good ideas, you can post your ideas here for discussion. The idea includes 3 contents

• Project Description
• Project Output Standard
• Technical Requirements

Deadline April 20, 2023.

Last year project is in here: #9723
Mentor guide: https://summer-ospp.ac.cn/help/en/mentor/

Our community have 8 max topic for this year.

[brave-lee]

Reform dependence mode: #11970

• Description:
  Present situation: The current dependency is on a single task and All. In fact, All depends on workflow, not all tasks
  Planned reform:

1. The mode of relying on a single task remains unchanged. All is changed to rely on all tasks in the workflow, and a new type of workflow dependency is added.
2. The front end adds a radio box in the dependency area to select dependent tasks or workflows. If you select dependent tasks, the task selection box will be displayed. If you now rely on workflows, the task selection box will be hidden
3. For the backend, All is still 0 for the task code. When workflow is relied on, the corresponding task code can be represented by - 1 or other

• Output criteria:
  Complete requirements and submit it to the development branch

• Technical Requirements
  Familiar with Java language

---

• 描述：
  当前状况：当前的依赖是依赖单个任务和All，这里的All其实是依赖工作流，并不是依赖所有的任务
  计划改造：
  1、依赖单个任务的方式不变，All改造成依赖工作流中所有任务，新增一种类型依赖工作流。
  2、前端在依赖的地方增加个单选框，选择依赖任务或者工作流，如果选择依赖任务，任务的选择框显示出来，如果现在依赖工作流，任务的选择框隐藏
  3、对于后端，All对于的任务code还是0，依赖工作流时，对应的任务code可使用-1或其他表示

• 产出要求：
  完成需求，并将代码提交到开发分支

• 技术要求：
  熟悉 Java 语言

[zhongjiajie]

LGTM, but should add more description about this topic, see more detail in last year desc #9723

[SbloodyS]

Fault tolerance optimization #9664

• Difficulty: Advanced

Dolphinscheduler master/worker fault tolerance optimization

• Project Description
  When the current master/worker node hangs. It will perform fault-tolerant operation and restart the failed tasks after the master/worker node is restarted. This will cause the Shell/Spark/Flink task that the user is executing in the Linux process or yarn application to be repeated running.

Therefore, I suggest that after the master/worker is restarted, if these tasks are in normal execution, there will not be re executed.

1. Shell node because the execution path of the process of the current DS task node is unique. Therefore, we can query whether the process exists through ps -ef | grep XXX. If so, the task will not be restarted.
2. The task node related to yarn checks whether the task is running through yarn application -status application_id. If so, the task will not be restarted.

• Project Output Standard

1. Shell nodes are able to obtain their previous running status normally during the recovery of fault tolerance.
2. Yarn related tasks are able to obtain their previous running status normally during the recovery of fault tolerance.

• Technical Requirements

1. Familiar with Linux process management
2. Understand Hadoop Yarn components and be able to quickly learn the use of API according to official documents
3. Familiar with Java language

恢复容错优化

• 难度: 进阶

Dolphinscheduler master/worker 恢复容错优化

• 项目描述
  当前主/工作节点挂起时。它将执行容错操作，并在主/工作节点重新启动后重新启动失败的任务。这将导致用户在Linux进程或yarn应用程序中执行的Shell/Spark/Flink任务重复运行。

因此，我建议在主/工作程序重新启动后，如果这些任务处于正常执行状态，则不会重新执行。

1. Shell节点，因为当前DS任务节点的进程的执行路径是唯一的。因此，我们可以通过 ps-ef | grep XXX 查询该进程是否存在。如果是，则不会重新启动任务。
2. Yarn 相关的任务节点通过 yarn application-status application_id 检查任务是否正在运行。如果是，则不会重新启动任务。

• 项目产出要求

1. 在容错恢复过程中，Shell节点能够正常获取其先前的运行状态。
2. 在容错恢复过程中，Yarn 相关的任务能够正常获取其先前的运行状态。

• 技术要求

1. 熟悉 Linux 进程管理
2. 理解 Hadoop Yarn 组件并且能根据官方文档快速学习使用相应API
3. 熟悉 Java 语言

[SbloodyS]

Cool, this is a good one! Besides, currently there is a potential bug during failover process: #13355 But I'm not sure whether fixing this bug is suitable to be part of the proposal because I think it is kind of difficult.

Considering the student's abilities and time, I think this is a bit difficult.

[Radeity]

It seems that after #13798, we can directly take over running task instances on Worker when doing Master failover.

[SbloodyS]

It seems that after #13798, we can directly take over running task instances on Worker when doing Master failover.

Yes. What I purpose is a more elegant and idempotent approach.

[zhongjiajie]

LGTM, can we add more descriptions for this topic?

[SbloodyS]

LGTM, can we add more descriptions for this topic?

Sure. Will add in these few days.

[EricGao888]

Upgrade Datasource Center to Connection Center

Description

1. DolphinScheduler has a component called Datasource Center to manage external connections for sql tasks, such as mysql, hive, spark, etc.
2. However, not only sql tasks but also some other kinds of DS task plugins need external connections, such as AWS EMR task, Zeppelin task, K8S task, etc. We could enrich the scenarios where Datasource Center is needed for the management of connections, especially those with credentials, for external system and upgrade it to Connection Center.

related issue:

• [DSIP-19][Feature] Add connection center feature for DS #10283

Goal

1. Change the name of Datasource Center to Connection Center.
2. Refactor part of AWS EMR, Zeppelin, K8S, Sagemaker task plugins so that users could manage the external connections in Connection Center.
3. Remove Cluster Management and K8S Namespace Manage in Security Center, because it is not the job for a big data orchestration tool to manage K8S clusters. Users could configure K8S connections in Connection Center for K8S task plugin.

Skills:

[Backend] Familiar with Java Programming.
[Backend] Have some knowledge in Spring Boot Framework.
[Frontend] Have some knowledge in VUE

[zhongjiajie]

LGTM, and for more detail description, you can see #9723