[DSIP-83][datasource-plugin] Add DolphinDB datasource by eco8848 · Pull Request #16962 · apache/dolphinscheduler

eco8848 · 2025-01-15T06:16:13Z

Purpose of the pull request

This pull request adds DolphinDB datasource.

close #16818

Brief change log

Add DolphinDB datasource module.
Add DolphinDB documentation.
Add DolphinDB constant definitions, connection parameters, data source processors and other components.
Add DolphinDB JDBC driver dependency.

Verify this pull request

This change added tests and can be verified as follows:

Add DataSourceUtilsTest class to test the datasource utility methods.
Add DolphinDBDataSourceChannelFactoryTest class to verify the creation methods of the datasource channel factory.
Add DolphinDBDataSourceChannelTest class to validate the creation of the datasource client through the channel.
Add DolphinDBDataSourceProcessorTest class to test various functionalities of the datasource processor.

boring-cyborg · 2025-01-15T06:16:16Z

Thanks for opening this pull request! Please check out our contributing guidelines. (https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/join/pull-request.md)

eco8848 · 2025-01-15T06:27:53Z

Related DSIP #16818

.../apache/dolphinscheduler/plugin/datasource/dolphindb/param/DolphinDBDataSourceProcessor.java

+
+        String[] hostSeperator = connectionParams.getAddress().split(Constants.DOUBLE_SLASH);
+        String[] hostPortArray = hostSeperator[hostSeperator.length - 1].split(Constants.COMMA);
+        dataSourceParamDTO.setPort(Integer.parseInt(hostPortArray[0].split(Constants.COLON)[1]));


SbloodyS

Please add dolphinscheduler-datasource-dolphindb to https://github.com/apache/dolphinscheduler/blob/dev/config/plugins_config

dolphinscheduler-bom/pom.xml

SbloodyS · 2025-02-04T09:13:04Z

Please fix CI errors. @eco8848

eco8848 · 2025-02-11T03:57:48Z

I have referred to the official license requirements at https://dolphinscheduler.apache.org/zh-cn/docs/3.2.2/contribute/join/DS-License. It is necessary to add a description of the newly introduced third-party library in tools/dependencies/known-dependencies.txt, but it seems to conflict with #16838. What should I do now to comply with the requirements? @SbloodyS

… build config

eco8848 · 2025-02-13T01:59:10Z

The CI errors have already been fixed. @SbloodyS

eco8848 · 2025-02-28T08:34:49Z

I saw that there is a new check requires a milestone label. So is there a general plan for the integration of this PR? @SbloodyS

sonarqubecloud · 2025-03-17T09:55:22Z

Quality Gate passed

Issues
30 New issues
0 Accepted issues

Measures
0 Security Hotspots
64.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Copilot

Pull Request Overview

This pull request introduces a new DolphinDB datasource plugin to support DolphinDB connections in the system. Key changes include:

Implementation of the DolphinDB datasource module with connection parameters, processors, and client/channel classes.
Addition of comprehensive test cases for various components of the datasource.
Updates to documentation and dependency management for the new JDBC driver.

Reviewed Changes

Copilot reviewed 23 out of 28 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
dolphinscheduler-datasource-dolphindb/src/test/java/org/apache/dolphinscheduler/plugin/datasource/dolphindb/utils/DataSourceUtilsTest.java	Added tests for datasource utilities.
dolphinscheduler-datasource-dolphindb/src/test/java/org/apache/dolphinscheduler/plugin/datasource/dolphindb/param/DolphinDBDataSourceProcessorTest.java	Added tests for datasource processor functionality.
dolphinscheduler-datasource-dolphindb/src/test/java/org/apache/dolphinscheduler/plugin/datasource/dolphindb/DolphinDBDataSourceChannelTest.java	Added tests for datasource channel client creation.
dolphinscheduler-datasource-dolphindb/src/test/java/org/apache/dolphinscheduler/plugin/datasource/dolphindb/DolphinDBDataSourceChannelFactoryTest.java	Added tests for datasource channel factory.
dolphinscheduler-datasource-dolphindb/src/main/java/org/apache/dolphinscheduler/plugin/datasource/dolphindb/param/DolphinDBDataSourceProcessor.java	Implements the datasource processor for DolphinDB.
dolphinscheduler-datasource-dolphindb/src/main/java/org/apache/dolphinscheduler/plugin/datasource/dolphindb/param/DolphinDBDataSourceParamDTO.java	Defines the parameter DTO for DolphinDB.
dolphinscheduler-datasource-dolphindb/src/main/java/org/apache/dolphinscheduler/plugin/datasource/dolphindb/param/DolphinDBConnectionParam.java	Implements connection parameters for DolphinDB.
dolphinscheduler-datasource-dolphindb/src/main/java/org/apache/dolphinscheduler/plugin/datasource/dolphindb/DolphinDBPooledDataSourceClient.java	Defines the pooled datasource client for DolphinDB.
dolphinscheduler-datasource-dolphindb/src/main/java/org/apache/dolphinscheduler/plugin/datasource/dolphindb/DolphinDBDataSourceChannelFactory.java	Implements the datasource channel factory for DolphinDB.
dolphinscheduler-datasource-dolphindb/src/main/java/org/apache/dolphinscheduler/plugin/datasource/dolphindb/DolphinDBDataSourceChannel.java	Defines the datasource channel for DolphinDB.
dolphinscheduler-datasource-dolphindb/src/main/java/org/apache/dolphinscheduler/plugin/datasource/dolphindb/DolphinDBAdHocDataSourceClient.java	Defines the ad hoc datasource client for DolphinDB.
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/constants/DataSourceConstants.java	Adds constants for the DolphinDB driver, JDBC URL prefix, and validation query.
docs/docs/zh/guide/datasource/dolphindb.md & docs/docs/en/guide/datasource/dolphindb.md	Added documentation for the new DolphinDB datasource.
docs/configs/docsdev.js	Updated navigation configuration to include DolphinDB documentation.

Files not reviewed (5)

config/plugins_config: Language not supported
dolphinscheduler-bom/pom.xml: Language not supported
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-all/pom.xml: Language not supported
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-dolphindb/pom.xml: Language not supported
dolphinscheduler-datasource-plugin/pom.xml: Language not supported

Copilot · 2025-04-21T07:09:40Z

.../apache/dolphinscheduler/plugin/datasource/dolphindb/param/DolphinDBDataSourceProcessor.java

+        if (!baseDataSourceParamDTO.getType().equals(DbType.REDSHIFT)) {
+            // due to redshift use not regular hosts


The condition incorrectly checks for DbType.REDSHIFT instead of DbType.DOLPHINDB, which may lead to incorrect host validation. Adjust the condition to compare against DbType.DOLPHINDB.

Suggested change

if (!baseDataSourceParamDTO.getType().equals(DbType.REDSHIFT)) {

// due to redshift use not regular hosts

if (baseDataSourceParamDTO.getType().equals(DbType.DOLPHINDB)) {

// validate host for DolphinDB

# Conflicts: # tools/dependencies/known-dependencies.txt

SbloodyS · 2025-08-06T01:53:23Z

Closing for inactivity.

eco8848 requested review from EricGao888, SbloodyS, caishunfeng, ruanwenjun and songjianet as code owners January 15, 2025 06:16

github-actions bot added UI ui and front end related backend test document labels Jan 15, 2025

github-actions bot assigned eco8848 Jan 15, 2025

SbloodyS changed the title ~~[Feature-15953][datasource-plugin] Add DolphinDB datasource~~ [DSIP-83][datasource-plugin] Add DolphinDB datasource Jan 15, 2025

SbloodyS added this to the 3.3.0 milestone Jan 15, 2025

SbloodyS added the DSIP label Jan 15, 2025

github-advanced-security bot found potential problems Jan 15, 2025

View reviewed changes

SbloodyS reviewed Jan 22, 2025

View reviewed changes

dolphinscheduler-bom/pom.xml Show resolved Hide resolved

eco8848 force-pushed the Feature-15953 branch from 34168bb to 28e1bab Compare January 22, 2025 08:59

davidzollo added the first time contributor First-time contributor label Jan 22, 2025

[DS-15953][feat] add dolphindb datasource

94799bf

eco8848 force-pushed the Feature-15953 branch from faa1bf6 to 94799bf Compare February 7, 2025 06:28

eco8848 and others added 4 commits February 11, 2025 14:43

[DS-15953][fix] delete unused license and update dolphindb datasource…

29db49c

… build config

Merge branch 'apache:dev' into Feature-15953

782df8d

Merge branch 'dev' into Feature-15953

2c2fadb

Merge branch 'dev' into Feature-15953

13686e2

eco8848 requested a review from SbloodyS February 13, 2025 02:03

SbloodyS removed this from the 3.3.0 milestone Feb 23, 2025

Merge branch 'dev' into Feature-15953

88cf1c3

Merge branch 'dev' into Feature-15953

453dc7b

davidzollo added 3.3.0 and removed 3.3.0 labels Mar 17, 2025

davidzollo added this to the 3.3.0 milestone Mar 17, 2025

SbloodyS modified the milestones: 3.3.0-alpha, 3.3.1 Mar 30, 2025

nielifeng requested a review from Copilot April 21, 2025 07:09

Copilot AI reviewed Apr 21, 2025

View reviewed changes

Merge branch 'dev' into Feature-15953

4a3f7e7

# Conflicts: # tools/dependencies/known-dependencies.txt

SbloodyS closed this Aug 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DSIP-83][datasource-plugin] Add DolphinDB datasource#16962

[DSIP-83][datasource-plugin] Add DolphinDB datasource#16962
eco8848 wants to merge 8 commits intoapache:devfrom
eco8848:Feature-15953

eco8848 commented Jan 15, 2025 •

edited by SbloodyS

Loading

Uh oh!

boring-cyborg bot commented Jan 15, 2025

Uh oh!

eco8848 commented Jan 15, 2025

Uh oh!

Check notice

SbloodyS left a comment

Uh oh!

Uh oh!

SbloodyS commented Feb 4, 2025

Uh oh!

eco8848 commented Feb 11, 2025

Uh oh!

eco8848 commented Feb 13, 2025

Uh oh!

eco8848 commented Feb 28, 2025

Uh oh!

sonarqubecloud bot commented Mar 17, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 21, 2025

Uh oh!

SbloodyS commented Aug 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

		if (!baseDataSourceParamDTO.getType().equals(DbType.REDSHIFT)) {
		// due to redshift use not regular hosts

Conversation

eco8848 commented Jan 15, 2025 • edited by SbloodyS Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of the pull request

Brief change log

Verify this pull request

Uh oh!

boring-cyborg bot commented Jan 15, 2025

Uh oh!

eco8848 commented Jan 15, 2025

Uh oh!

Check notice

SbloodyS left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SbloodyS commented Feb 4, 2025

Uh oh!

eco8848 commented Feb 11, 2025

Uh oh!

eco8848 commented Feb 13, 2025

Uh oh!

eco8848 commented Feb 28, 2025

Uh oh!

sonarqubecloud bot commented Mar 17, 2025

Quality Gate passed

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Apr 21, 2025

Choose a reason for hiding this comment

Uh oh!

SbloodyS commented Aug 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

eco8848 commented Jan 15, 2025 •

edited by SbloodyS

Loading