Skip to content

[Improvement-17724][TaskPlugin] EmrTask resource leak repair#17719

Closed
niumy0701 wants to merge 20 commits intoapache:devfrom
niumy0701:DSIP-23-EmrTask
Closed

[Improvement-17724][TaskPlugin] EmrTask resource leak repair#17719
niumy0701 wants to merge 20 commits intoapache:devfrom
niumy0701:DSIP-23-EmrTask

Conversation

@niumy0701
Copy link

@niumy0701 niumy0701 commented Nov 25, 2025

Please refer to the details of main issue #14877
fix #17724

Purpose of the pull request

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

Pull Request Notice

Pull Request Notice

If your pull request contains incompatible change, you should also add it to docs/docs/en/guide/upgrade/incompatible.md

Copy link
Member

@SbloodyS SbloodyS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should create an issue first and link to it. DSIP-23 is a main issue, and you need to create a sub issue. @niumy0701

@niumy0701
Copy link
Author

You should create an issue first and link to it. DSIP-23 is a main issue, and you need to create a sub issue. @niumy0701

done
sub issue is #17724

@niumy0701 niumy0701 changed the title [DSIP-23][TaskPlugin] EmrTask resource leak repair [Improvement-17724][TaskPlugin] EmrTask resource leak repair Nov 25, 2025
@niumy0701 niumy0701 requested a review from SbloodyS November 25, 2025 08:02
@SbloodyS SbloodyS requested a review from Copilot November 26, 2025 02:00
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes resource leaks in the EMR task plugin by ensuring that the AWS EMR client is properly shut down after task execution completes or is cancelled. Previously, EMR client connections were not being released, which could lead to resource exhaustion over time in long-running DolphinScheduler instances.

  • Added emrClient.shutdown() calls in trackApplicationStatus() and cancelApplication() methods for both EmrJobFlowTask and EmrAddStepsTask
  • Refactored EmrAddStepsTask.cancelApplication() with try-finally block to guarantee cleanup even when exceptions occur
  • Added test coverage to verify shutdown behavior in both success and failure scenarios

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File Description
dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/main/java/org/apache/dolphinscheduler/plugin/task/emr/EmrJobFlowTask.java Added emrClient.shutdown() calls in trackApplicationStatus() and cancelApplication() to fix resource leak
dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/main/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTask.java Added emrClient.shutdown() in trackApplicationStatus() and wrapped cancelApplication() in try-finally to ensure cleanup
dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/test/java/org/apache/dolphinscheduler/plugin/task/emr/EmrJobFlowTaskTest.java Added test to verify shutdown is called when cancelling job flow
dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/test/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTaskTest.java Added tests to verify shutdown is called in error scenarios when cancelling steps

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.

Copy link
Member

@ruanwenjun ruanwenjun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@niumy0701
Copy link
Author

niumy0701 commented Nov 29, 2025

same #17718 (review)

thanks , It has been modified and can be reviewed again

@niumy0701 niumy0701 requested a review from ruanwenjun November 29, 2025 11:30
@niumy0701 niumy0701 requested a review from ruanwenjun December 5, 2025 03:07
@github-actions github-actions bot removed the test label Dec 8, 2025
@sonarqubecloud
Copy link

sonarqubecloud bot commented Dec 8, 2025

Quality Gate Failed Quality Gate failed

Failed conditions
52.0% Coverage on New Code (required ≥ 60%)

See analysis details on SonarQube Cloud

@ruanwenjun
Copy link
Member

Please provide relevant verification before and after the repair.

@ruanwenjun ruanwenjun closed this Dec 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement][TaskPlugin] EmrTask resource leak issue

4 participants