[Improvement-17724][TaskPlugin] EmrTask resource leak repair#17719
[Improvement-17724][TaskPlugin] EmrTask resource leak repair#17719niumy0701 wants to merge 20 commits intoapache:devfrom
Conversation
There was a problem hiding this comment.
You should create an issue first and link to it. DSIP-23 is a main issue, and you need to create a sub issue. @niumy0701
done |
There was a problem hiding this comment.
Pull request overview
This PR fixes resource leaks in the EMR task plugin by ensuring that the AWS EMR client is properly shut down after task execution completes or is cancelled. Previously, EMR client connections were not being released, which could lead to resource exhaustion over time in long-running DolphinScheduler instances.
- Added
emrClient.shutdown()calls intrackApplicationStatus()andcancelApplication()methods for both EmrJobFlowTask and EmrAddStepsTask - Refactored
EmrAddStepsTask.cancelApplication()with try-finally block to guarantee cleanup even when exceptions occur - Added test coverage to verify shutdown behavior in both success and failure scenarios
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/main/java/org/apache/dolphinscheduler/plugin/task/emr/EmrJobFlowTask.java | Added emrClient.shutdown() calls in trackApplicationStatus() and cancelApplication() to fix resource leak |
| dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/main/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTask.java | Added emrClient.shutdown() in trackApplicationStatus() and wrapped cancelApplication() in try-finally to ensure cleanup |
| dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/test/java/org/apache/dolphinscheduler/plugin/task/emr/EmrJobFlowTaskTest.java | Added test to verify shutdown is called when cancelling job flow |
| dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/test/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTaskTest.java | Added tests to verify shutdown is called in error scenarios when cancelling steps |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.
...-task-emr/src/test/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTaskTest.java
Outdated
Show resolved
Hide resolved
...-task-emr/src/test/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTaskTest.java
Outdated
Show resolved
Hide resolved
...-task-emr/src/test/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTaskTest.java
Outdated
Show resolved
Hide resolved
...-task-emr/src/test/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTaskTest.java
Outdated
Show resolved
Hide resolved
thanks , It has been modified and can be reviewed again |
...uler-task-emr/src/main/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTask.java
Outdated
Show resolved
Hide resolved
...duler-task-emr/src/main/java/org/apache/dolphinscheduler/plugin/task/emr/EmrJobFlowTask.java
Outdated
Show resolved
Hide resolved
|
|
Please provide relevant verification before and after the repair. |


Please refer to the details of main issue #14877
fix #17724
Purpose of the pull request
Brief change log
Verify this pull request
This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(or)
Pull Request Notice
Pull Request Notice
If your pull request contains incompatible change, you should also add it to
docs/docs/en/guide/upgrade/incompatible.md