YARN-11843. Fix potential deadlock when auto-correction of container allocation is enabled #7855

TaoYang526 · 2025-08-05T02:25:16Z

Details please refer to YARN-11843.

Description of PR

How was this patch tested?

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

…allocation is enabled

TaoYang526 · 2025-08-05T02:28:23Z

@shameersss1 @slfan1989 Could you please help to review this PR? Thanks!

shameersss1

LGTM +1

...main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java

shameersss1 · 2025-08-05T02:56:00Z

@TaoYang526 - It is good catch indeed.
Just curious to know - May i know the use case behind using auto-correction feature ?

TaoYang526 · 2025-08-05T03:23:50Z

@TaoYang526 - It is good catch indeed. Just curious to know - May i know the use case behind using auto-correction feature ?

@shameersss1 Currently, we don't have a specific use case for the auto-correction feature. We have always been aware that YARN has an over-allocation issue, but considered it a minor concern, so no action was taken. When I discovered YARN-11702, I wanted to take a look and found that it could resolve most over-allocation issues, except for some in-scheduling cases. It's sufficient to reduce unnecessary consumption in the scheduling cycle and we may enable this feature in the future.

shameersss1 · 2025-08-05T05:15:41Z

@TaoYang526 - It is good catch indeed. Just curious to know - May i know the use case behind using auto-correction feature ?

@shameersss1 Currently, we don't have a specific use case for the auto-correction feature. We have always been aware that YARN has an over-allocation issue, but considered it a minor concern, so no action was taken. When I discovered YARN-11702, I wanted to take a look and found that it could resolve most over-allocation issues, except for some in-scheduling cases. It's sufficient to reduce unnecessary consumption in the scheduling cycle and we may enable this feature in the future.

Thanks for the details.

hadoop-yetus · 2025-08-05T12:23:44Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	52m 19s		trunk passed
+1 💚	compile	1m 8s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 54s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	0m 57s		trunk passed
+1 💚	mvnsite	0m 59s		trunk passed
+1 💚	javadoc	0m 59s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 49s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	1m 59s		trunk passed
+1 💚	shadedclient	41m 49s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 48s		the patch passed
+1 💚	compile	0m 57s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 57s		the patch passed
+1 💚	compile	0m 47s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 47s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 44s		the patch passed
+1 💚	mvnsite	0m 50s		the patch passed
+1 💚	javadoc	0m 46s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 41s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	1m 54s		the patch passed
+1 💚	shadedclient	42m 38s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	117m 4s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		270m 30s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7855/3/artifact/out/Dockerfile
GITHUB PR	#7855
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 8e43c4d936c4 5.15.0-144-generic #157-Ubuntu SMP Mon Jun 16 07:33:10 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `29cd1d7`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7855/3/testReport/
Max. process+thread count	939 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7855/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

zeekling · 2025-08-10T15:51:16Z

...main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java

-            completedContainer(rmContainer, SchedulerUtils.createAbnormalContainerStatus(
-                rmContainer.getContainerId(), SchedulerUtils.EXPIRED_CONTAINER),
-                RMContainerEventType.EXPIRE);
+            asyncContainerRelease(rmContainer);


Will it cause the message at the end of the container to be different?

Yes, the message will be updated from "Container expired since it was unused" to "Container released by application", which I believe is more appropriate.

Yes, the message will be updated from "Container expired since it was unused" to "Container released by application", which I believe is more appropriate.

No, I think the old is more clear.

For the old one, "expired since it was unused" is not accurate because it was released due to over-allocation, not because the waiting period expired. For the new one, "Container released by application" has been used in different scenarios that actively released by application, I think over-allocation could be included in this scope. @shameersss1 could you please share your thoughts about this?

@TaoYang526 +1
Container released makes more sense here.

TaoYang526 · 2025-08-14T06:04:55Z

@slfan1989 @aajisaka Could you please help review this PR when you have time?

…er#recoverResourceRequestForContainer and add UT.

TaoYang526 · 2025-08-18T10:20:43Z

Updated the expected state in AbstractYarnScheduler#recoverResourceRequestForContainer and added UT for that.

hadoop-yetus · 2025-08-18T15:17:29Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	23m 19s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	8m 20s		Maven dependency ordering for branch
-1 ❌	mvninstall	10m 8s	/branch-mvninstall-root.txt	root in trunk failed.
+1 💚	compile	6m 5s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	5m 7s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	2m 1s		trunk passed
+1 💚	mvnsite	2m 11s		trunk passed
+1 💚	javadoc	2m 14s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	2m 2s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	3m 57s		trunk passed
+1 💚	shadedclient	43m 27s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 32s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 23s		the patch passed
+1 💚	compile	5m 19s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	5m 19s		the patch passed
+1 💚	compile	4m 59s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	4m 59s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 51s		the patch passed
+1 💚	mvnsite	1m 57s		the patch passed
+1 💚	javadoc	1m 59s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 49s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	4m 7s		the patch passed
+1 💚	shadedclient	42m 57s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	5m 48s		hadoop-yarn-common in the patch passed.
-1 ❌	unit	116m 15s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-server-resourcemanager in the patch failed.
+1 💚	asflicense	1m 1s		The patch does not generate ASF License warnings.
		300m 56s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7855/4/artifact/out/Dockerfile
GITHUB PR	#7855
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 70f8d11d8504 5.15.0-144-generic #157-Ubuntu SMP Mon Jun 16 07:33:10 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `f05899c`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7855/4/testReport/
Max. process+thread count	895 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7855/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

YARN-11843. Fix potential deadlock when auto-correction of container …

b8c0647

…allocation is enabled

github-actions bot added YARN trunk labels Aug 5, 2025

shameersss1 approved these changes Aug 5, 2025

View reviewed changes

shameersss1 reviewed Aug 5, 2025

View reviewed changes

...main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java Outdated Show resolved Hide resolved

Update comment.

cc6bc5f

This comment was marked as outdated.

Sign in to view

Fix checkstyle warning.

29cd1d7

TaoYang526 self-assigned this Aug 8, 2025

zeekling reviewed Aug 10, 2025

View reviewed changes

Adjust expected state from EXIPRED to RELEASED in AbstractYarnSchedul…

f05899c

…er#recoverResourceRequestForContainer and add UT.

TaoYang526 requested a review from shameersss1 August 18, 2025 10:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

YARN-11843. Fix potential deadlock when auto-correction of container allocation is enabled #7855

YARN-11843. Fix potential deadlock when auto-correction of container allocation is enabled #7855

TaoYang526 commented Aug 5, 2025

Uh oh!

TaoYang526 commented Aug 5, 2025

Uh oh!

shameersss1 left a comment

Uh oh!

Uh oh!

shameersss1 commented Aug 5, 2025

Uh oh!

TaoYang526 commented Aug 5, 2025 •

edited

Loading

Uh oh!

shameersss1 commented Aug 5, 2025

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

hadoop-yetus commented Aug 5, 2025

Uh oh!

zeekling Aug 10, 2025

Uh oh!

TaoYang526 Aug 11, 2025

Uh oh!

zeekling Aug 17, 2025

Uh oh!

TaoYang526 Aug 17, 2025

Uh oh!

shameersss1 Aug 25, 2025

Uh oh!

TaoYang526 commented Aug 14, 2025

Uh oh!

TaoYang526 commented Aug 18, 2025

Uh oh!

hadoop-yetus commented Aug 18, 2025

Uh oh!

Uh oh!

YARN-11843. Fix potential deadlock when auto-correction of container allocation is enabled #7855

Are you sure you want to change the base?

YARN-11843. Fix potential deadlock when auto-correction of container allocation is enabled #7855

Conversation

TaoYang526 commented Aug 5, 2025

Description of PR

How was this patch tested?

For code changes:

Uh oh!

TaoYang526 commented Aug 5, 2025

Uh oh!

shameersss1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shameersss1 commented Aug 5, 2025

Uh oh!

TaoYang526 commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shameersss1 commented Aug 5, 2025

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

hadoop-yetus commented Aug 5, 2025

Uh oh!

zeekling Aug 10, 2025

Choose a reason for hiding this comment

Uh oh!

TaoYang526 Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

zeekling Aug 17, 2025

Choose a reason for hiding this comment

Uh oh!

TaoYang526 Aug 17, 2025

Choose a reason for hiding this comment

Uh oh!

shameersss1 Aug 25, 2025

Choose a reason for hiding this comment

Uh oh!

TaoYang526 commented Aug 14, 2025

Uh oh!

TaoYang526 commented Aug 18, 2025

Uh oh!

hadoop-yetus commented Aug 18, 2025

Uh oh!

Uh oh!

TaoYang526 commented Aug 5, 2025 •

edited

Loading