HADOOP-19624 Thread leak in ABFS AbfsClientThrottlingAnalyzer #7852

mattkduran · 2025-08-03T18:02:17Z

Description of PR

The ABFS driver's auto-throttling feature (fs.azure.enable.autothrottling=true) creates Timer threads in AbfsClientThrottlingAnalyzer that are never properly cleaned up, leading to a memory leak that eventually causes OutOfMemoryError in long-running applications like Hive Metastore.

Impact:

Thread count grows indefinitely (observed >100,000 timer threads)
Affects any long-running service that creates multiple ABFS filesystem instances

Root Cause:

AbfsClientThrottlingAnalyzer creates Timer objects in its constructor but provides no mechanism to cancel them. When AbfsClient instances are closed, the associated timer threads continue running indefinitely.

Solution

Implement proper resource cleanup by making the throttling components implement Closeable and ensuring timers are cancelled when ABFS clients are closed.

Changes Made

AbfsClientThrottlingAnalyzer.java

Added: implements Closeable
Added: close() method that calls timer.cancel() and timer.purge()
Purpose: Ensures timer threads are properly terminated when analyzer is no longer needed

AbfsThrottlingIntercept.java (Interface)

Added: extends Closeable
Added: close() method signature
Purpose: Establishes cleanup contract for all throttling intercept implementations

AbfsClientThrottlingIntercept.java

Added: close() method that closes both readThrottler and writeThrottler
Purpose: Coordinates cleanup of both read and write throttling analyzers

AbfsNoOpThrottlingIntercept.java

Added: No-op close() method
Purpose: Satisfies interface contract for no-op implementation

AbfsClient.java

Added: IOUtils.cleanupWithLogger(LOG, intercept) in existing close() method
Purpose: Integrates throttling cleanup into existing client resource management

https://github.com/mattkduran/ABFSleaktest
https://www.mail-archive.com/[email protected]/msg43483.html

How was this patch tested?

Standalone Validation Tool

This fix was validated using a standalone reproduction and testing tool that directly exercises the ABFS auto-throttling components outside of a full Hadoop deployment.
Repository: ABFSLeakTest

Testing Scope

Problem reproduction confirmed - demonstrates the timer thread leak
Fix validation confirmed - proves close() method resolves the leak
Resource cleanup verified - shows proper timer cancellation
Limited integration testing - standalone tool, not full Hadoop test suite

Test Results

Leak Reproduction Evidence

# Without fix: Timer threads accumulate over filesystem creation cycles
Cycle    Total Threads    ABFS Timer Threads    Status
1        50->52          0->2                   LEAK DETECTED
50       150->152        98->100               LEAK GROWING  
200      250->252        398->400              LEAK CONFIRMED

Final Analysis: 400 leaked timer threads named "abfs-timer-client-throttling-analyzer-*"
Memory Impact: ~90MB additional heap usage

# Direct analyzer testing:
🔴 Without close(): +3 timer threads (LEAKED)
✅ With close():    +0 timer threads (NO LEAK)

Test Environment

Java Version: OpenJDK 11.0.x
Hadoop Version: 3.3.6/3.4.1 (both affected)
Test Duration: 200 filesystem creation/destruction cycles
Thread Monitoring: JMX ThreadMXBean

For code changes:

[ X ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

This method closes the throttling analyzer and releases any associated resources.

hadoop-yetus · 2025-08-03T20:44:33Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	20m 54s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	46m 41s		trunk passed
+1 💚	compile	0m 42s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 36s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	0m 32s		trunk passed
+1 💚	mvnsite	0m 41s		trunk passed
+1 💚	javadoc	0m 42s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 34s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	1m 10s		trunk passed
+1 💚	shadedclient	41m 9s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
-1 ❌	mvninstall	0m 21s	/patch-mvninstall-hadoop-tools_hadoop-azure.txt	hadoop-azure in the patch failed.
-1 ❌	compile	0m 23s	/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt	hadoop-azure in the patch failed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.
-1 ❌	javac	0m 23s	/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt	hadoop-azure in the patch failed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.
-1 ❌	compile	0m 21s	/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09.txt	hadoop-azure in the patch failed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09.
-1 ❌	javac	0m 21s	/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09.txt	hadoop-azure in the patch failed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09.
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 3 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚	checkstyle	0m 21s		the patch passed
-1 ❌	mvnsite	0m 23s	/patch-mvnsite-hadoop-tools_hadoop-azure.txt	hadoop-azure in the patch failed.
-1 ❌	javadoc	0m 22s	/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt	hadoop-azure in the patch failed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.
-1 ❌	javadoc	0m 26s	/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09.txt	hadoop-azure in the patch failed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09.
-1 ❌	spotbugs	0m 22s	/patch-spotbugs-hadoop-tools_hadoop-azure.txt	hadoop-azure in the patch failed.
+1 💚	shadedclient	44m 10s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	0m 25s	/patch-unit-hadoop-tools_hadoop-azure.txt	hadoop-azure in the patch failed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		160m 49s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7852/1/artifact/out/Dockerfile
GITHUB PR	#7852
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 39b56ca5b682 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `fee9861`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7852/1/testReport/
Max. process+thread count	536 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7852/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

anujmodi2021

Thanks for the patch and thorough testing of the issue @mattkduran
I have a few suggestions and comments. Please do take a look at them.

Also we need at least one test (unit or integration) to be inlcuded in this patch. Can you plan for one? Idea is to have coverage of code impacted here.

Also, I see a few PR checks failing, If you click on the link of each -1 commented by hadoop-yetus, you should be able to see the issue reported and fix them.

Once all of this is done, we can wait for a few more reviews and get this checked in.

Thanks again for all the efforts.

anujmodi2021 · 2025-08-26T09:21:22Z

...doop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsThrottlingIntercept.java

@@ -26,7 +26,7 @@
 */
 @InterfaceAudience.Private
 @InterfaceStability.Unstable
-public interface AbfsThrottlingIntercept {
+public interface AbfsThrottlingIntercept extends Closable {


Shouldn't this be extends Closeable?

anujmodi2021 · 2025-08-26T09:23:18Z

...-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsNoOpThrottlingIntercept.java

@@ -40,4 +42,12 @@ public void updateMetrics(final AbfsRestOperationType operationType,
  public void sendingRequest(final AbfsRestOperationType operationType,
      final AbfsCounters abfsCounters) {
  }
+
+/**
+ * No-op implementation of close method.


Nit: javadoc to include @ throws

anujmodi2021 · 2025-08-26T09:23:41Z

...zure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClientThrottlingIntercept.java

+
+  /**
+   * Closes the throttling intercept and releases associated resources.
+   * This method closes both the read and write throttling analyzers.


Nit: Javadoc to include @ throws

anujmodi2021 · 2025-08-26T09:30:25Z

...azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClientThrottlingAnalyzer.java

+  /**
+ * Closes the throttling analyzer and releases associated resources.
+ * This method cancels the internal timer and cleans up any pending timer tasks.
+ * It is safe to call this method multiple times.


Fix javadoc here a well.

mattkduran added 8 commits July 16, 2025 13:02

Import closable

0492703

Class implements closable

792a86f

Add in close method

05ee721

This method closes the throttling analyzer and releases any associated resources.

Import in IOException

f894117

Add in implementation to call close method

5cf275c

Add in cleanup method

0a0f4e1

Adding in close method

25c6d89

Added no-op method

fee9861

github-actions bot added trunk TOOLS ABFS labels Aug 3, 2025

anujmodi2021 requested changes Aug 26, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-19624 Thread leak in ABFS AbfsClientThrottlingAnalyzer #7852

HADOOP-19624 Thread leak in ABFS AbfsClientThrottlingAnalyzer #7852

mattkduran commented Aug 3, 2025 •

edited

Loading

Uh oh!

hadoop-yetus commented Aug 3, 2025

Uh oh!

anujmodi2021 left a comment

Uh oh!

anujmodi2021 Aug 26, 2025

Uh oh!

anujmodi2021 Aug 26, 2025

Uh oh!

anujmodi2021 Aug 26, 2025

Uh oh!

anujmodi2021 Aug 26, 2025

Uh oh!

Uh oh!

HADOOP-19624 Thread leak in ABFS AbfsClientThrottlingAnalyzer #7852

Are you sure you want to change the base?

HADOOP-19624 Thread leak in ABFS AbfsClientThrottlingAnalyzer #7852

Conversation

mattkduran commented Aug 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Impact:

Root Cause:

Solution

Changes Made

How was this patch tested?

Standalone Validation Tool

Testing Scope

Test Results

Test Environment

For code changes:

Uh oh!

hadoop-yetus commented Aug 3, 2025

Uh oh!

anujmodi2021 left a comment

Choose a reason for hiding this comment

Uh oh!

anujmodi2021 Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

anujmodi2021 Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

anujmodi2021 Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

anujmodi2021 Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mattkduran commented Aug 3, 2025 •

edited

Loading