fix: prevent unbounded memory leak during concurrent validation under…#239
Open
KyleOps wants to merge 1 commit intohapifhir:masterfrom
Open
fix: prevent unbounded memory leak during concurrent validation under…#239KyleOps wants to merge 1 commit intohapifhir:masterfrom
KyleOps wants to merge 1 commit intohapifhir:masterfrom
Conversation
… load Introduced a ReentrantLock and rate-limiting to `getValidationService()` in ValidationServiceFactoryImpl.kt. When the JVM hits the low memory threshold during high concurrency, the application previously allowed all concurrent requests to spin up parallel initialization threads, spiraling memory usage and crashing the application. This fix ensures only a single thread can trigger the engine reload within a 5-minute cooldown period, allowing concurrent requests to queue or utilize the existing engine rather than flooding the JVM with unbounded heavy cache initialization threads.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR fixes a memory and thread exhaustion vulnerability in
ValidationServiceFactoryImpl.kt.Under high concurrency, if the JVM's free memory dips below the
engineReloadThreshold, thegetValidationService()method re-initializesValidationService. Because this check was not synchronized, a burst of incoming validation requests during a low-memory state would all evaluate the condition astruesimultaneously.This resulted in an unbounded loop where the application spawned dozens of massive
ValidationEngineinstances and hundreds of detached background downloader threads to reload presets. The JVM quickly spiraled into maxed CPU usage, gigabytes of leaked memory, and eventual crashes.Impact & Context
We discovered this in while utilizing the validator API in conjunction with au-fhir-inferno. Under heavy testing load, the validator-wrapper would hit the memory threshold and trigger this un-synchronized cache reset. The true impact was three-fold:
.tgzfiles frompackages.fhir.organd leak file descriptors on the host container.Changes
@VolatiletovalidationService.ReentrantLock(reloadLock) to ensure only one thread can lock and initialize the cache.RELOAD_COOLDOWN_MS) so the engine isn't perpetually thrashed under sustained heavy load.