Avoid off-heap memory growth from JDK direct buffer caching#186
Merged
Ian Streeter (istreeter) merged 1 commit intodevelopfrom Feb 18, 2026
Merged
Avoid off-heap memory growth from JDK direct buffer caching#186Ian Streeter (istreeter) merged 1 commit intodevelopfrom
Ian Streeter (istreeter) merged 1 commit intodevelopfrom
Conversation
Pull Request Test Coverage Report for Build 22133410831Details
💛 - Coveralls |
Ian Streeter (istreeter)
force-pushed
the
fix-off-heap-memory-growth
branch
2 times, most recently
from
2342cb7 to
f7761fe
Compare
Open database files via FileInputStream and pass to the InputStream-based DatabaseReader.Builder constructor, instead of passing a File which internally uses FileChannel.read into a heap ByteBuffer. The FileChannel path causes the JDK to allocate and cache temporary direct ByteBuffers in thread-local storage (sun.nio.ch.Util.BufferCache), leading to off-heap memory growth in scenarios where the database is initialized repeatedly.
Ian Streeter (istreeter)
force-pushed
the
fix-off-heap-memory-growth
branch
from
f7761fe to
bb835a2
Compare
private-to-public-sync bot
pushed a commit
to snowplow/enrich
that referenced
this pull request
Feb 25, 2026
Includes snowplow/scala-maxmind-iplookups#186 which reduces how much off-heap memory gets used when initializing the IP Lookup enrichment
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Jira ref: PDP-2490
Read database files via Files.readAllBytes and pass as an InputStream to DatabaseReader.Builder, instead of passing a File which internally uses FileChannel.read into a heap ByteBuffer. The FileChannel path causes the JDK to allocate and cache temporary direct ByteBuffers in thread-local storage (sun.nio.ch.Util.BufferCache), leading to off-heap memory growth in scenarios where the database is initialized repeatedly.