[Native Image] JFR native memory usage increases steadily

[joggeli34]

Describe the Issue

We use the build flag --enable-monitoring=jfr,nmt as we like to use pyroscope for profiling. This works in general well, but we see a constant increase of memory that is not on the heap.

On the left, the container memory. On the right the heap:

With the native memory tracking, we see that the JFR native memory is big and it seems to be growing the longer we let the application run:

Total
    (reserved=33774140KB, committed=762642KB)
    (malloc=219708KB, count=24817)
    (peak malloc=288450KB, count at peak=57367)
    (mmap: reserved=33554432KB, committed=542933KB)
    (mmap: peak reserved=33554432KB, peak committed=753190KB)
Java Heap
    (reserved=33461440KB, committed=450453KB)
    (malloc=0KB, count=0)
    (peak malloc=0KB, count at peak=0)
    (mmap: reserved=33461440KB, committed=450453KB)
    (mmap: peak reserved=33461440KB, peak committed=660710KB)
JFR
    (reserved=198613KB, committed=198613KB)
    (malloc=198613KB, count=24706)
    (peak malloc=232462KB, count at peak=62502)
    (mmap: reserved=0KB, committed=0KB)
    (mmap: peak reserved=0KB, peak committed=0KB)

We use the pyroscope-java, which creates a lot of JFR recordings with this code:
https://github.com/grafana/pyroscope-java/blob/main/agent/src/main/java/io/pyroscope/javaagent/JFRJDKProfilerDelegate.java

Otherwise we use default setting for the JFR, so we add no additional settings as start parameter.
By my understanding, the JFR default settings should use a lot less memory and should not lead to constant growing memory. Or is there a setting I missed to limit that?

Using the latest version of GraalVM can resolve many issues.

• I tried with the latest version of GraalVM.

GraalVM Version

java 25 2025-09-16 LTS
Java(TM) SE Runtime Environment Oracle GraalVM 25+37.1 (build 25+37-LTS-jvmci-b01)
Java HotSpot(TM) 64-Bit Server VM Oracle GraalVM 25+37.1 (build 25+37-LTS-jvmci-b01, mixed mode, sharing)

Operating System and Version

docker image: quay.io/quarkus/ubi9-quarkus-micro-image:2.0

Troubleshooting Confirmation

• I tried the suggestions in the troubleshooting guide.

Run Command

./application -XX:+ExitOnOutOfMemoryError -Djava.net.preferIPv4Stack=true -Xmx1500m -Xmn256m -XX:MaxDirectMemorySize=128m -XX:+PrintNMTStatistics -XX:MaxHeapFree=128m -XX:MaxRAM=1800m

Expected Behavior

We expect the JFR native memory to stay limited on a small size (some megabytes)

Actual Behavior

The JFR native memory increases the longer the application is run

Steps to Reproduce

In our application, it is always the case when enabling pyroscope with jfr profiling

Additional Context

No response

Run-Time Log Output and Error Messages

Native memory log:

Total
    (reserved=33774140KB, committed=762642KB)
    (malloc=219708KB, count=24817)
    (peak malloc=288450KB, count at peak=57367)
    (mmap: reserved=33554432KB, committed=542933KB)
    (mmap: peak reserved=33554432KB, peak committed=753190KB)
Compiler
    (reserved=0KB, committed=0KB)
    (malloc=0KB, count=0)
    (peak malloc=0KB, count at peak=0)
    (mmap: reserved=0KB, committed=0KB)
    (mmap: peak reserved=0KB, peak committed=0KB)
Code
    (reserved=0KB, committed=0KB)
    (malloc=0KB, count=0)
    (peak malloc=0KB, count at peak=0)
    (mmap: reserved=0KB, committed=0KB)
    (mmap: peak reserved=0KB, peak committed=0KB)
GC
    (reserved=3KB, committed=3KB)
    (malloc=3KB, count=66)
    (peak malloc=6KB, count at peak=113)
    (mmap: reserved=0KB, committed=0KB)
    (mmap: peak reserved=0KB, peak committed=0KB)
Heap Dump
    (reserved=0KB, committed=0KB)
    (malloc=0KB, count=0)
    (peak malloc=0KB, count at peak=0)
    (mmap: reserved=0KB, committed=0KB)
    (mmap: peak reserved=0KB, peak committed=0KB)
Image Heap
    (reserved=92992KB, committed=92480KB)
    (malloc=0KB, count=0)
    (peak malloc=0KB, count at peak=0)
    (mmap: reserved=92992KB, committed=92480KB)
    (mmap: peak reserved=92992KB, peak committed=92480KB)
Java Heap
    (reserved=33461440KB, committed=450453KB)
    (malloc=0KB, count=0)
    (peak malloc=0KB, count at peak=0)
    (mmap: reserved=33461440KB, committed=450453KB)
    (mmap: peak reserved=33461440KB, peak committed=660710KB)
JFR
    (reserved=198613KB, committed=198613KB)
    (malloc=198613KB, count=24706)
    (peak malloc=232462KB, count at peak=62502)
    (mmap: reserved=0KB, committed=0KB)
    (mmap: peak reserved=0KB, peak committed=0KB)
JNI
    (reserved=0KB, committed=0KB)
    (malloc=0KB, count=1)
    (peak malloc=0KB, count at peak=1)
    (mmap: reserved=0KB, committed=0KB)
    (mmap: peak reserved=0KB, peak committed=0KB)
jvmstat
    (reserved=0KB, committed=0KB)
    (malloc=0KB, count=0)
    (peak malloc=0KB, count at peak=0)
    (mmap: reserved=0KB, committed=0KB)
    (mmap: peak reserved=0KB, peak committed=0KB)
JVMTI
    (reserved=0KB, committed=0KB)
    (malloc=0KB, count=0)
    (peak malloc=0KB, count at peak=0)
    (mmap: reserved=0KB, committed=0KB)
    (mmap: peak reserved=0KB, peak committed=0KB)
Native Memory Tracking
    (reserved=387KB, committed=387KB)
    (malloc=387KB, count=24817)
    (peak malloc=1257KB, count at peak=80474)
    (mmap: reserved=0KB, committed=0KB)
    (mmap: peak reserved=0KB, peak committed=0KB)
PGO
    (reserved=0KB, committed=0KB)
    (malloc=0KB, count=0)
    (peak malloc=0KB, count at peak=0)
    (mmap: reserved=0KB, committed=0KB)
    (mmap: peak reserved=0KB, peak committed=0KB)
Serviceability)
    (reserved=0KB, committed=0KB)
    (malloc=0KB, count=0)
    (peak malloc=0KB, count at peak=0)
    (mmap: reserved=0KB, committed=0KB)
    (mmap: peak reserved=0KB, peak committed=0KB)
Threading
    (reserved=2KB, committed=2KB)
    (malloc=2KB, count=20)
    (peak malloc=15KB, count at peak=117)
    (mmap: reserved=0KB, committed=0KB)
    (mmap: peak reserved=0KB, peak committed=0KB)
Unsafe
    (reserved=20701KB, committed=20701KB)
    (malloc=20701KB, count=24)
    (peak malloc=82734KB, count at peak=57)
    (mmap: reserved=0KB, committed=0KB)
    (mmap: peak reserved=0KB, peak committed=0KB)
Internal
    (reserved=0KB, committed=0KB)
    (malloc=0KB, count=0)
    (peak malloc=1KB, count at peak=1)
    (mmap: reserved=0KB, committed=0KB)
    (mmap: peak reserved=0KB, peak committed=0KB)

[roberttoyonaga]

@joggeli34 Thank you for reporting this. I am not very familiar with Pyroscope and how it's used. Are the graphs you show created from many short lived recordings (with periods of no recording in between), or is there one long lived recording?

A side note probably not directly related to the memory problem you describe: I notice that Pyroscope is enabling the jdk.ObjectAllocationInNewTLAB and jdk.ObjectAllocationOutsideTLAB events . These events have very high overhead and are not meant for continuous monitoring purposes. Maybe the overhead is acceptable if the recording is very short lived.

[roberttoyonaga]

It would also be interesting to see the native memory used by JFR specifically over time. If you have a JFR snapshot, you can view that by looking at the jdk.NativeMemoryUsage event. This will sample every 1s by default.

[joggeli34]

Thanks for the fast answer.
The Pyroscope does short lived recordings, by default 10sec. When one is finished it starts the next. https://github.com/grafana/pyroscope-java/blob/f41780db98ec2603d6058cdd5bfda8425fb7e5b4/agent/src/main/java/io/pyroscope/javaagent/impl/ContinuousProfilingScheduler.java#L128
I will try create a JFR snapshot next.

[joggeli34]

I wasn't able yet to create a meaningful JFR snapshot. I am running the application with:

-XX:StartFlightRecording=duration=10m,delay=15s,filename=recording.jfr,settings=/work/nmt.jfc

and the nmt.jfc contains:

<?xml version="1.0" encoding="UTF-8"?>
<configuration version="2.0"
  label="Native Memory Only"
  description="Enable only jdk.NativeMemoryUsage">
  <events>
    <event name="jdk.NativeMemoryUsage">
      <setting name="enabled">true</setting>
    </event>

    <event name="jdk.ExecutionSample"><setting name="enabled">false</setting></event>
    <event name="jdk.NativeMethodSample"><setting name="enabled">false</setting></event>
    <event name="jdk.MethodProfilingSample"><setting name="enabled">false</setting></event>
    <event name="jdk.ObjectAllocationSample"><setting name="enabled">false</setting></event>
    <event name="jdk.ObjectAllocationInNewTLAB"><setting name="enabled">false</setting></event>
    <event name="jdk.ObjectAllocationOutsideTLAB"><setting name="enabled">false</setting></event>
    <event name="jdk.JavaMonitorEnter"><setting name="enabled">false</setting></event>
    <event name="jdk.ThreadPark"><setting name="enabled">false</setting></event>
  </events>
</configuration>

But somehow the snapshot includes the ExecutionSamples and ObjectAllocations anyway making it huge, so that I can't run it for a longer and more meaningful time. Probably because it is run in parallel to the other recording from Pyroscope containing all this? Or I do something wrong as I'm not that used in using JFR

[roberttoyonaga]

The Pyroscope does short lived recordings, by default 10sec. When one is finished it starts the next.

This means the overhead will be consistently very high because there is no downtime between recordings. This is only a problem because pyroscope is enabling jdk.ObjectAllocationInNewTLAB and jdk.ObjectAllocationOutsideTLAB. These are profiling events not meant for continuous monitoring (jdk.ObjectAllocationSample is more suitable since it samples instead). Regardless, the native memory usage should not be increasing like that over time. There may be a leak somewhere.

But somehow the snapshot includes the ExecutionSamples and ObjectAllocations anyway making it huge, so that I can't run it for a longer and more meaningful time. Probably because it is run in parallel to the other recording from Pyroscope containing all this? Or I do something wrong as I'm not that used in using JFR

Maybe the big size is why Pyroscope is creating many short lived recordings (to limit the size) then discarding them after a period of time.
Yes that's probably why you see those events even though they are disabled in your .jfc file. If recordings are running in parallel, JFR will use the union of the enabled events. This is because the underlying data capture and processing are the same regardless of the number of recordings running (it doesn't double the work done by JFR). The "recordings" are just a way of grouping events and settings conveniently for the user.

With the native memory tracking, we see that the JFR native memory is big and it seems to be growing the longer we let the application run:

Does NMT show a consistent increase in committed memory used by JFR over time? How did you find this if you don't have access to a JFR snapshot? Did you dump multiple NMT reports over a period of time using jcmd?

Do you see the RSS increase over time reproducible in other applications as well? Do you know how sensitive it is to load? If possible, I can try to reproduce it myself and try to track it down with NMT.

[joggeli34]

For the dk.ObjectAllocationInNewTLAB and jdk.ObjectAllocationOutsideTLAB I will raise a issue at the pyroscope project to make it optional. As for us we are at the moment mainly interested in the cpu profiling.

Does NMT show a consistent increase in committed memory used by JFR over time? How did you find this if you don't have access to a JFR snapshot? Did you dump multiple NMT reports over a period of time using jcmd?

We were trying investigating why the memory of the application raises almost constantly over time when we build it native. There are other of our application that don't suffer from this problems. But this don't have pyroscope enabled as there the profiling is not that relevant for us as they are not mostly cpu bound. The memory usage of the containers looked as follows, where you see that there are application with constant memory usage at the bottom and some growing over time (20day chart):

We only activated -XX:+PrintNMTStatistics as our containers don't have jcmd installed. There we noticed, that the longer the application had run, the bigger the values of JFR were while all the other NMT Statistics stayed quite constant. The biggest value was more then 700MB:

2025-09-22T17:35:25.415Z	JFR
2025-09-22T17:35:25.415Z	    (reserved=713064KB, committed=713064KB)
2025-09-22T17:35:25.415Z	    (malloc=713064KB, count=88700)
2025-09-22T17:35:25.415Z	    (peak malloc=728570KB, count at peak=98207)
2025-09-22T17:35:25.415Z	    (mmap: reserved=0KB, committed=0KB)
2025-09-22T17:35:25.415Z	    (mmap: peak reserved=0KB, peak committed=0KB)

I don't know if it is sensitive to load or to what else. It is the only application we have jfr and pyroscope enabled. We have a rather constant load (more during the day then in the night) but constant over long time. Here the charts of one of the instance of the applications:

I will create a version today that will collect the `jdk.NativeMemoryUsage" as prometheus metrics with a RecordingStream:

        rs = new RecordingStream();
        rs.enable("jdk.NativeMemoryUsage").withPeriod(Duration.ofSeconds(1));
        rs.onEvent("jdk.NativeMemoryUsage", this::updateMetrics);

I will give you the results tomorrow after it is run through the night.

[roberttoyonaga]

Thank you for your very detailed response. I think your assessment is correct. The culprit does seem like a leak in JFR. Pyroscope probably exacerbates the problem because it forces high overhead events to be enabled for a long duration.

I am going to try reproducing the problem without Pyroscope. I'll create a custom build of GraalVM with finer-grained NMT categories to try to pinpoint where a leak could be. I plan to use a simple Quarkus app, drive with a constant load, and enable the same high overhead events Pyroscope is using.

[joggeli34]

I now let it run with the added RecordingStream for 24h collecting the JFR native memory usage:

It seams that the addition of this RecordingStream that runs the whole time makes that the memory is not increasing anymore. Also the container memory as a whole is not increasing anymore:

I assume that the memory leak only appears when all the JFR recordings are stopped and then restarted. So that on one point there is no active recording. I assume then some parts of JFR are completely stopped and restarted?

[roberttoyonaga]

Just an update:
I made a few tests using finer-grained NMT that I ran over-night and was able to find the leak. The problem is in the JFR data buffer allocation. The leak only manifests when many recordings are started/stopped in quick succession. The leak does not seem to manifest when I make a single long lived recording.

I will work on a fix and link it here once I get a PR up. Thanks again for reporting this @joggeli34