Memory regression in opentelemetry prometheus exporter v0.57.0 version with go 1.24

[ns-jvillarfernandez]

Description

We've detected a memory leak / regression in one of our containers and the culprit seems to be go.opentelemetry.io/otel/exporters/prometheus v0.57.0 , or a combo change of go.opentelemetry.io/otel/exporters/prometheus v0.57.0 and go version upgrade from 1.23 to 1.24.

1. Our containers started to OOM after several hours running
2. we verified that the traffic pattern hadn't change
3. attaching to a running pod and using go-tool we saw that this was tied to go.opentelemetry.io/otel/exporters/prometheus v0.57.0, precisely math/rand.newSource. Please, see the commands and screenshot below

$ kubectl port-forward pod/XXXXXXX 8086:8085
Forwarding from 127.0.0.1:8086 -> 8085
Forwarding from [::1]:8086 -> 8085
Handling connection for 8086

$ go tool pprof http://localhost:8086/debug/pprof/heap
Fetching profile over HTTP from http://localhost:8086/debug/pprof/heap
Saved profile in /Users/XXXXXXX/pprof/pprof.forward.alloc_objects.alloc_space.inuse_objects.inuse_space.002.pb.gz
File: XXXXXXX
Build ID: 944b0f39f5443eb2ef822291ecd1bb226a3c768b
Type: inuse_space
Time: 2025-05-16 10:49:15 CEST
Entering interactive mode (type "help" for commands, "o" for options)

(pprof) top
Showing nodes accounting for 286.11MB, 80.53% of 355.27MB total
Dropped 159 nodes (cum <= 1.78MB)
Showing top 10 nodes out of 151
      flat  flat%   sum%        cum   cum%
  123.63MB 34.80% 34.80%   123.63MB 34.80%  math/rand.newSource (inline)
   61.10MB 17.20% 52.00%    61.10MB 17.20%  go.opentelemetry.io/otel/sdk/metric/exemplar.newStorage (inline)
   31.04MB  8.74% 60.73%    31.04MB  8.74%  go.opentelemetry.io/otel/sdk/metric/internal/aggregate.reset[go.shape.struct { FilteredAttributes []go.opentelemetry.io/otel/attribute.KeyValue; Time time.Time; Value go.shape.int64; SpanID []uint8 "json:\",omitempty\""; TraceID []uint8 "json:\",omitempty\"" }]
   25.04MB  7.05% 67.78%    25.04MB  7.05%  go.opentelemetry.io/otel/sdk/metric/internal/aggregate.reset[go.shape.struct { FilteredAttributes []go.opentelemetry.io/otel/attribute.KeyValue; Time time.Time; Value go.shape.float64; SpanID []uint8 "json:\",omitempty\""; TraceID []uint8 "json:\",omitempty\"" }]
   14.56MB  4.10% 71.88%    14.56MB  4.10%  bufio.NewWriterSize
    7.03MB  1.98% 73.86%     7.03MB  1.98%  bufio.NewReaderSize
       7MB  1.97% 75.83%    17.50MB  4.93%  go.opentelemetry.io/otel/exporters/prometheus.addExemplars[go.shape.int64]
    6.19MB  1.74% 77.57%    12.71MB  3.58%  io.copyBuffer
    5.52MB  1.55% 79.12%     5.52MB  1.55%  bytes.growSlice
       5MB  1.41% 80.53%        5MB  1.41%  go.opentelemetry.io/otel/attribute.computeDistinctFixed

Using go tool web

Environment

• OS: Linux
• Architecture: x86_64
• Go Version: 1.254
• opentelemetry-go version: v0.57.0

Steps To Reproduce

1. Use go 1.24 and go.opentelemetry.io/otel/exporters/prometheus v0.57.0
2. Leave the container running with traffic for several hours and scrapping metrics
3. Memory pattern shows a clear memory increase over time and it get to the limit and gets OOMkilled

Expected behavior

No memory leak

[ns-jvillarfernandez]

Could be related with #6732

[dmathieu]

6732 hasn't been released yet, it's not included in 0.57.0.

[ns-obaro]

6732 hasn't been released yet, it's not included in 0.57.0.

When is the new release schedule? thanks!

[dmathieu]

See #6793

[pree-dew]

@dmathieu Can I pick this task?

[pree-dew]

I tried to replicate this issue, from the few rounds of debugging, it looks like issue is independent of Prometheus exporter and moreover an issue in exemplar functionality. It is more evident when we are emitting high cardinality metrics.

I ran 2 setups with v1.35.0 and v1.36.0 (as they have slight change in FixedSizeReservoir implementation)

v1.35.0

• go.opentelemetry.io/otel/sdk v1.35.0
• go.opentelemetry.io/otel/sdk/metric v1.35.0

Top memory consumers

pprof) top
Showing nodes accounting for 60.50MB, 100% of 60.50MB total
Showing top 10 nodes out of 27
      flat  flat%   sum%        cum   cum%
   38.70MB 63.97% 63.97%    38.70MB 63.97%  math/rand.newSource (inline)
   14.52MB 24.00% 87.96%    14.52MB 24.00%  go.opentelemetry.io/otel/sdk/metric/exemplar.newStorage (inline)
       5MB  8.27% 96.23%        5MB  8.27%  go.opentelemetry.io/otel/attribute.computeDistinctFixed (inline)
       1MB  1.65% 97.88%        1MB  1.65%  go.opentelemetry.io/otel/sdk/metric/internal/aggregate.newBuckets[go.shape.float64]
    0.78MB  1.29% 99.17%    47.49MB 78.50%  go.opentelemetry.io/otel/sdk/metric/internal/aggregate.(*valueMap[go.shape.int64]).measure
    0.50MB  0.83%   100%     7.01MB 11.58%  go.opentelemetry.io/otel/sdk/metric/exemplar.NewHistogramReservoir
         0     0%   100%        5MB  8.27%  go.opentelemetry.io/otel/attribute.NewSet (inline)
         0     0%   100%        5MB  8.27%  go.opentelemetry.io/otel/attribute.NewSetWithFiltered
         0     0%   100%        5MB  8.27%  go.opentelemetry.io/otel/attribute.computeDistinct
         0     0%   100%        5MB  8.27%  go.opentelemetry.io/otel/metric.WithAttributes

Source for top consumer

(pprof) list go.opentelemetry.io/otel/sdk/metric/exemplar.NewFixedSizeReservoir
Total: 60.50MB
ROUTINE ======================== go.opentelemetry.io/otel/sdk/metric/exemplar.NewFixedSizeReservoir in /usr/local/go/pkg/mod/go.opentelemetry.io/otel/sdk/metric@v1.35.0/exemplar/fixed_size_reservoir.go
         0    46.71MB (flat, cum) 77.20% of Total
         .          .     26:func NewFixedSizeReservoir(k int) *FixedSizeReservoir {
         .    46.71MB     27:	return newFixedSizeReservoir(newStorage(k))
         .          .     28:}
         .          .     29:
         .          .     30:var _ Reservoir = &FixedSizeReservoir{}
         .          .     31:
         .          .     32:// FixedSizeReservoir is a [Reservoir] that samples at most k exemplars. If

(pprof) list go.opentelemetry.io/otel/sdk/metric/exemplar.newFixedSizeReservoir
Total: 60.50MB
ROUTINE ======================== go.opentelemetry.io/otel/sdk/metric/exemplar.newFixedSizeReservoir in /usr/local/go/pkg/mod/go.opentelemetry.io/otel/sdk/metric@v1.35.0/exemplar/fixed_size_reservoir.go
         0    38.70MB (flat, cum) 63.97% of Total
         .          .     55:func newFixedSizeReservoir(s *storage) *FixedSizeReservoir {
         .          .     56:	r := &FixedSizeReservoir{
         .          .     57:		storage: s,
         .    38.70MB     58:		rng:     rand.New(rand.NewSource(time.Now().UnixNano())),
         .          .     59:	}
         .          .     60:	r.reset()
         .          .     61:	return r
         .          .     62:}
         .          .     63:

(pprof) list go.opentelemetry.io/otel/sdk/metric/exemplar.newStorage
Total: 60.50MB
ROUTINE ======================== go.opentelemetry.io/otel/sdk/metric/exemplar.newStorage in /usr/local/go/pkg/mod/go.opentelemetry.io/otel/sdk/metric@v1.35.0/exemplar/storage.go
   14.52MB    14.52MB (flat, cum) 24.00% of Total
         .          .     23:func newStorage(n int) *storage {
   14.52MB    14.52MB     24:	return &storage{store: make([]measurement, n)}
         .          .     25:}
         .          .     26:
         .          .     27:// Collect returns all the held exemplars.
         .          .     28://
         .          .     29:// The Reservoir state is preserved after this call.

Memory stats after few gc

Memory: 179.47MB | Heap: 189.17MB | Requests: 19711 (65.7/s) | Exemplars: 19711 | GC: 9

Comments on above profile

• On encountering every unique timeseries (unique attribute set), internally it generates a new FixedSizeReservoir which creates storage and rng. This rng becomes one of the highest memory consumer as internally it creates a fixed size array for every new timeseries. This memory looks like it keeps growing, possibly the root cause of memory leak. Source

====================================================================================

v1.36.0

• go.opentelemetry.io/otel/sdk v1.36.0
• go.opentelemetry.io/otel/sdk/metric v1.36.0

Top memory consumers

top                                                         Showing nodes accounting for 4100.38kB, 100% of 4100.38kB total
Showing top 10 nodes out of 23
      flat  flat%   sum%        cum   cum%
 3588.19kB 87.51% 87.51%  3588.19kB 87.51%  go.opentelemetry.io/otel/sdk/metric/exemplar.newStorage (inline)
  512.19kB 12.49%   100%   512.19kB 12.49%  go.opentelemetry.io/otel/attribute.computeDistinctFixed (inline)
         0     0%   100%   512.19kB 12.49%  go.opentelemetry.io/otel/attribute.NewSet (inline)
         0     0%   100%   512.19kB 12.49%  go.opentelemetry.io/otel/attribute.NewSetWithFiltered
         0     0%   100%   512.19kB 12.49%  go.opentelemetry.io/otel/attribute.computeDistinct
         0     0%   100%   512.19kB 12.49%  go.opentelemetry.io/otel/metric.WithAttributes
         0     0%   100%  2050.50kB 50.01%  go.opentelemetry.io/otel/sdk/metric.(*float64Inst).Record
         0     0%   100%  2050.50kB 50.01%  go.opentelemetry.io/otel/sdk/metric.(*float64Inst).aggregate (inline)
         0     0%   100%  2050.50kB 50.01%  go.opentelemetry.io/otel/sdk/metric.(*inserter[go.shape.float64]).cachedAggregator.func1.reservoirFunc[go.shape.float64].1
         0     0%   100%  1537.69kB 37.50%  go.opentelemetry.io/otel/sdk/metric.(*inserter[go.shape.int64]).cachedAggregator.func1.reservoirFunc[go.shape.int64].1

Source for top consumer

list go.opentelemetry.io/otel/sdk/metric/exemplar.newStorageTotal: 4MB
ROUTINE ======================== go.opentelemetry.io/otel/sdk/metric/exemplar.newStorage in /usr/local/go/pkg/mod/go.opentelemetry.io/otel/sdk/metric@v1.36.0/exemplar/storage.go
    3.50MB     3.50MB (flat, cum) 87.51% of Total
         .          .     23:func newStorage(n int) *storage {
    3.50MB     3.50MB     24:	return &storage{store: make([]measurement, n)}
         .          .     25:}
         .          .     26:
         .          .     27:// Collect returns all the held exemplars.
         .          .     28://
         .          .     29:// The Reservoir state is preserved after this call.

Memory stats after few gc

Memory: 71.14MB | Heap: 80.05MB | Requests: 19740 (65.8/s) | Exemplars: 19740 | GC: 10

Comments on above profile

• For unique timeseries (unique attribute set), a new storage gets created (max size is number of cpus) which becomes highest memory consumer.
• This still consumes way less memory as compared to v1.35.0
• Random sampling is not happening with rand.NewSource so no extra space is required.
• This doesn't look like growing continuously, more over looks like occupy the memory according to cardinality. On every Collect storage resets.
  Source

[ns-obaro]

Hi @pree-dew thanks for the analysis of the issue, unfortunately even using the v1.36.0 of the library the issue is still there. The memory is not growing at the same pace as it used to do in the v1.35.0 but is still growing so it seems the memory is not released in some part. This is the memory consumed by one of our microservices using the v1.36.0 library:

Showing nodes accounting for 43.04MB, 78.18% of 55.05MB total
Showing top 10 nodes out of 143
      flat  flat%   sum%        cum   cum%
    9.52MB 17.29% 17.29%     9.52MB 17.29%  go.opentelemetry.io/otel/sdk/metric/internal/aggregate.reset[go.shape.struct { FilteredAttributes []go.opentelemetry.io/otel/attribute.KeyValue; Time time.Time; Value go.shape.int64; SpanID []uint8 "json:\",omitempty\""; TraceID []uint8 "json:\",omitempty\"" }]
    8.01MB 14.56% 31.84%     8.01MB 14.56%  go.opentelemetry.io/otel/sdk/metric/exemplar.newStorage
    6.51MB 11.83% 43.67%     6.51MB 11.83%  go.opentelemetry.io/otel/sdk/metric/internal/aggregate.reset[go.shape.struct { FilteredAttributes []go.opentelemetry.io/otel/attribute.KeyValue; Time time.Time; Value go.shape.float64; SpanID []uint8 "json:\",omitempty\""; TraceID []uint8 "json:\",omitempty\"" }]
       5MB  9.08% 52.75%        5MB  9.08%  github.com/prometheus/client_golang/prometheus.(*constHistogram).Write
    3.50MB  6.36% 59.11%     3.50MB  6.36%  github.com/prometheus/client_golang/prometheus.newExemplar
       3MB  5.45% 64.56%        3MB  5.45%  github.com/prometheus/client_golang/prometheus.MakeLabelPairs
       2MB  3.63% 68.19%        8MB 14.53%  go.opentelemetry.io/otel/exporters/prometheus.addHistogramMetric[go.shape.float64]
       2MB  3.63% 71.83%        2MB  3.63%  go.opentelemetry.io/otel/attribute.computeDistinctFixed
       2MB  3.63% 75.46%     8.52MB 15.47%  github.com/prometheus/client_golang/prometheus.processMetric
    1.50MB  2.72% 78.18%     1.50MB  2.72%  encoding/hex.EncodeToString (inline)

it seems go.opentelemetry.io/otel/sdk/metric/internal/aggregate.reset its retaining some memory.

[pellared]

@ns-obaro, what about the following?

It is more evident when we are emitting high cardinality metrics.

Is it not the root-cause of the issue?

[ns-obaro]

@ns-obaro, what about the following?

It is more evident when we are emitting high cardinality metrics.

Is it not the root-cause of the issue?

We have some metrics with high cardinality but the memory consumption should stop or be released at some point but we don't see that as you can see in this graph:

[pree-dew]

@ns-obaro Thank you for your inputs. Can you help me a bit more here?

• In the highlighted area in above screenshot, can you check if your sampling rate(no. of samples per min) has increased during that time.
• After that, is the sampling rate maintained or has it decreased? Do you still see the same memory consumption after the rate is decreased?

[pellared]

We have some metrics with high cardinality but the memory consumption should stop or be released at some point

High cardinality means many unique combinations of metric attributes (labels), which leads to many aggregation points.
In the OpenTelemetry Metrics SDK, once an attribute set has been seen, it is tracked in memory for further aggregation.
These aggregations (including exemplars, reservoirs, and metric storage) persist for the lifetime of the SDK

Therefore, the Metrics SDK still needs to keep all the metric aggregations (and exemplars, reservoirs) for all instrument attribute values that were encountered. Therefore, the memory is growing.

[pree-dew]

@pellared My assumption here is, if the system has seen peak cardinality then the memory will not increase beyond that. So it will maintain the same memory which was allocated during the peak cardinality.

Is this correct? That is what I observed during my analysis.

This doesn't look like growing continuously, more over looks like occupy the memory according to cardinality.

[pellared]

The cardinality limits are currently an opt-in and experimental feature.
See: https://pkg.go.dev/go.opentelemetry.io/otel/sdk/metric/internal/x#section-readme