OpenTelemetry Go: Increasing memory usage and OOM when tracing is enabled

[varunsdevang]

We’re seeing steadily increasing memory usage (not released) in a few high-traffic Go services when OpenTelemetry tracing is enabled, eventually leading to OOM errors. This does not occur when tracing is disabled.

Setup:

OpenTelemetry Go SDK

Memory limit: 4GB per service

Issue observed only in high-throughput services

Tracing details:

Span names include UUIDs / high-cardinality values

DB spans with commands enabled

Tracing for incoming requests, outgoing gRPC and HTTP calls

Behavior:

Memory keeps growing over time and does not stabilize

GC doesn’t reclaim enough memory

Disabling traces stabilizes memory usage

Looking for insights on:

Impact of high-cardinality span names on memory usage

Known issues with DB command spans or exporters under load

Recommended sampling / span limits / batching configs for Go tracing

Similar experiences with OTel Go in production

Any guidance or best practices would be appreciated.

[flc1125]

If there is a minimized, reproducible code example, it might be more helpful for troubleshooting the issue.

[varunsdevang]

Sure. Here are some of the wrapper methods. We use go swagger for http server. Currently there is no sampling and there are methods/APIs which produce over 5000+ spans. Mongo Command attributes are enabled by default. we have handled span.End() with differ properly.

func HTTPTracingMiddleware(next http.Handler) http.Handler {

	formatter := getSpanNameFormatter()
	if !IsTracingEnabled() {
		return next
	}

	return otelhttp.NewHandler(next, "",
		otelhttp.WithTracerProvider(otel.GetTracerProvider()),
		otelhttp.WithSpanNameFormatter(formatter))
}

// SpanWithAttributes creates a new span with the given attributes and returns a context and a function to end the span.
func SpanWithAttributes(ctx context.Context, spanName string, spanKind string, attributes map[string]string) (context.Context, func()) {
	if !IsTracingEnabled() {
		return ctx, func() {}
	}

	if ctx == nil {
		return ctx, func() {}
	}

	ctx, span := otel.Tracer("").Start(ctx, spanName, trace.WithSpanKind(getSpanKind(spanKind)))

	for key, value := range attributes {
		span.SetAttributes(attribute.String(key, value))
	}

	return ctx, func() {
		span.End()
	}
}

func NewMongoDBMonitor() *event.CommandMonitor {
	// Get global config for tracing settings
	if !IsTracingEnabled() {
		return &event.CommandMonitor{}
	}

	inner := otelmongo.NewMonitor(
		otelmongo.WithTracerProvider(otel.GetTracerProvider()),
		otelmongo.WithCommandAttributeDisabled(isMongoCommandMonitoringDisabled()),
	)

	return &event.CommandMonitor{
		Started: func(ctx context.Context, evt *event.CommandStartedEvent) {
			if HasValidSpan(ctx) {
				inner.Started(ctx, evt)
			}
		},
		Succeeded: func(ctx context.Context, evt *event.CommandSucceededEvent) {
			if HasValidSpan(ctx) {
				inner.Succeeded(ctx, evt)
			}
		},
		Failed: func(ctx context.Context, evt *event.CommandFailedEvent) {
			if HasValidSpan(ctx) {
				inner.Failed(ctx, evt)
			}
		},
	}
}

[pellared]

Memory limit: 4GB per service

Have you set the GOMEMLIMIT environment variable?

[varunsdevang]

No, the memory limit is through k8s

[pellared]

Then this is probably the reason why you get OOM. Read more e.g. here https://www.ardanlabs.com/blog/2024/02/kubernetes-memory-limits-go.html

[flc1125]

This isn't the final answer, as the information provided is insufficient.

However, based on my experience, I initially suspect that the validation logic of HasValidSpan might not be rigorous enough, resulting in the span being created but not being effectively Ended afterward.

If we provide more information, it might be easier to identify the problem, such as: pprof and the relatively complete code.

[varunsdevang]

I did pprof of the affected service in my local environment. but i was not able reproduce the issue.

func HasValidSpan(ctx context.Context) bool {
	if ctx == nil {
		return false
	}
	span := trace.SpanFromContext(ctx)
	return span.SpanContext().IsValid()
}

Initialization

func initTracing(ctx context.Context, cfg Config) (*sdktrace.TracerProvider, error) {
	// Read configuration from environment
	endpoint := os.Getenv("TRACING_COLLECTOR_ENDPOINT")
	disableSpanFormatter := os.Getenv("SPAN_NAME_FORMATTER_DISABLED")

	if strings.EqualFold(disableSpanFormatter, "true") {
		cfg.Tracing.SpanNameFormatterDisabled = true
	}

	// Create OTLP gRPC exporter
	exporter, err := otlptracegrpc.New(ctx, otlptracegrpc.WithEndpoint(endpoint))
	if err != nil {
		return nil, fmt.Errorf("failed to create OTLP exporter: %w", err)
	}

	// Create TracerProvider with batching and service resource attributes
	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(resource.NewWithAttributes(
			semconv.SchemaURL,
			semconv.ServiceNameKey.String(cfg.ServiceName),
			semconv.ServiceVersionKey.String(cfg.ServiceVersion),
			semconv.TelemetrySDKLanguageGo,
		)),
	)

	// Set global TracerProvider and propagator
	otel.SetTracerProvider(tp)
	otel.SetTextMapPropagator(
		propagation.NewCompositeTextMapPropagator(
			propagation.TraceContext{},
			propagation.Baggage{},
		),
	)

	return tp, nil
}

[varunsdevang]

Thanks for all the suggestion, I'll try these fixes