OtlpGrpcSpanExporter with Netty transport creates unbounded grpc-default-worker threads

[punya]

Describe the bug

When using OtlpGrpcSpanExporter with the gRPC Netty transport (grpc-netty or grpc-netty-shaded), the exporter creates unbounded grpc-default-worker threads over time, leading to memory exhaustion and eventual OOM.

Steps to reproduce

1. Configure OtlpGrpcSpanExporter using the managed channel gRPC sender (i.e., exclude opentelemetry-exporter-sender-okhttp and add opentelemetry-exporter-sender-grpc-managed-channel)
2. Run the application under normal load
3. Monitor thread count over time

What did you expect to see?

A stable, bounded number of gRPC/Netty worker threads.

What did you see instead?

Thread count grows continuously. In our case, we observed 5-7 new grpc-default-worker threads created approximately every 10 seconds, with none of them terminating. Each thread uses ~1MB of stack space, leading to significant memory growth (~2.7GB/hour in native memory).

Root cause analysis

When using ManagedChannelBuilder.forTarget() without explicitly configuring an event loop group, Netty defaults to using ThreadPerTaskExecutor for its internal worker threads. Unlike a bounded thread pool, this executor creates a new thread for each task and does not reuse threads.

The GrpcExporterBuilder in opentelemetry-java creates a channel via ManagedChannelBuilder but does not configure:

1. A bounded EventLoopGroup via NettyChannelBuilder.eventLoopGroup()
2. Or limit the channel's internal threading behavior

Suggested fix

When building the managed channel for Netty transport, use NettyChannelBuilder directly with a bounded NioEventLoopGroup:

NioEventLoopGroup eventLoopGroup = new NioEventLoopGroup(2); // or some reasonable bounded number
NettyChannelBuilder.forTarget(endpoint)
    .eventLoopGroup(eventLoopGroup)
    .channelType(NioSocketChannel.class)
    // ... other configuration
    .build();

This ensures Netty reuses a fixed pool of event loop threads rather than creating unbounded new threads.

Environment

• OS: Linux (containers)
• Java version: 21
• OpenTelemetry version: 1.38.x
• gRPC version: 1.78.0
• Transport: grpc-netty-shaded

Additional context

This issue is distinct from:

• OtlpGrpcExporter/Netty still active after SdkTracerProvider#shutdown() #3521 (threads active after shutdown) - that was about cleanup, this is about unbounded creation during normal operation
• DNS creates unbounded number of grpc-default-executor threads, and can't be overriden grpc/grpc-java#3090 (unbounded DNS executor threads) - that was about the offload executor, this is about Netty's internal event loop threads

The default OkHttp sender may not exhibit this behavior, but users who switch to the managed channel sender with Netty transport will encounter it.

[punya]

cc @matt-goroff who actually found the issue.

[punya]

Update: Reproduction Project

I've created a self-contained reproduction project demonstrating the thread behavior:

Findings from reproduction:

1. A single channel with ManagedChannelBuilder creates 2 * availableProcessors() Netty event loop threads by default (e.g., 10 threads on a 5-core machine)

2. These threads have names like grpc-nio-worker-ELG-1-*

3. ManagedChannelBuilder.executor() only controls the callback executor, not the Netty event loop threads

4. In our production environment, we observed the exact behavior described in the issue (5-7 new threads every 10 seconds with continuous growth), but this precise pattern was difficult to reproduce in isolation. It may depend on specific conditions like connection failures, retry patterns, or application load.

Simple reproduction (build.gradle):

plugins {
    id 'java'
    id 'application'
}

java {
    toolchain { languageVersion = JavaLanguageVersion.of(21) }
}

repositories { mavenCentral() }

dependencies {
    implementation 'io.grpc:grpc-netty-shaded:1.64.0'
    implementation 'io.grpc:grpc-protobuf:1.64.0'
    implementation 'io.grpc:grpc-stub:1.64.0'
    implementation 'io.opentelemetry.proto:opentelemetry-proto:1.1.0-alpha'
}

application { mainClass = 'repro.ThreadLeakRepro' }

ThreadLeakRepro.java:

package repro;

import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;

public class ThreadLeakRepro {
    public static void main(String[] args) throws InterruptedException {
        System.out.println("Initial grpc threads: " + countGrpcThreads());
        
        ManagedChannel channel = ManagedChannelBuilder
                .forAddress("localhost", 4317)
                .usePlaintext()
                .build();
        
        // Trigger channel initialization
        channel.getState(true);
        Thread.sleep(1000);
        
        System.out.println("After channel creation: " + countGrpcThreads());
        printThreadNames();
        
        channel.shutdown();
    }
    
    private static int countGrpcThreads() {
        return (int) Thread.getAllStackTraces().keySet().stream()
                .filter(t -> t.getName().contains("grpc") || t.getName().contains("netty"))
                .count();
    }
    
    private static void printThreadNames() {
        Thread.getAllStackTraces().keySet().stream()
                .map(Thread::getName)
                .filter(n -> n.contains("grpc") || n.contains("netty"))
                .sorted()
                .forEach(System.out::println);
    }
}

Output on a 5-core machine:

Initial grpc threads: 0
After channel creation: 10
grpc-nio-worker-ELG-1-1
grpc-nio-worker-ELG-1-2
...
grpc-nio-worker-ELG-1-10

The issue is that there's no obvious way to limit these threads when using the standard OtlpGrpcSpanExporter API - users would need to either:

1. Switch to OkHttp sender (which may have different tradeoffs)
2. Build a custom channel with NettyChannelBuilder.eventLoopGroup()

[jack-berg]

Here is the code where we initialized a ManagedChannel for you if you dont specify your own via OtlpGrpc{Signal}ExporterBuilder#setChannel(ManagedChannel).

What's strange about this issue is that we're not doing anything particularly interesting. So this implies that the default netty transport managed channel implementation also creates unbounded worker threads, which is surprising to me.

    ManagedChannelBuilder<?> channelBuilder =
        ManagedChannelBuilder.forAddress(endpoint.getHost(), endpoint.getPort());
    if (!endpoint.getScheme().equals("https")) {
      channelBuilder.usePlaintext();
    }
    return channelBuilder.build();

[jack-berg]

When using ManagedChannelBuilder.forTarget() without explicitly configuring an event loop group, Netty defaults to using ThreadPerTaskExecutor for its internal worker threads.

I'm looking at the source and see something else. I see the default eventLoopGroup set here: https://github.com/grpc/grpc-java/blob/master/netty/src/main/java/io/grpc/netty/NettyChannelBuilder.java#L88-L89

Which resolves from: https://github.com/grpc/grpc-java/blob/master/netty/src/main/java/io/grpc/netty/Utils.java#L127-L128

Which is constructed here: https://github.com/grpc/grpc-java/blob/76a631bcf772288f95050adfae3d890ba9ecca1a/netty/src/main/java/io/grpc/netty/Utils.java#L489-L508

If I'm reading the code right, it looks like it tries to initialize the thread group to the io.netty.eventLoopThreads system property, or the number of processors if not set.

What am I missing?

[ps-mir]

I've struggled so far to reproduce this with the given ThreadLeakRepro setup

java -version
openjdk version "21" 2023-09-19
OpenJDK Runtime Environment (build 21+35-2513)
OpenJDK 64-Bit Server VM (build 21+35-2513, mixed mode, sharing)

➜ gradle run
> Task :run
Initial grpc threads: 0
After channel creation: 5
grpc-default-executor-0
grpc-default-worker-ELG-1-1
grpc-default-worker-ELG-1-2
grpc-default-worker-ELG-1-3
grpc-default-worker-ELG-1-4

BUILD SUCCESSFUL in 2s

nproc=4 (4 Core)

Result is same with gRPC 1.64/1.78, Java 21/25

Not sure what I'm missing.

[github-actions]

This has been automatically marked as stale because it has been marked as needing author feedback and has not had any activity for 7 days. It will be closed if no further activity occurs within 7 days of this comment.