Thread pool of long-lived workers builds up arbitrarily large traces

[Tomboyo]

My team maintains a SpringBoot application which creates an executor service via Executors.newFixedThreadPool. A spring-boot @Scheduled method on one of our @Services checks the number of workers (instances of a class that implements Runnable) active in the pool, and if it is below a desired number, calls #exectue(Runnable) to schedule workers up to the desired number. Typically, those workers are long-lived; their #run() method draws tasks from a blocking queue indefinitely, and the @Scheduled method is just there to supervise the number of instances in case one unexpectedly fails. We use the java agent to collect traces, and locally with Jaeger I can see that the auto-instrumentation is aggregating all the spans from the disparate workers into a single trace with the @Scheduled method as the root. This causes the traces to become arbitrarily large over time. What we actually want to achieve is a trace per task processed by each worker (so, start a trace for each #processTask method invocation called by the #run loop).

I would like to understand what's going on and what somebody who understands how this works would do to fix it. From what I can gather from the docs and source code, the auto-instrumentation will start a span for the @Scheduled method, and the executors auto-instrumentation seems to instrument the #exectue(Runnable) method such that spans created by the workers become part of this singular trace, is that right?

My only successful attempt at solving the problem was to disable the executors instrumentation and pop a @WithSpan on our #processTask method that the workers invoke in their #run loop. I don't know yet if this affects any other part of my app, but I'm also curious if there's a more surgical way to override the auto-instrumentation behavior for this worker pool.

Thanks!

[laurit]

Executor instrumentation propagates the context from calling the execute method into the Runnable. If execute was called inside some span then that span will be the parent of all the work that is going to happen inside the Runnable. One approach to avoid this would be to reset the context before calling execute with

try (Scope ignore = Context.root().makeCurrent()) {
  execute(...);
}

Another option would be to call

opentelemetry-java-instrumentation/javaagent-extension-api/src/main/java/io/opentelemetry/javaagent/extension/ignore/IgnoredTypesBuilder.java

Line 93 in 1cddf1b

IgnoredTypesBuilder ignoreTaskClass(String classNameOrPrefix);

with the class name of the runnable. This could be done inside an agent extension https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/examples/extension Another possibility would be to add an option for users to supply the names of the ignore task classes similarly to https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/javaagent-tooling/src/main/java/io/opentelemetry/javaagent/tooling/ignore/UserExcludedClassesConfigurer.java

[Tomboyo]

Thanks for this, this was really informative! The Context.root().makeCurrent() approach works perfectly.