[java] How to inject traceid into the exception handler of the thread pool

[MartyAlien]

hi all:

When using Java's thread pool, there is a trace ID in asynchronous threads. However, when an exception occurs in asynchronous threads that has not been manually captured, it will be captured by the exception handler configured by the thread pool. The trace ID cannot be obtained from the exception handler, resulting in a lack of key information about the thread pool exception handler when using log link tracing.

[svrnm]

Thanks for your question @MartyAlien

@open-telemetry/java-approvers please take a look

[trask]

Transferring to https://github.com/open-telemetry/opentelemetry-java

[trask]

hi @MartyAlien! I think I understand your question, that the exception handler occurs after the scope (which contains the trace id) is closed, but could you include some code snippets to make sure we're talking about the same thing?

[breedx-splk]

@MartyAlien can you also clarify if you're using the java auto-instrumentation agent or if you're doing manual instrumentation with the sdk?

[MartyAlien]

hi @trask , sure!This is my code snippet, it's about initializing a thread pool:

    // ... ignore other details
    ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
    // ... ignore other details
    executor.setThreadFactory(new NamedThreadFactory(prefix
              , null
              , false
              , (t, e) -> log.warn("【{}】Exception message - {}", t.getName(), e.getMessage(), e))
      );
    // ... ignore other details
    executor.initialize();

hi @breedx-splk The jar package where the code is located adds opentelemetriy-javaagent.jar and its related startup parameters at startup.
The approximate startup parameters are as follows, please ignore the variable names involved:

-javaagent:/opt/opentelemetry-javaagent.jar
-Dotel.exporter.otlp.traces.endpoint=$OB_ENDPOINT/traces
-Dotel.exporter.otlp.metrics.endpoint=$OB_ENDPOINT/v1/metrics
-Dotel.exporter.otlp.logs.endpoint=$OB_ENDPOINT/v1/logs
-Dotel.exporter.otlp.traces.headers="Authorization=Basic $AUTH_TOKEN"
-Dotel.exporter.otlp.metrics.headers="Authorization=Basic $AUTH_TOKEN"
-Dotel.exporter.otlp.logs.headers="Authorization=Basic $AUTH_TOKEN,stream-name=$APP_NAME"
-Dotel.exporter.otlp.traces.protocol=http/protobuf
-Dotel.exporter.otlp.metrics.protocol=http/protobuf
-Dotel.exporter.otlp.logs.protocol=http/protobuf
-Dotel.service.name=$APP_NAME
-Dotel.resource.attributes="service.name=$APP_NAME"
-Dotel.logs.exporter=otlp
-Dotel.metrics.exporter=otlp
-Dotel.traces.exporter=otlp
-Dotel.metric.export.interval=10000
-Dotel.traces.sampler=parentbased_always_on
-Dotel.traces.sampler.arg=1.0
-Dotel.experimental.resource.disabled-keys=process_command_line,process.command_args,service_process_command_args,process.runtime.description,process.runtime.name,process.executable.path
-Dotel.instrumentation.logback-appender.experimental-log-attributes=true
-Duser.timezone=GMT+8

The screenshot of the log shows that the exception encountered by the asynchronous thread was captured by the exception handler in the thread factory and a log record was printed, but the traceId is missing from the log record

[trask]

transferring to the instrumentation repo to see if there's anything we can do

[laurit]

@MartyAlien it would help if you could provide a minimal application that reproduces the issue. I'd guess ThreadPoolTaskExecutor is from spring, but I have no idea what NamedThreadFactory is. At first glance I'd say that implementing this is not going to be easy.

[MartyAlien]

@laurit ThreadPoolTaskExecutor is from org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.The Maven coordinates of this class are:

      <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
        <version>2.6.13</version>
      </dependency>

[laurit]

Assuming NamedThreadFactory is custom code I don't think there is anything we can do here. We don't even know how it captures the exception. Perhaps it sets an uncaught exception handler or decorates the runnable passed to the constructor of the Thread. You might have better luck using org.springframework.core.task.TaskDecorator to wrap the original Runnable and catch and log the exception there.

[github-actions]

This has been automatically marked as stale because it has been marked as needing author feedback and has not had any activity for 7 days. It will be closed automatically if there is no response from the author within 7 additional days from this comment.