Spring Webflux Context issue with `TRACE_CONTEXT_KEY`

[dlefler-chwy]

I have been attempting to get open telemetry working in my Spring Webflux application. The application uses a data dog java agent. Below is the configuration:

Spring boot version: 3.4.1
Spring webflux version: 6.2.1
opentelemetry-sdk version: 1.47.0
opentelemetry-api version: 1.47.0
opentelemetry-sdk-extension-autoconfigure: 1.47.0
opentelemetry-spring-webflux-5.3: 2.12.0-alpha

I am able to successfully instrument my application using just the open telemetry api. However, I want to add custom span attributes to my spans and in order to do this, I added opentelemetry-spring-webflux-5.3: 2.12.0-alpha. The reason I thought that this library would help access the correct span was that TRACE_CONTEXT_KEY would be added to the reactor context. When debugging my application, I do not see this in the reactor context and stepping through the code,storeOpenTelemetryContext is never called.

Is there something wrong with the way I have set this up? I have set up the correct beans following the README for the webflux library. Any help would be greatly appreciated!

[trask]

The application uses a data dog java agent

hi @dlefler-chwy! I suspect you'll need to go to data dog for this question

[dlefler-chwy]

The application uses a data dog java agent

hi @dlefler-chwy! I suspect you'll need to go to data dog for this question

Hi @trask! Thanks for the pointer. I took some time yesterday to try to see if I could get my code to run locally with the opentelemetry-javaagent as well. I was able to do this and experienced the same behavior.

I am relatively new to webflux as well as setting up observability but what would make you think that the opentelemetry-spring-webflux-5.3 library would not be working correctly due to the datadog agent? Is it something to do with the agent triggering a write to the reactor context for TRACE_CONTEXT_KEY or something like that? I see plenty of attempts to read TRACE_CONTEXT_KEY from the reactor context.

[trask]

if you can create a minimal repro (preferably as a github repository with instructions) of the issue using the opentelemetry-javaagent that would be great

[dlefler-chwy]

if you can create a minimal repro (preferably as a github repository with instructions) of the issue using the opentelemetry-javaagent that would be great

@trask here is a project reproing the issue: https://github.com/dlefler-chwy/webflux-otel-context-issue

[laurit]

I think the issue is that you are using global open telemetry in https://github.com/dlefler-chwy/webflux-otel-context-issue/blob/fe913a0547e203a53299d4c7fc291097ad288730/src/main/java/com/example/reactivewebservice/ReactiveWebServiceApplication.java#L22 Global open telemetry is meant to be used with the java agent where it returns the open telemetry instance configured by the agent. By default it will return a no-op instance that doesn't do anything. If you'd replace it with AutoConfiguredOpenTelemetrySdk.initialize() then your app would probably produce some spans. Secondly if you wish to add an attribute to current span you should use Span.current().setAttribute("foo", "bar"). Open telemetry context is not always set to reactor context.

[dlefler-chwy]

I think the issue is that you are using global open telemetry in https://github.com/dlefler-chwy/webflux-otel-context-issue/blob/fe913a0547e203a53299d4c7fc291097ad288730/src/main/java/com/example/reactivewebservice/ReactiveWebServiceApplication.java#L22 Global open telemetry is meant to be used with the java agent where it returns the open telemetry instance configured by the agent. By default it will return a no-op instance that doesn't do anything. If you'd replace it with AutoConfiguredOpenTelemetrySdk.initialize() then your app would probably produce some spans. Secondly if you wish to add an attribute to current span you should use Span.current().setAttribute("foo", "bar"). Open telemetry context is not always set to reactor context.

Thanks @laurit.

I tried running a debugger on the application start up for my sample application and the OpenTelemetry bean does not appear to be configuring to the no-op instance. I also attempted to use your suggested AutoConfiguredOpenTelemetrySdk which works but also results in the warning message:

io.opentelemetry.api.GlobalOpenTelemetry - You are currently using the OpenTelemetry Instrumentation Java Agent; all GlobalOpenTelemetry.set calls are ignored - the agent provides the global OpenTelemetry object used by your application

I think the warning makes sense since the sample project does use the agent so we would expect it to configure the OpenTelemty instance over the sdk.

As per the suggestion about adding to the span, shouldn't using Span.current().setAttribute("foo", "bar") not work in a reactive application? From what I can tell, this uses ThreadLocal to store the Span context which means that calling Span.current may return an incorrect Span for the current request. Is there something here I am missing?

[laurit]

I tried running a debugger on the application start up for my sample application and the OpenTelemetry bean does not appear to be configuring to the no-op instance. I also attempted to use your suggested AutoConfiguredOpenTelemetrySdk which works but also results in the warning message:
io.opentelemetry.api.GlobalOpenTelemetry - You are currently using the OpenTelemetry Instrumentation Java Agent; all GlobalOpenTelemetry.set calls are ignored - the agent provides the global OpenTelemetry object used by your application
I think the warning makes sense since the sample project does use the agent so we would expect it to configure the OpenTelemty instance over the sdk.

So you are using the agent. In that case GlobalOpenTelemetry.get() will return the OpenTelemetry instance configured by the agent. Since agent already includes the spring webflux instrumentation adding the same instrumentation again manually does not make sense.

As per the suggestion about adding to the span, shouldn't using Span.current().setAttribute("foo", "bar") not work in a reactive application? From what I can tell, this uses ThreadLocal to store the Span context which means that calling Span.current may return an incorrect Span for the current request. Is there something here I am missing?

Instrumentation restores the ThreadLocal context for the reactive callbacks so Span.current() should work.

[dlefler-chwy]

Thanks @laurit for clarifying usage. I think that this thread can be closed as I think auto instrumentation should be able to support our use cases.