Wrong span parent due to WithSpan when using java-agent with Ktor server and client

[LangdalP]

Describe the bug

I have stumbled upon a weird bug. The setup is this:

• A Ktor server, version 3.3.0
• The server is running with opentelemetry-javaagent version 2.20.0
• A route is calling a third party API endpoint with a Ktor client. We call this client instance "client A".
• The third party API requires a bearer token. To fetch the token we use the Auth plugin and fetch the token with a second client, called "client B".
• There is also a method annotated with @WithSpan: fetchCountriesReturnStatus()

With this relatively common configuration, we observe the following issue: Whenever a request (using client A) triggers fetching a new token (using client B), that request's span has the wrong parent set. Please see the screenshot at the bottom of this post. The POST span is a token request triggered by the Auth plugin, using client B. The GET span is the span for the actual request the server wants to make, using client A. Note that the GET span is placed directly under the GET / span. It should be under the fetchCountriesReturnState span. I have not been able to pinpoint the exact reason this happens, but it is fairly consistent.

Steps to reproduce

I have made a reproducible case in a separate GitHub repository here. Instructions are provided in the README.

Expected behavior

The GET span in the screenshot should be nested inside the fetchCountriesReturnState span.

Actual behavior

The GET span for the request in the screenshot is placed directly under the root level GET / span.

Javaagent or library instrumentation version

opentelemetry-javaagent version 2.20.0
opentelemetry-instrumentation-annotations version 2.20.0

Environment

JDK: openjdk 17.0.16 2025-07-15
OS: MacOS Sequoia 15.6.1

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[breedx-splk]

Hey @LangdalP thanks for the issue report and the repro repo.

I think I was able to reproduce this. In our product, it looks similar:

I think you have some oddities or confusing statements in the wording in your issue, such as

The GET / span in the screenshot is placed directlyt under the GET / span.

I don't think that's what you meant...but I think you mean that the bottommost GET should be parented under the internal fetchCountriesReturnState span. Is that right? Maybe revise your description for clarity please if so.

Anyway, this stuff gets complicated quickly in the async world. In the spirit of classic, dumb guy troubleshooting, I sprinkled a bunch of println statements around in the code, and you can see my version here: https://gist.github.com/breedx-splk/51ccb2464441d23fe7321aa280339855

With that in place, the output for the server startup and first request looks something like this:

> Task :run
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[otel.javaagent 2025-09-23 16:03:50:766 -0700] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 2.20.1
2025-09-23 16:03:51.721 [main] INFO  Application - Autoreload is disabled because the development mode is off.
loading bearer tokens start
after loading bearer tokens
2025-09-23 16:03:51.843 [main] INFO  Application - Application started in 0.252 seconds.
2025-09-23 16:03:51.865 [DefaultDispatcher-worker-1] INFO  Application - Responding at http://0.0.0.0:8080
pre fetch countries
in fetch pre get
inside loadTokens before
inside loadTokens after
in fetch post get
have the status in hand
after fetchery
post fetch countries

That sequence looks good and not terribly surprising to me, so this does feel like a context problem in the instrumentation. it should be noted that the client get inside the route is what ultimately initiates the token loading, it's deferred as late as possible by ktor. My hunch is that the deferred fetch of the tokens is what's causing the problem, but I haven't dug into it any deeper.

Thanks!

[laurit]

@LangdalP I didn't try running your application, but I know that @WithSpan annotation doesn't always work correctly with coroutines. Try creating that span manually and see whether that resolves your issue. See

opentelemetry-java-instrumentation/instrumentation/ktor/ktor-3.0/testing/src/main/kotlin/io/opentelemetry/instrumentation/ktor/v3_0/AbstractKtorHttpServerTest.kt

Lines 105 to 116 in 5c027b5

	val span = getTesting().openTelemetry.getTracer("test").spanBuilder("controller").setSpanKind(SpanKind.INTERNAL).startSpan()
	try {
	withContext(Context.current().with(span).asContextElement()) {
	wrapped()
	}
	span.end()
	} catch (e: Exception) {
	span.setStatus(StatusCode.ERROR)
	span.recordException(if (e is ExecutionException) e.cause ?: e else e)
	span.end()
	throw e
	}

and

opentelemetry-java-instrumentation/instrumentation/kotlinx-coroutines/kotlinx-coroutines-1.0/javaagent/src/test/kotlin/io/opentelemetry/javaagent/instrumentation/kotlinxcoroutines/KotlinCoroutinesInstrumentationTest.kt

Lines 472 to 480 in 5c027b5

	private suspend fun a(iter: Long) {
	var span = tracer.spanBuilder("a").startSpan()
	span.setAttribute("iter", iter)
	withContext(span.asContextElement()) {
	delay(10)
	a2(iter)
	}
	span.end()
	}

for samples on how to create a span and add it to coroutine context.

[LangdalP]

@breedx-splk
Thanks, I have adjusted the confusing wording.

@laurit
I tried your suggestion of creating the span manually myself. It did seem to fix the problem, indicating the problem is connected to the use of @WithSpan.

From blog posts such as this I was led to believe that @WithSpan should work with coroutines, but these results indicate that bugs remain. Do you know if there is any ongoing work to make it more robust?

[laurit]

From blog posts such as this I was led to believe that @WithSpan should work with coroutines, but these results indicate that bugs remain. Do you know if there is any ongoing work to make it more robust?

I don't know of anybody working on this. I tried improving it, but failed. In retrospect I suspect that attempting to support @WithSpan might have been a mistake, it might be better to provide some sort of library method like

withSpan("foo") {
...
}

and change the @WithSpan instrumentation so that it wouldn't try to create spans for corotines.

[LangdalP]

Thanks for your reply. Is there something fundamentally challenging or incompatible about using annotations such as @WithSpan for this? Since you seem to have explored it, can you say a few words about why it is so hard to make @WithSpan work with coroutines? I am quite curious about the subject, and not well-versed when it comes to annotation processing.

If that approach is as riddled with footguns as your answer implies, I agree that attempting to support @WithSpan is a mistake.

[laurit]

Is there something fundamentally challenging or incompatible about using annotations such as @WithSpan for this?

@WithSpan instrumentation works by instrumenting the method and turns it into something like

start span
try {
  original method body
} finally {
  end span
}

this strategy obviously doesn't work for coroutines as coroutines are compiled into something like (I'm writing this by heart, hopefully close enough to reality)

method(int index) {
  swith(index) {
    case 1:
     a chunk of the method body
     return continuation
    case 2:
     a chunk of the method body
     return continuation
  }
}

first you execute code starting with the initial state until you reach something that can suspend and return a continuation. Later to resume execution the same method is called again and resumes execution from the next state. With the regular @WithSpan instrumentation you'd get a span for each of the executed states. To overcome this we need to instrument coroutines differently, start the span in initial state and end it when execution reaches terminal state. Each time execution enters/leaves the method context needs to be restored/saved. That last part is what we have not managed to do correctly.

[trask]

it might be better to provide some sort of library method like

withSpan("foo") {
...
}

and change the @WithSpan instrumentation so that it wouldn't try to create spans for corotines.

👍

[trask]

Let's consider disabling @WithSpan coroutine support in 3.0 by default at least.