SpanId changes on WebClient request

[pratapjai143]

Spring 3.1.4

As suggested by micrometer community micrometer-metrics/micrometer#5472 I am opening an issue with Spring.

Would you please suggest if there is any way to propagate the same span id to child service rather than creating a new span id?

[bclozel]

Please notice that this is likely to be a WebClient feature and does not need a change in Micrometer, I would recommend opening am issue for Spring Framework.

@jonatan-ivanov could you elaborate on what this means from an Observation API point of view?

[jonatan-ivanov]

See this comment from @chemicL: micrometer-metrics/micrometer#5472 (comment)

This issue confuses me a bit since the description does not match to what my comment was referring to. :)

I think the description of this refers to Brave's joint span feature which I explained in micrometer-metrics/micrometer#5472 and Framework has nothing to do with it, you can enable it with a Boot property. If this is what @pratapjai143 wants, this issue can be closed.

What we discussed in micrometer-metrics/micrometer#5472 (comment) and what my comment was referring to is a WebClient interceptor that is wrapped by the client span (right now it seems it is wrapped by the parent of the client span).

I'm not really sure what @pratapjai143 really wants.

[pratapjai143]

Hi @jonatan-ivanov I have already enabled the span-joining-supported: true still all the span ids are different in the downstream.

Would you please tell me if I have multiple request to child service for a single call to my parent then how can I trace each request separately in child service in relation to parent service even if the trace id is same.

Please let me know if I need to elaborate the issue again.

Thanks.

[pratapjai143]

User calls A (traceId: 0, spanId: 1, parentId: null)

• A calls B first time (traceId: 0, spanId: 2, parentId: 1)
  

• A calls B second time (traceId: 0, spanId: 3, parentId: 1)
    B calls C (traceId: 0, spanId: 4, parentId: 3)
  

Parent Log:-

Child Log:-

If you see the child log only trace id is same however that trace id is same across all the requests sent from the parent service for a single request from use to parent service.

If one of the request will be failed in the child service while processing how can I related which request is failed by looking into the parent and child service logs as the span id is not same and the trace id can't be used to trace a specific request in child service as the trace id is same for all requests.

Hope this gives more visibility of the issue.

Thanks.

[chemicL]

@jonatan-ivanov my suggestion to execute interceptors (Filters actually) on behalf of the child Observation can be achieved via the following patch to the framework codebase:

--- a/spring-webflux/src/main/java/org/springframework/web/reactive/function/client/DefaultWebClient.java
+++ b/spring-webflux/src/main/java/org/springframework/web/reactive/function/client/DefaultWebClient.java
@@ -463,8 +463,9 @@ final class DefaultWebClient implements WebClient {
                                ClientRequest request = requestBuilder.build();
                                observationContext.setUriTemplate((String) request.attribute(URI_TEMPLATE_ATTRIBUTE).orElse(null));
                                observationContext.setRequest(request);
-                               Mono<ClientResponse> responseMono = filterFunction.apply(exchangeFunction)
-                                               .exchange(request)
+                               ExchangeFilterFunction effectiveFilterFunction = filterFunction;
+                               Mono<ClientResponse> responseMono = Mono.defer(
+                                               () -> effectiveFilterFunction.apply(exchangeFunction).exchange(request))
                                                .checkpoint("Request to " +
                                                                WebClientUtils.getRequestDescription(request.method(), request.url()) +
                                                                " [DefaultWebClient]")

As to what @pratapjai143 explains now, I think it's unrelated to spring-framework. Side note: I'm not sure how that works in Brave but I'd assume you have to enable that feature in service C, not necessarily in B. But I'm no expert here. Nevertheless, it looks like a very tailored approach to a specific case and can lead to confusion and further issues.

I believe there's plenty of ways to achieve that debugging capability that you expect, e.g. printing the parentSpanId alongside the spanId, or using baggage items and including them in the logs, or just not using tracing for correlation as that's sometimes an orthogonal concern, yet we (as engineers) tend to abuse tracing for that purpose. If the Spring Boot configuration for Brave does not achieve preserving the incoming spanId then that's material for further investigation - if you can provide a minimal reproducer you can file an issue for Boot I suppose.

[bclozel]

Thanks Dariusz!
I've opened #33559 to fix this limitation in Spring Framework.

I'm closing this issue as there's nothing actionable for our codebase and I find it highly confusing.