OpenTelemetry for gRPC endpoints

[LarsSven]

Hi all,

I have an application with a single gRPC endpoint. I want to use OpenTelemetry to trace the execution, and I am using the @WithSpan annotation to create spans within the trace. However, when I test it, I do get all the spans, but they're seen as separate:

However I would like to see them as actual spans within my main trace. Is this a limitation of gRPC, or am I doing something wrong?

I have done no other configuration than installing the opentelemetry extension for Quarkus and adding @WithSpan to a few functions. From the documentation, I believe that is sufficient?

I hope someone can help me with this, thank you.

[quarkus-bot[bot]]

/cc @alesj (grpc), @brunobat (opentelemetry), @cescoffier (grpc), @radcortez (opentelemetry)

[brunobat]

The gRPC server is already instrumented with OpenTelemetry. You only need to add @WithSpan on methods you create and need to track.
Please take a look at: https://github.com/quarkusio/quarkus/blob/main/integration-tests/opentelemetry-grpc/src/main/java/io/quarkus/it/opentelemetry/grpc/HelloService.java

Would you be able to provide a sample project that replicates the issue?

[LarsSven]

Thank you for the repsonse @brunobat,

I created a separate branch at https://gitlab.com/rug-digitallab/themis/judgement/companion-container/-/tree/ls-demo-otel?ref_type=heads. This branch has our current otel implementation added and has the features that make the program difficult to run disabled. You should be able to just run it pretty easily on any operating system through quarkusDev, as long as it has a docker daemon running. So make sure you run the ls-demo-otel branch.

Here is a file that you can run to make a request to the gRPC endpoint.
GrpcClient.kt.md

If you run into any issues let me know and I can try to make a smaller program that demos the issue. I have never worked with opentelemetry before so it may be something I did wrong, but when I view the trace on jaeger, I just see 4 separate spans that don't seem to be connected.

[LarsSven]

(Keep in mind that the GrpcClient should be inside the project, as it uses some of the same dependencies. Also make sure you initialised the git submodules through git submodule init --recursive)

[cescoffier]

You are creating the client directly instead of using a client managed by Quarkus. Thus, the tracing interceptor is not used.

[LarsSven]

Ah okay, I did not know the request had to come from a Quarkus client.

Thank you, I will look into that.

[LarsSven]

So the issue was not related to that. It turns out that it was related to contexts not being automatically propagated through kotlin coroutines, so I need to manually propagate the context.