Kafka streams & Distributed tracing / opentelemetry

[EKrol2]

Is there a recommended way to implement distributed tracing (opentelemetry) in kafka streams for quarkus?

There are a very nice quarkus guides for kafka streams and for opentelemetry:
https://quarkus.io/guides/kafka-streams
https://quarkus.io/guides/opentelemetry

For combining them however I haven't been able to find proper documentation.

As I understand it, the underlying issue is that the incoming traceID is not propagated to the statestores in kafkastreams.

I have found a couple approaches that might work:

1. Javaagent injection: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/README.md
2. Building it myself: https://iwtyo.today/today-i-learned-opentelemetry-with-smallrye-reactive-messaging-and-kafkastream
3. Brave: https://github.com/openzipkin/brave/blob/master/instrumentation/kafka-streams/RATIONALE.md
   (Although for the last one my team would need to rewrite our normal quarkus kafka services aswell, which do work mostly out of the box with opentelemetry)

Is there a recommended approach, or perhaps something on the roadmap for combining kafka streams and opentelemetry?

[quarkus-bot[bot]]

/cc @alesj (kafka,kafka-streams), @brunobat (opentelemetry,tracing), @cescoffier (kafka), @gunnarmorling (kafka-streams), @ozangunalp (kafka,kafka-streams), @radcortez (opentelemetry,tracing), @rquinio (kafka-streams)

[brunobat]

Give it a try. quarkus-smallrye-reactive-messaging-kafka is already instrumented with OpenTelemetry. If you also include thequarkus-opentelemetry extension if should work out of the box.
Traces are automatically sent to the OTel Collector GRPC address.

[rquinio]

I didn't find low-level Kafka clients instrumentation in smallrye-reactive-messaging-kafka that could be re-used in KafkaStreams, it seems to be done at smallrye- reactive-messaging higher level ?

[EKrol2]

In the following list for Apache Kafka streams API, the standalone library instrumentation is listed as N/A: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md

[ozangunalp]

@rquinio Indeed reactive messaging kafka instrumentation is higher level and I don't think that it can be reused for kafka streams.

[rquinio]

I think 2. would be the way to go:

• a KafkaStreamClientSupplier with KafkaConsumer / KafkaProducer decorators. At the time of OpenTracing, it was suggested in Opentracing with Kafka Streams #5471 but was never implemented.
• or maybe even directly via config as described in https://opentelemetry.io/blog/2022/instrument-kafka-clients/#using-the-interceptors ?
  • kafka-streams.consumer.interceptor.classes=io.opentelemetry.instrumentation.kafkaclients.v2_6.TracingConsumerInterceptor and
  • kafka-streams.producer.interceptor.classes=io.opentelemetry.instrumentation.kafkaclients.v2_6.TracingProducerInterceptor

However in the Kafka consumer instrumentations I've seen, the receiver span is always closed on creation...
This means there's no active span while inside the kstream topology, so you can't attach functional metadata from business logic.
I wonder if the instrumentation could keep the receiver span opened, track uncommitted records and only close the span when the corresponding offset is committed ?

About 1. Javaagent injection: I see warnings about the Java agent in Quarkus otel doc, and it won't work with GraalVM.
About 3. Brave SDK: I guess it would be incompatible with quarkus-opentelemetry, which works with OTel APIs.

[EKrol2]

We'll give it a good try, and give an update in some time. Thanks