OTEL traceId not propagated across kafka

[kirankolli-addepar]

I have a micros service architecture of quarkus applications some of which are communicating over kafka.
Quarkus version = 2.16.5Final
What I observe is that the traceId is not getting propagated over kafka.
All applications have the following dependencies added :
implementation 'io.quarkus:quarkus-opentelemetry-exporter-otlp' implementation("io.opentelemetry:opentelemetry-extension-trace-propagators")

and the below properties set :

%dev.quarkus.opentelemetry.enabled=true %dev.quarkus.opentelemetry.tracer.exporter.otlp.endpoint=http://... %dev.quarkus.log.console.format=%d{HH:mm:ss} %-5p traceId=%X{traceId}, parentId=%X{parentId}, spanId=%X{spanId} [%c{2.}] (%t) %s%e%n

I find that the communication over REST is being traced but the tracing is lost when the message goes over kafka.
Is this something that is not supported or do I need to addition configuration ?

Note : Kafka communication is being done via smallrye annotations :
@Channel MutinyEmitter<?> emitter and being sent via MutinyEmitter.send()
and received via @Incoming (org.eclipse.microprofile.reactive.messaging.Incoming) annotations

[quarkus-bot[bot]]

/cc @alesj (kafka), @brunobat (opentelemetry,tracing), @cescoffier (kafka), @ozangunalp (kafka), @radcortez (opentelemetry,tracing)

[kirankolli-addepar]

To clarify I think that the trace ids are not propagated when message publishing to kafka is done on a different thread. Is there any way around it ?

[cescoffier]

What do you mean 'from a different thread'? Are you creating your own thread?

The trace ids are stored in the duplicated context. So, if that context is propagated along your processing, it should be fine.

[kirankolli-addepar]

Question also posted here : https://stackoverflow.com/questions/76654066/quarkus-open-telemetry-distributed-tracing-does-not-carry-over-traceid-across

[ozangunalp]

I think @Ladicek's answer is correct in the stackoverflow question. The propagation would not work in this case.
You'd either need the context-propagation or propagate the Otel context manually.
Manual propagation would look something like:

// get current OpenTelemetry context
var oTelCtx = QuarkusContextStorage.INSTANCE.current();
...
// then using the emitter you'd construct the message with your own TracingMetadata
emitter.emit(KafkaRecord.of(...).addMetadata(TracingMetadata.withCurrent(oTelCtx)));

[radcortez]

Correct. This is because the thread pool is being managed manually. If you delegate everything to Quarkus, it should work automatically.

[fcauti]

Thanks @ozangunalp.
To be noticed that addMetadata method does not modify the current message object, so need to publish the new returned message, like you did in your example, or reassign it in case you have already in a variable

KafkaRecord record = KafkaRecord.of(...);
record = KafkaRecord.from(record.addMetadata(TracingMetadata.withCurrent(Context.current())));

Context.current() seems to be equivalent to QuarkusContextStorage.INSTANCE.current()

[ozangunalp]

Yes, Quarkus sets the context storage provider so when you do Context.current() it is equivalent to QuarkusContextStorage.INSTANCE.current()