[Messaging] Context propagation not fully working for messaging (with new conventions)

[cbos]

Is your feature request related to a problem? Please describe.

As described in https://github.com/open-telemetry/semantic-conventions/blob/main/docs/messaging/messaging-spans.md#context-propagation context propagation is supported for spans with messaging.

For example JMS supports sending properties as part of the message, with that traceparent, tracestate and baggage headers are send with the message.
The receiving service can take these properties and reconstruct the context.

With the current/old semantic conventions that is working fine.

But when this feature is enabled it is not working:
otel.instrumentation.messaging.experimental.receive-telemetry.enabled

With this feature spans are linked in the same way as with the new semantic conventions.
See #13192

The headers are still sent, but only span is linked, but the baggage is not propagated.

Describe the solution you'd like

A possible solution is to copy the baggage header on creation of the new context:

Code for this solution can look like this:
ing-bank@91ae29b

Describe alternatives you've considered

No response

Additional context

No response

[trask]

@cbos I think this is a question for the semantic conventions repository? I can transfer it there if you agree

EDIT oh I just saw your #13192 (comment)

[trask]

@cbos I'm not sure the answer to this question, so I've added it to Monday's Semantic Convention SIG meeting to see if I can get an answer. If you're available, it might be helpful to join and get your thoughts there (https://github.com/open-telemetry/community#specification-sigs).

[cbos]

@trask
Due to a private appointment I am not able to join coming Monday, otherwise I would have joined to discuss this topic.
If you want more details or screenshots as examples of the situations I can provide that. Either on this ticket or via CNCF Slack, you can ping me there as well.

[trask]

hi @cbos, we discussed this topic a bit, you can find recording at https://zoom.us/rec/share/51r49fyS-mcCeNE5P-o2ktiDEtf2HLqEEeZkOrU9Pzpw2WROPURjnN4XHSgGAi9P.k6j7WqdCFNZOyEC1 [discussion starting at 12 minutes 30 seconds into the recording]

let me know if we understood this situation correctly and if our comments about capturing parent span in addition to the span link make sense / address this, thanks!

[cbos]

@trask
Thanks for bringing it on the table in the meeting and providing the update.

I am a bit confused now.

Based on this talk giving by a member of messaging SIG I got the impression that all messaging would go the 'link' route:
https://www.youtube.com/watch?v=GvF1ivr7RlA "Disintegrated Telemetry: The Pains of Monitoring Asynchronous Workflows"

That was also my impression based on the semantic convertion documentation:

https://github.com/open-telemetry/semantic-conventions/blob/main/docs/messaging/messaging-spans.md#consumer-spans

These conventions use spans links as the default mechanism to correlate producers and consumer(s) because:
It is the only consistent trace structure that can be guaranteed, given the many different messaging systems models available.

But somehow, I missed this part:

Message creation context as parent of "Process" span
Exclusively for single messages scenarios, the "Process" span MAY use the message's creation context as its parent, thus achieving a direct parent-child relationship between producer and consumer(s). Instrumentations SHOULD document whether they use the message creation context as a parent for "Process" spans and MAY provide configuration options allowing users to control this behavior.

That makes it more complicated. Because from instrumentation perspective you don't know upfront if it will be batch processing or single message processing.

But in all cases 'Baggage' is still valuable to be carried on IMHO.

According to this documentation https://opentelemetry.io/docs/concepts/signals/baggage/ baggage is 'alongside' context.
Based on this wording it looks like it is not part of the context, but next to it.

I can imaging that context (like trace id and parent span etc) is only linked in quite some cases, but baggage does not only serve a technical goal, but also a functional goal to be able to enrich the whole flow with certain data.

Baggage is best used to include information typically available only at the start of a request further downstream. This can include things like Account Identification, User IDs, Product IDs, and origin IPs, for example.

Propagating this information using baggage allows for deeper analysis of telemetry in a backend. For example, if you include information like a User ID on a span that tracks a database call, you can much more easily answer questions like “which users are experiencing the slowest database calls?” You can also log information about a downstream operation and include that same User ID in the log data.

This is also relevant to propagate this to 'linked' spans as still handles the original request, but technically implemented with different techniques.

So I think the propagation of baggage is very useful. I can image that it should be controlled with a setting/feature toggle if a specific implementation should use it or not, controlled on deployment.

I can image that this first needs to be agreed on at semantic convention level.

[trask]

are you thinking to merge baggage entries from multiple messages somehow in the batch processing case?

[cbos]

are you thinking to merge baggage entries from multiple messages somehow in the batch processing case?

Good question. I am not sure.
The baggage documentation is described from the perspective of a single request / single message:
https://opentelemetry.io/docs/concepts/signals/baggage/

I think also on conceptual level it needs an agreement how to deal with it. But for single message and batch processing.