Opentelemetry baggage propagation fix #1174

romank0 · 2025-10-21T14:41:39Z

What was changed

Fixed OpenTelemetry baggage propagation in the inbound interceptor by explicitly attaching the extracted context before starting spans. Changed from passing context=extracted_ctx as a parameter to using context.attach(extracted_ctx) + context.detach(token).

Why?

The previous implementation used start_as_current_span(context=extracted_ctx) which only uses the provided context to determine the parent span for trace propagation. When building the new span context, OpenTelemetry always uses context.get_current() that is the active context from the stack, not the context= parameter. But the active context is not set from the unpacked values received in headers.

This meant that while trace parent-child relationships worked correctly, baggage values from the extracted context were not copied into the new span context, making them unavailable within Temporal activities/workflows.

By calling context.attach(extracted_ctx) first, we make the extracted context active on the context stack. This ensures that when the new span context is created, it copies all data (including baggage) from the extracted context, properly propagating baggage across service boundaries.

This aligns with the standard pattern used by other OpenTelemetry instrumentations (django, gRPC, etc.).

Checklist

Closes [Feature Request] Make sure OTel baggage propagates properly throughout activities #362
How was this tested:
- Unit/intergration tests in this PR
- Tested in real system basically the same way test_opentelemetry_baggage_propagation_basic test does this, namely:
  1. setting baggage
  2. invoking the workflow that executes an activity
  3. checking the baggage is available in the activity
- Verified trace parent-child relationships still work correctly
Any docs updates needed?
I don't think so.

Change the activity interceptor to use context.attach()/detach() pattern instead of passing context as a parameter to start_as_current_span(). The fix follows the standard OpenTelemetry pattern used by other instrumentations (django, gRPC, etc.) and ensures proper context management with try/finally for detach.

Add additional tests to verify baggage propagation in scenarios: - multiple values - local activity - retries in activity

CLAassistant · 2025-10-21T14:41:47Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ romank0
❌ Roman Konoval

Roman Konoval seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

CLAassistant · 2025-10-21T14:41:47Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

Roman Konoval seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

cretz · 2025-10-21T14:52:55Z

temporalio/contrib/opentelemetry.py

+        extracted_ctx = self.root._context_from_headers(input.headers)
+
+        if extracted_ctx:
+            token = opentelemetry.context.attach(extracted_ctx)


Should we just move this attach/detach into _start_as_current_span to happen always? If we don't want it always, we can add an option on that call. But I think we still want to pass context to OTel's start_as_current_span call right?

After the attach the current context alread has all the information from the headers (including tracing information) so there is no need to pass it to context to start_as_current_span.
But I think it makes sense to modify _start_as_current_span to do attach if the context was provided (that would be the case for inbound scenarios). I'll do that.

I've checked that in official instrumentations they do pass context to start_as_current_span so I implemented both your suggestions.

Two important edge case tests: - exceptions handling - when no current context is available

tconley1428 · 2025-10-21T15:18:38Z

temporalio/contrib/opentelemetry.py

-                        Status(
-                            status_code=StatusCode.ERROR,
-                            description=f"{type(exc).__name__}: {exc}",
+        if context:


I think I would prefer a ternary here, but that's a nit.

tconley1428 · 2025-10-21T15:19:33Z

temporalio/contrib/opentelemetry.py

+                    raise
+        finally:
+            if token:
+                opentelemetry.context.detach(token)


@VegetarianOrc we should evaluate how this intersects with your changes to protect detach elsewhere.

tconley1428 · 2025-10-21T15:22:44Z

tests/contrib/test_opentelemetry.py

        )


+@activity.defn


We usually put the activity/workflow definitions with the test they support if they are only used for one. In this case I would just move them below the one benign exception test.

Actually potentially break them up to go with their respective tests.

tconley1428 · 2025-10-21T15:26:46Z

tests/contrib/test_opentelemetry.py

+        ), "tenant.id baggage should propagate to activity"
+
+
+async def test_opentelemetry_baggage_propagation_multiple_values(


Does this really provide additional coverage compared to _basic? That already reads two baggages.

No, I'll remove it.

tconley1428 · 2025-10-21T15:29:45Z

tests/contrib/test_opentelemetry.py

+async def test_opentelemetry_baggage_propagation_local_activity(
+    client: Client, env: WorkflowEnvironment
+):
+    exporter = InMemorySpanExporter()


I don't think the exporter is needed for many(any?) of these tests since you aren't actually checking anything on outgoing spans. I could be wrong though, does the baggage not happen otherwise?

Indeed exporter is not needed, for baggage to work only tracer with a TracerProvider is required.

tconley1428 · 2025-10-21T15:30:51Z

tests/contrib/test_opentelemetry.py

+        assert all(v == "test-user-retry" for v in retry_attempt_baggage_values)
+
+
+async def test_opentelemetry_baggage_exception_handling(


I'm not sure I understand what this test is trying to validate. It seems to be nothing more than the basic test, but with exceptions to assert truth instead of returning the value. What benefit does it give?

tconley1428 · 2025-10-21T15:32:28Z

.gitignore

 /sdk-python.iml
 /.zed
 *.DS_Store
+tags


Can you elaborate on or remove this?

It is a ctags file used for source code navigation e.g. in vim.

tconley1428

Two other things you'll have to do. First, poe lint is failing. Second, your commits have multiple email addresses associated:

You'll need to add the second email to your github account so that the CLA is signed for all the commits. Either that or modify the commits to the correct email in some way.

Roman Konoval added 4 commits October 21, 2025 15:16

Add tags to gitignore

fc5c674

adds failing baggage propagation test

8343352

adds baggage propagation tests

57a6c42

Add additional tests to verify baggage propagation in scenarios: - multiple values - local activity - retries in activity

romank0 requested a review from a team as a code owner October 21, 2025 14:41

cretz reviewed Oct 21, 2025

View reviewed changes

romank0 force-pushed the baggage-propagation-fix branch from 5905dcf to 3fd9236 Compare October 21, 2025 14:53

Roman Konoval and others added 3 commits October 21, 2025 17:06

adds more tests for baggage propagation

4223936

Two important edge case tests: - exceptions handling - when no current context is available

moves context handling to _start_as_current_span

9bee5ea

Merge branch 'main' into baggage-propagation-fix

411c6e3

romank0 force-pushed the baggage-propagation-fix branch from 3fd9236 to 411c6e3 Compare October 21, 2025 15:16

tconley1428 reviewed Oct 21, 2025

View reviewed changes

tconley1428 requested changes Oct 21, 2025

View reviewed changes

		), "tenant.id baggage should propagate to activity"


		async def test_opentelemetry_baggage_propagation_multiple_values(

		assert all(v == "test-user-retry" for v in retry_attempt_baggage_values)


		async def test_opentelemetry_baggage_exception_handling(

		)


		@activity.defn

Opentelemetry baggage propagation fix #1174

Are you sure you want to change the base?

Opentelemetry baggage propagation fix #1174

Conversation

romank0 commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was changed

Why?

Checklist

Uh oh!

CLAassistant commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CLAassistant commented Oct 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tconley1428 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

romank0 commented Oct 21, 2025 •

edited

Loading

CLAassistant commented Oct 21, 2025 •

edited

Loading