Add distributed tracing and structured error handling for extensions #6321
+746
−43
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JeffreyCA
requested review from
hemarina,
rajeshkamal5050,
tg-msft,
vhvb1989,
wbreza and
weikanglim
as code owners
vhvb1989
approved these changes
Dec 5, 2025
2 tasks
rajeshkamal5050
approved these changes
Dec 5, 2025
wbreza
reviewed
Dec 8, 2025
Contributor
wbreza
left a comment
wbreza
left a comment
There was a problem hiding this comment.
I worry that adding these tracing fields directly to all our messages will cause long term maintenance issues and require many updates and each new service will need to need to include the same values.
We should consider leveraging gRPC metadata that are passing along via HTTP2 headers similar to the approach we are taking with authorization.
Authorization is sent transparently to/from extensions via this method and so far has been easy to maintain/update.
weikanglim
reviewed
Dec 8, 2025
cli/azd/grpc/proto/event.proto
Outdated
Comment on lines
29
to
31
| // W3C traceparent format for distributed tracing propagation. | ||
| // Format: "00-{traceId}-{spanId}-{flags}" | ||
| string trace_parent = 98; |
Contributor
There was a problem hiding this comment.
I'm not liking the embedding of trace_parent as a message field, echoing what @wbreza mentioned earlier. This would make a lot of sense as a grpc header.
Contributor
Author
There was a problem hiding this comment.
@weikanglim @wbreza Can you take another look? I pushed some updates to use metadata, although it isn't strictly needed to fix #6292/#6316.
Setting the TRACEPARENT env var and updating the extension to hydrate the context from it is the actual fix.
Contributor
There was a problem hiding this comment.
I wouldn't mind removing the grpc request-based tracing entirely in favor of ONLY TRACEPARENT process invocation tracing.
weikanglim
approved these changes
Dec 8, 2025
JeffreyCA
force-pushed
the
jeffreyca/6292/ext-improve-tracing
branch
from
56fbab3 to
eea682a
Compare
Collaborator
Azure Dev CLI Install InstructionsInstall scriptsMacOS/Linux
bash: pwsh: WindowsPowerShell install MSI install Standalone Binary
MSI
Documentationlearn.microsoft.com documentationtitle: Azure Developer CLI reference
|
weikanglim
approved these changes
Dec 9, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #6292
This PR enhances the extension framework by adding support for distributed tracing propagation and improving error handling telemetry. It ensures that trace contexts are preserved across the gRPC boundary between
azdand extensions, and that service-specific errors from extensions are correctly captured in telemetry.A follow-up PR will be made to update the agents extension to use these new capabilities.
Changes
Distributed Tracing
TRACEPARENTandTRACESTATEenvironment variables when launching extension process and addedazdext.NewContext()helper to hydrate extension context fromTRACEPARENT/TRACESTATE.traceparent/tracestateheader injection and extraction using gRPC client and server interceptors.azdext.NewContext()to hydrate the extension's root context fromTRACEPARENT/TRACESTATEenvironment variablese.Structured Error Handling
ExtensionResponseErrorto capture detailed error information includingErrorCode,StatusCode, andServiceName.MapErrorto recognize these structured errors and emit precise telemetry signals (e.g.,ext.service.openai.429instead of generic errors).FrameworkServiceErrorMessageandServiceTargetErrorMessagedefinitions to transport structured error data.Bug Fixes
SendAndWaitandSendAndWaitWithProgressinMessageBrokerto prevent race conditions during concurrent stream writes.Validation
Tracing Comparison:
{"Status": {
"Code": "Error",
"Description": "UnknownError"
},
"Attributes": { ... }
...
}
{"Status": {
"Code": "Error",
"Description": "ext.service.ai.500"
},
"Attributes": {
"error.service.name": "ai",
"error.service.host": "services.ai.azure.com",
"error.service.statusCode": "500",
...
}
...
}