Skip to content

Conversation

@blairhyy-amazon
Copy link
Member

@blairhyy-amazon blairhyy-amazon commented Jul 22, 2025

Description of changes:

  1. Add auto-instrumentation support for the following AWS resources.
  • Populate aws.stream.arn in Span by extracting StreamArn from the request body.
  • Populate aws.table.arn in Span by extracting TableArn from the response body.
  1. Add auto-instrumentation support for remote resource access key.
  • Populate aws.auth.account.access_key and aws.auth.region in Span from STS credentials in client config
  1. Generate cross-account metrics attributes when remote resource identifier is present
  • If remote resource arn is available, extract account id and region from arn.
  • Otherwise, pass account access key id and region from span to metric if available.
  1. Add unit tests, contract tests. Done E2E tests with CW agent.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@blairhyy-amazon blairhyy-amazon requested a review from a team as a code owner July 22, 2025 20:43
@blairhyy-amazon
Copy link
Member Author

Re-implement AWS SDK v3 Cross-Account Support with Recursion Fix

This PR reintroduces the AWS SDK v3 cross-account support (previously reverted in #192) with fixes for the EKS E2E test failures.

Background
The original implementation caused recursive instrumentation when running in EKS with IAM Roles for Service Accounts (IRSA). This occurred because IRSA uses the AWS STS AssumeRoleWithWebIdentity API (ref), which triggers additional instrumented API calls during credential resolution.

Issues Fixed

  1. Recursive instrumentation during credential resolution in IRSA environments
  2. Repetitive middleware stack addition overrides to the same SDK client instance
  3. Unnecessary span generation for internal credential-related calls

Implementation Details

  1. Added context propagation with a credential capture skip flag for cross-account support
  2. Implemented instrumentation suppression for internal credential extraction calls
  3. Introduced __adotMiddlewarePatched guard to prevent duplicate middleware stack additions

Testing

  • Verified E2E test pass in EKS environment with IRSA (ref)
  • Passed contract tests and unit tests

Related

@blairhyy-amazon blairhyy-amazon force-pushed the fix-cross-account branch 2 times, most recently from 63da336 to c964852 Compare July 29, 2025 00:22
jj22ee
jj22ee previously approved these changes Jul 29, 2025
Copy link
Contributor

@jj22ee jj22ee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving, but I'm not 100% sure about the pure reliance on OTel's Context Management to prevent this infinite recursion issue.

We've got a good unit test for this use case but I'm still a bit wary about not having a fallback mitigation for infinite recursion. Will rely on the @aws-observability/aws-application-signals-maintainers approver on whether or not we want to proceed with only this mitigation (please see this comment thread).

…ccount support (aws-observability#192)

**Description of changes:**

1. Add auto-instrumentation support for the following AWS resources.
- Populate `aws.stream.arn` in Span by extracting StreamArn from the
request body.
- Populate `aws.table.arn` in Span by extracting TableArn from the
response body.

2. Add auto-instrumentation support for remote resource access key.
- Populate `aws.auth.account.access_key` and `aws.auth.region` in Span
from STS credentials in client config
 
3. Generate cross-account metrics attributes when remote resource
identifier is present
- If remote resource arn is available, extract account id and region
from arn.
- Otherwise, pass account access key id and region from span to metric
if available.

4. Add unit tests, contract tests. Done E2E tests with CW agent. 

By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
@blairhyy-amazon blairhyy-amazon force-pushed the fix-cross-account branch 3 times, most recently from d19f5d9 to be3f78f Compare July 29, 2025 20:01
Copy link
Contributor

@jj22ee jj22ee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving since last review was stale

@blairhyy-amazon blairhyy-amazon merged commit 03bdef0 into aws-observability:main Jul 30, 2025
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants