Skip to content

Fix: Lambda Topology Issue (#1016) #1085

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
yiyuan-he merged 14 commits into from
Jun 3, 2025
Merged

Fix: Lambda Topology Issue (#1016) #1085

yiyuan-he merged 14 commits into from
Jun 3, 2025

Conversation

@yiyuan-he
Copy link
Contributor

@yiyuan-he yiyuan-he commented May 28, 2025 •
edited
Loading

What does this pull request do?

Cherry picks the Lambda Topology fix from our Lambda Layer v1 branch.

Original PR: #1016

Test strategy

Unit tests and followed e2e test strategy used in the original PR to sanity check correct behavior in service topology.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@yiyuan-he
Fix: Lambda Topology Issue (aws-observability#1016)
0db4b31 
**Issue #, if available:**

Lambda Topology issue -- more context in PRs for Python and JavaScript:
-
aws-observability/aws-otel-python-instrumentation#319
-
aws-observability/aws-otel-js-instrumentation#149

**Description of changes:**

- Apply fix for the Lambda Topology issue. The logic mimics the fix in
our other ADOT SDKs.
- Adding back AWS Resource support for Lambda.
-
aws-observability#907
  - We previously removed support due to the Lambda Topology issue

**Test plan:**

Set up two Lambda functions with Java runtimes and tested with custom
Lambda layer with fix built-in. Tested both AWS SDK v1 and v2. Below are
screenshots of the topology for various configurations.

**v1 Topology (lambdaA & lambdaB instrumented)**
<img width="1311" alt="Screenshot 2025-02-07 at 11 48 51 AM"
src="https://github.com/user-attachments/assets/48234604-ae4b-49cd-926f-05cdd74038a7"
/>

**v2 Topology (lambdaA & lambdaB instrumented)**
<img width="1222" alt="Screenshot 2025-02-07 at 11 26 34 AM"
src="https://github.com/user-attachments/assets/cf7446f3-888f-4756-8ce0-e5ed1e97c9b5"
/>

We observe the following correct behaviors for topology above:
- Service entity node for `Invoke` call to downstream lambda.
- AWS Resource node for `GetFunction` call to downstream lambda.
- AWS Resource node for `ListBuckets` call to downstream s3.

**v1 Topology (lambdaB not instrumented)**
<img width="965" alt="Screenshot 2025-02-07 at 12 17 59 PM"
src="https://github.com/user-attachments/assets/67c361c0-4b8b-4d54-b1dd-0f21a9eee6ff"
/>

**v2 Topology (lambdaB not instrumented)**
<img width="965" alt="Screenshot 2025-02-07 at 12 17 59 PM"
src="https://github.com/user-attachments/assets/67c361c0-4b8b-4d54-b1dd-0f21a9eee6ff"
/>

We observe the following correct behaviors for topology above:
- Downstream lambda called with `Invoke` is correctly treated as
RemoteService entity when not instrumented

Additionally, I generated the spans locally to verify the lambda
instrumentation patch behaves correctly.

**v1 Invoke**
<img width="1281" alt="Screenshot 2025-02-06 at 10 10 23 PM"
src="https://github.com/user-attachments/assets/8d025453-4658-47c7-8c50-261be8b665f5"
/>

**v2 Invoke**
<img width="1281" alt="Screenshot 2025-02-06 at 10 11 49 PM"
src="https://github.com/user-attachments/assets/46b382d0-9475-4871-9773-ed78e609d4a2"
/>

**v1 GetFunction**
<img width="1281" alt="Screenshot 2025-02-06 at 10 08 53 PM"
src="https://github.com/user-attachments/assets/a59e2de6-d50c-47bf-b9d6-171c1ce7cc02"
/>

**v2 GetFunction**
<img width="1281" alt="Screenshot 2025-02-06 at 10 10 05 PM"
src="https://github.com/user-attachments/assets/bec349e0-92c7-4d80-b778-8fa29f3b1ab2"
/>

We observe the following correct behaviors in the spans above:
- For `Invoke` calls, we see `aws.remote.service` and
`aws.remote.environment` correctly populated in the spans.
- For non-`Invoke` calls (i.e. `GetFunction`), we see AWS Resource
attributes such as `aws.remote.resource.identifier` and
`aws.cloudformation.primary.identifier` correctly populated.


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
@yiyuan-he yiyuan-he requested a review from a team as a code owner May 28, 2025 15:30
@yiyuan-he yiyuan-he force-pushed the cherry-pick-lambda branch from 987eb35 to d4ac453 Compare May 30, 2025 02:50
thpierce
thpierce reviewed May 30, 2025
View reviewed changes
@codecov-commenter
Copy link

codecov-commenter commented May 30, 2025

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 91.17647% with 3 lines in your changes missing coverage. Please review.

Project coverage is 71.57%. Comparing base (09e6487) to head (13026de).
Report is 422 commits behind head on main.

Files with missing lines Patch % Lines
...vaagent/providers/AwsMetricAttributeGenerator.java 90.00% 1 Missing and 2 partials ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files 
@@              Coverage Diff              @@
##               main    #1085       +/-   ##
=============================================
- Coverage     85.71%   71.57%   -14.15%     
- Complexity       19      376      +357     
=============================================
  Files             3       33       +30     
  Lines            49     1393     +1344     
  Branches          5      191      +186     
=============================================
+ Hits             42      997      +955     
- Misses            3      334      +331     
- Partials          4       62       +58

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

thpierce
thpierce reviewed Jun 2, 2025
View reviewed changes
thpierce
thpierce previously approved these changes Jun 3, 2025
View reviewed changes
@yiyuan-he yiyuan-he merged commit 2dad73c into aws-observability:main Jun 3, 2025
5 checks passed
@yiyuan-he yiyuan-he deleted the cherry-pick-lambda branch June 3, 2025 23:54
This was referenced Jun 4, 2025
Closed
Merged
liustve added a commit that referenced this pull request Jun 5, 2025
@liustve @jj22ee
@thpierce @yiyuan-he @srprash @AsakerMohd
Release/v2.11.1 (#1094)
b435fb1 
*Description of changes:*

Merges changes from mainline to v2.11.1

Namely:
#1085
and
#1089

```
The following dependencies are using the latest release version:
 - com.sparkjava:spark-core:2.9.4
 - com.squareup.okhttp3:okhttp:4.12.0
 - io.opentelemetry:opentelemetry-extension-aws:1.20.1

The following dependencies have later release versions:
 - com.amazonaws:aws-java-sdk-bom [1.12.599 -> 1.12.785]
     https://aws.amazon.com/sdkforjava
 - com.fasterxml.jackson:jackson-bom [2.16.0 -> 2.19.0]
     https://github.com/FasterXML/jackson-bom
 - com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin [0.50.0 -> 0.52.0]
 - com.google.guava:guava-bom [33.0.0-jre -> 33.4.8-jre]
     https://github.com/google/guava
 - com.google.protobuf:protobuf-bom [3.25.1 -> 4.31.1]
     https://developers.google.com/protocol-buffers/
 - com.linecorp.armeria:armeria-bom [1.26.4 -> 1.32.5]
     https://armeria.dev/
 - commons-logging:commons-logging [1.2 -> 1.3.5]
     https://commons.apache.org/proper/commons-logging/
 - io.grpc:grpc-bom [1.59.1 -> 1.73.0]
     https://github.com/grpc/grpc-java
 - io.opentelemetry.contrib:opentelemetry-aws-resources [1.39.0-alpha -> 1.46.0-alpha]
     https://github.com/open-telemetry/opentelemetry-java-contrib
 - io.opentelemetry.contrib:opentelemetry-aws-xray [1.39.0-adot1 -> 1.46.0]
     https://github.com/open-telemetry/opentelemetry-java-contrib
 - io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha [2.11.0-adot2-alpha -> 2.16.0-alpha]
     https://github.com/open-telemetry/opentelemetry-java-instrumentation
 - io.opentelemetry.javaagent:opentelemetry-javaagent [2.11.0-adot2 -> 2.16.0]
     https://github.com/open-telemetry/opentelemetry-java-instrumentation
 - io.opentelemetry.proto:opentelemetry-proto [1.0.0-alpha -> 1.7.0-alpha]
     https://github.com/open-telemetry/opentelemetry-proto-java
 - net.bytebuddy:byte-buddy [1.14.10 -> 1.17.5]
     https://bytebuddy.net
 - org.apache.logging.log4j:log4j-bom [2.21.1 -> 2.24.3]
     https://logging.apache.org/log4j/2.x/
 - org.assertj:assertj-core [3.24.2 -> 3.27.3]
     https://assertj.github.io/doc/#assertj-core
 - org.curioswitch.curiostack:protobuf-jackson [2.2.0 -> 2.7.0]
     https://github.com/curioswitch/protobuf-jackson
 - org.junit:junit-bom [5.10.1 -> 5.13.0]
     https://junit.org/junit5/
 - org.slf4j:slf4j-api [1.7.36 -> 2.0.17]
     http://www.slf4j.org
 - org.slf4j:slf4j-simple [1.7.36 -> 2.0.17]
     http://www.slf4j.org
 - org.springframework.boot:spring-boot-dependencies [2.7.17 -> 3.5.0]
     https://spring.io/projects/spring-boot
 - org.testcontainers:testcontainers-bom [1.19.3 -> 1.21.1]
     https://java.testcontainers.org
 - software.amazon.awssdk:bom [2.21.33 -> 2.31.56]
     https://aws.amazon.com/sdkforjava

Gradle release-candidate updates:
 - Gradle: [8.10 -> 8.14.1]


```

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

---------

Co-authored-by: Jonathan Lee <[email protected]>
Co-authored-by: Thomas Pierce <[email protected]>
Co-authored-by: Michael He <[email protected]>
Co-authored-by: ADOT Patch workflow <[email protected]>
Co-authored-by: Prashant Srivastava <[email protected]>
Co-authored-by: Mohamed Asaker <[email protected]>
@lukeina2z lukeina2z mentioned this pull request Jul 8, 2025
Merged
lukeina2z added a commit that referenced this pull request Aug 7, 2025
@lukeina2z
[Lambda Java v2.11.x] Merge All Code Changes from v1.33.x Branch into…
4cd8ed4 
… v2.11.x (#1114)

This change merges all private Lambda Java updates from the v1.33 branch
into the v2.11.x branch. I performed a 'git rebase v2.11' on the v1.33
branch, reviewed all changes, and completed the build and testing
process. The resulting Lambda layer generated trace data identical to
the version built directly from the v2.11.x branch (excluding this PR).

Here is the list of all migrated PRs:

Build layer during CI/CD workflows + some minor refactoring #989 

support java11 runtime for lambda #1001 

Unique artifact names for upload and merge for download #1014 

Bug fixes] Lambda - duplicate lambda spans + appsignals from unsampled
spans #1000

Fix: Lambda Topology Issue (#1016)
 
Fix: Lambda Topology Issue (#1016) #1085
 
feat: Support microservice span in Lambda Java environment. #1053

Test
Tested Java11, 17, and 21 Lambda functions. Manually tested PR-1000 and
PR-1053. Both work as expected in the v2.11 branch. MicroService
(SpringBoot) support works well. I verified attribute
Trace.lambda.multiple server can be found in the Lambda server span,
once we have Servlet instrumentation enabled with
OTEL_INSTRUMENTATION_SERVLET_ENABLED.

Note: The changes in the patch files are not included in this PR. They
should have been reviewed and incorporated as part of this migration:

Upgrade Java Lambda Layer to 2.x #1076 

Lambda with SpringBoot MicroService:
<img width="1367" alt="lambda"
src="https://github.com/user-attachments/assets/5cf5be29-4986-454c-b61b-773d6cde3848"
/>

Service Map and added microservice attribute 'Trace.lambda.multiple
server'.
<img width="1864" alt="traceMap"
src="https://github.com/user-attachments/assets/f7ff1771-61f0-4013-b571-90370a726aa9"
/>

AppSignals
<img width="1875" alt="appSignals"
src="https://github.com/user-attachments/assets/24f1b3a8-851c-4c97-bb50-087ee275b86d"
/>



By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@thpierce thpierce thpierce approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@yiyuan-he @codecov-commenter @thpierce