Skip to content

Fix: Lambda Topology Issue #319

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
yiyuan-he merged 3 commits into from
Feb 4, 2025
Merged

Fix: Lambda Topology Issue #319

yiyuan-he merged 3 commits into from
Feb 4, 2025

Conversation

@yiyuan-he
Copy link
Contributor

@yiyuan-he yiyuan-he commented Jan 30, 2025 •
edited
Loading

Description of changes:

These changes fix the broken lambda topology in AppSignals console. Before these changes, if lambdaA calls lambdaB, we will have two different nodes to represent the downstream lambda.

image

Test plan:

Configuration:

  • lambdaA (AppSignals enabled)
  • lambdaB (AppSignals enabled)

image

Configuration:

  • lambdaA (AppSignals enabled)
  • lambdaB (AppSignals enabled)
  • We call both Invoke and GetFunction from lambdaA to verify that Invoke and non-Invoke calls are handled differently.
    • Invoke calls to downstream lambda will be modeled as Service entity type.
    • Non-Invoke (GetFunction in this case) calls to downstream lambda will be modeled as AWS Resource.

image

Configuration:

  • lambdaA (AppSignals enabled)
  • lambdaB (AppSignals enabled)
  • We modify lambdaB to make a downstream s3 call (ListBuckets). We see that the edges are propagated correctly and the entire topology is modeled correctly.

image

Configuration:

  • lambdaA (AppSignals enabled)
  • lambdaB (AppSignals disabled)
  • We see that lambdaB is correctly modeled as a RemoteService entity type when it does not have AppSignals enabled.

image

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@yiyuan-he yiyuan-he requested a review from a team as a code owner January 30, 2025 00:47
@yiyuan-he yiyuan-he changed the title lambda -> lambda issue fix [WIP] lambda topology issue fix Feb 3, 2025
@yiyuan-he yiyuan-he changed the title lambda topology issue fix Fix: Lambda Topology Issue Feb 3, 2025
@yiyuan-he yiyuan-he force-pushed the lambda-fix branch 3 times, most recently from 781f3b1 to 018351a Compare February 3, 2025 23:57
@yiyuan-he yiyuan-he merged commit 07aad1f into aws-observability:main Feb 4, 2025
12 checks passed
@yiyuan-he yiyuan-he mentioned this pull request Feb 5, 2025
Merged
yiyuan-he added a commit to aws-observability/aws-otel-js-instrumentation that referenced this pull request Feb 5, 2025
@yiyuan-he
Fix: Lambda Topology Issue (#149)
da8291e 
**Issue #, if available:**
Lambda Topology Issue Context:
aws-observability/aws-otel-python-instrumentation#319

**Description of changes:**
- Apply fix for the Lambda Topology issue. The logic mimics the fix in
our ADOT Python SDK.
- Adding back logic to generate `aws.lambda.function.name` and
`aws.lambda.function.arn` span attributes for Lambda. Previously, we
removed these changes in the AWS Resources expansion due to the Lambda
Topology issue.
  - More context about adding support for Lambda as AWS Resource: 
-
aws-observability/aws-otel-python-instrumentation#265

**Test plan:**

The same cases in the test plan of this
[PR](aws-observability/aws-otel-python-instrumentation#319)
were validated by building a custom JavaScript lambda layer. Below is a
screenshot of all the test cases in a single topology.

We can observe the following:
- Service entity node for `Invoke` call to downstream lambda.
- AWS Resource node for `GetFunction` call to downstream lambda.
- AWS Resource node for `ListBuckets` call to downstream s3.
- Another AWS Resource node for `GetBucketLocation` call to downstream
s3.

<img width="1359" alt="Screenshot 2025-02-05 at 10 15 26 AM"
src="https://github.com/user-attachments/assets/14d38cad-32c7-4e2c-a987-c424c7fb7296"
/>

Here we see downstream lambda called with `Invoke` is correctly treated
as RemoteService entity when not instrumented:

![image](https://github.com/user-attachments/assets/7f1cfcec-8827-4dc7-beaf-2a1b9b98e4f4)

Additionally, I generated the spans locally to validate the new lambda
instrumentation patch behaves correctly.

**Generated span for `Invoke` call**

We see the new `aws.remote.environment` attribute is present in span so
the downstream lambda will be treated as a Service type in Application
Signals backend.

<img width="1124" alt="invoke"
src="https://github.com/user-attachments/assets/81cefb44-c951-4ad0-8aeb-9a2064d7d0ea"
/>


**Generated span for `GetFunction` call**

We see `aws.remote.resource.identifier` and
`aws.cloudformation.primary.identifier` are set so the downstream lambda
will be treated as an AWS resource type in Application Signals backend.

<img width="1124" alt="get-function"
src="https://github.com/user-attachments/assets/abfd700b-7966-4d93-bb47-868c31be9a34"
/>


By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
This was referenced Feb 7, 2025
Merged
Merged
yiyuan-he added a commit to aws-observability/aws-otel-java-instrumentation that referenced this pull request Feb 11, 2025
@yiyuan-he
Fix: Lambda Topology Issue (#1016)
cd3c2c5 
**Issue #, if available:**

Lambda Topology issue -- more context in PRs for Python and JavaScript:
-
aws-observability/aws-otel-python-instrumentation#319
-
aws-observability/aws-otel-js-instrumentation#149

**Description of changes:**

- Apply fix for the Lambda Topology issue. The logic mimics the fix in
our other ADOT SDKs.
- Adding back AWS Resource support for Lambda.
-
#907
  - We previously removed support due to the Lambda Topology issue

**Test plan:**

Set up two Lambda functions with Java runtimes and tested with custom
Lambda layer with fix built-in. Tested both AWS SDK v1 and v2. Below are
screenshots of the topology for various configurations.

**v1 Topology (lambdaA & lambdaB instrumented)**
<img width="1311" alt="Screenshot 2025-02-07 at 11 48 51 AM"
src="https://github.com/user-attachments/assets/48234604-ae4b-49cd-926f-05cdd74038a7"
/>

**v2 Topology (lambdaA & lambdaB instrumented)**
<img width="1222" alt="Screenshot 2025-02-07 at 11 26 34 AM"
src="https://github.com/user-attachments/assets/cf7446f3-888f-4756-8ce0-e5ed1e97c9b5"
/>

We observe the following correct behaviors for topology above:
- Service entity node for `Invoke` call to downstream lambda.
- AWS Resource node for `GetFunction` call to downstream lambda.
- AWS Resource node for `ListBuckets` call to downstream s3.

**v1 Topology (lambdaB not instrumented)**
<img width="965" alt="Screenshot 2025-02-07 at 12 17 59 PM"
src="https://github.com/user-attachments/assets/67c361c0-4b8b-4d54-b1dd-0f21a9eee6ff"
/>

**v2 Topology (lambdaB not instrumented)**
<img width="965" alt="Screenshot 2025-02-07 at 12 17 59 PM"
src="https://github.com/user-attachments/assets/67c361c0-4b8b-4d54-b1dd-0f21a9eee6ff"
/>

We observe the following correct behaviors for topology above:
- Downstream lambda called with `Invoke` is correctly treated as
RemoteService entity when not instrumented

Additionally, I generated the spans locally to verify the lambda
instrumentation patch behaves correctly.

**v1 Invoke**
<img width="1281" alt="Screenshot 2025-02-06 at 10 10 23 PM"
src="https://github.com/user-attachments/assets/8d025453-4658-47c7-8c50-261be8b665f5"
/>

**v2 Invoke**
<img width="1281" alt="Screenshot 2025-02-06 at 10 11 49 PM"
src="https://github.com/user-attachments/assets/46b382d0-9475-4871-9773-ed78e609d4a2"
/>

**v1 GetFunction**
<img width="1281" alt="Screenshot 2025-02-06 at 10 08 53 PM"
src="https://github.com/user-attachments/assets/a59e2de6-d50c-47bf-b9d6-171c1ce7cc02"
/>

**v2 GetFunction**
<img width="1281" alt="Screenshot 2025-02-06 at 10 10 05 PM"
src="https://github.com/user-attachments/assets/bec349e0-92c7-4d80-b778-8fa29f3b1ab2"
/>

We observe the following correct behaviors in the spans above:
- For `Invoke` calls, we see `aws.remote.service` and
`aws.remote.environment` correctly populated in the spans.
- For non-`Invoke` calls (i.e. `GetFunction`), we see AWS Resource
attributes such as `aws.remote.resource.identifier` and
`aws.cloudformation.primary.identifier` correctly populated.


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
@yiyuan-he yiyuan-he deleted the lambda-fix branch March 3, 2025 18:39
yiyuan-he added a commit to yiyuan-he/aws-otel-java-instrumentation that referenced this pull request Mar 5, 2025
@yiyuan-he
Fix: Lambda Topology Issue (aws-observability#1016)
224134d 
**Issue #, if available:**

Lambda Topology issue -- more context in PRs for Python and JavaScript:
-
aws-observability/aws-otel-python-instrumentation#319
-
aws-observability/aws-otel-js-instrumentation#149

**Description of changes:**

- Apply fix for the Lambda Topology issue. The logic mimics the fix in
our other ADOT SDKs.
- Adding back AWS Resource support for Lambda.
-
aws-observability#907
  - We previously removed support due to the Lambda Topology issue

**Test plan:**

Set up two Lambda functions with Java runtimes and tested with custom
Lambda layer with fix built-in. Tested both AWS SDK v1 and v2. Below are
screenshots of the topology for various configurations.

**v1 Topology (lambdaA & lambdaB instrumented)**
<img width="1311" alt="Screenshot 2025-02-07 at 11 48 51 AM"
src="https://github.com/user-attachments/assets/48234604-ae4b-49cd-926f-05cdd74038a7"
/>

**v2 Topology (lambdaA & lambdaB instrumented)**
<img width="1222" alt="Screenshot 2025-02-07 at 11 26 34 AM"
src="https://github.com/user-attachments/assets/cf7446f3-888f-4756-8ce0-e5ed1e97c9b5"
/>

We observe the following correct behaviors for topology above:
- Service entity node for `Invoke` call to downstream lambda.
- AWS Resource node for `GetFunction` call to downstream lambda.
- AWS Resource node for `ListBuckets` call to downstream s3.

**v1 Topology (lambdaB not instrumented)**
<img width="965" alt="Screenshot 2025-02-07 at 12 17 59 PM"
src="https://github.com/user-attachments/assets/67c361c0-4b8b-4d54-b1dd-0f21a9eee6ff"
/>

**v2 Topology (lambdaB not instrumented)**
<img width="965" alt="Screenshot 2025-02-07 at 12 17 59 PM"
src="https://github.com/user-attachments/assets/67c361c0-4b8b-4d54-b1dd-0f21a9eee6ff"
/>

We observe the following correct behaviors for topology above:
- Downstream lambda called with `Invoke` is correctly treated as
RemoteService entity when not instrumented

Additionally, I generated the spans locally to verify the lambda
instrumentation patch behaves correctly.

**v1 Invoke**
<img width="1281" alt="Screenshot 2025-02-06 at 10 10 23 PM"
src="https://github.com/user-attachments/assets/8d025453-4658-47c7-8c50-261be8b665f5"
/>

**v2 Invoke**
<img width="1281" alt="Screenshot 2025-02-06 at 10 11 49 PM"
src="https://github.com/user-attachments/assets/46b382d0-9475-4871-9773-ed78e609d4a2"
/>

**v1 GetFunction**
<img width="1281" alt="Screenshot 2025-02-06 at 10 08 53 PM"
src="https://github.com/user-attachments/assets/a59e2de6-d50c-47bf-b9d6-171c1ce7cc02"
/>

**v2 GetFunction**
<img width="1281" alt="Screenshot 2025-02-06 at 10 10 05 PM"
src="https://github.com/user-attachments/assets/bec349e0-92c7-4d80-b778-8fa29f3b1ab2"
/>

We observe the following correct behaviors in the spans above:
- For `Invoke` calls, we see `aws.remote.service` and
`aws.remote.environment` correctly populated in the spans.
- For non-`Invoke` calls (i.e. `GetFunction`), we see AWS Resource
attributes such as `aws.remote.resource.identifier` and
`aws.cloudformation.primary.identifier` correctly populated.


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
yiyuan-he added a commit to yiyuan-he/aws-otel-java-instrumentation that referenced this pull request May 28, 2025
@yiyuan-he
Fix: Lambda Topology Issue (aws-observability#1016)
0db4b31 
**Issue #, if available:**

Lambda Topology issue -- more context in PRs for Python and JavaScript:
-
aws-observability/aws-otel-python-instrumentation#319
-
aws-observability/aws-otel-js-instrumentation#149

**Description of changes:**

- Apply fix for the Lambda Topology issue. The logic mimics the fix in
our other ADOT SDKs.
- Adding back AWS Resource support for Lambda.
-
aws-observability#907
  - We previously removed support due to the Lambda Topology issue

**Test plan:**

Set up two Lambda functions with Java runtimes and tested with custom
Lambda layer with fix built-in. Tested both AWS SDK v1 and v2. Below are
screenshots of the topology for various configurations.

**v1 Topology (lambdaA & lambdaB instrumented)**
<img width="1311" alt="Screenshot 2025-02-07 at 11 48 51 AM"
src="https://github.com/user-attachments/assets/48234604-ae4b-49cd-926f-05cdd74038a7"
/>

**v2 Topology (lambdaA & lambdaB instrumented)**
<img width="1222" alt="Screenshot 2025-02-07 at 11 26 34 AM"
src="https://github.com/user-attachments/assets/cf7446f3-888f-4756-8ce0-e5ed1e97c9b5"
/>

We observe the following correct behaviors for topology above:
- Service entity node for `Invoke` call to downstream lambda.
- AWS Resource node for `GetFunction` call to downstream lambda.
- AWS Resource node for `ListBuckets` call to downstream s3.

**v1 Topology (lambdaB not instrumented)**
<img width="965" alt="Screenshot 2025-02-07 at 12 17 59 PM"
src="https://github.com/user-attachments/assets/67c361c0-4b8b-4d54-b1dd-0f21a9eee6ff"
/>

**v2 Topology (lambdaB not instrumented)**
<img width="965" alt="Screenshot 2025-02-07 at 12 17 59 PM"
src="https://github.com/user-attachments/assets/67c361c0-4b8b-4d54-b1dd-0f21a9eee6ff"
/>

We observe the following correct behaviors for topology above:
- Downstream lambda called with `Invoke` is correctly treated as
RemoteService entity when not instrumented

Additionally, I generated the spans locally to verify the lambda
instrumentation patch behaves correctly.

**v1 Invoke**
<img width="1281" alt="Screenshot 2025-02-06 at 10 10 23 PM"
src="https://github.com/user-attachments/assets/8d025453-4658-47c7-8c50-261be8b665f5"
/>

**v2 Invoke**
<img width="1281" alt="Screenshot 2025-02-06 at 10 11 49 PM"
src="https://github.com/user-attachments/assets/46b382d0-9475-4871-9773-ed78e609d4a2"
/>

**v1 GetFunction**
<img width="1281" alt="Screenshot 2025-02-06 at 10 08 53 PM"
src="https://github.com/user-attachments/assets/a59e2de6-d50c-47bf-b9d6-171c1ce7cc02"
/>

**v2 GetFunction**
<img width="1281" alt="Screenshot 2025-02-06 at 10 10 05 PM"
src="https://github.com/user-attachments/assets/bec349e0-92c7-4d80-b778-8fa29f3b1ab2"
/>

We observe the following correct behaviors in the spans above:
- For `Invoke` calls, we see `aws.remote.service` and
`aws.remote.environment` correctly populated in the spans.
- For non-`Invoke` calls (i.e. `GetFunction`), we see AWS Resource
attributes such as `aws.remote.resource.identifier` and
`aws.cloudformation.primary.identifier` correctly populated.


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wangzlei wangzlei wangzlei approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@yiyuan-he @wangzlei