Fix: Lambda Topology Issue (#1016)

**Issue #, if available:**

Lambda Topology issue -- more context in PRs for Python and JavaScript:
-
aws-observability/aws-otel-python-instrumentation#319
-
aws-observability/aws-otel-js-instrumentation#149

**Description of changes:**

- Apply fix for the Lambda Topology issue. The logic mimics the fix in
our other ADOT SDKs.
- Adding back AWS Resource support for Lambda.
-
#907
  - We previously removed support due to the Lambda Topology issue

**Test plan:**

Set up two Lambda functions with Java runtimes and tested with custom
Lambda layer with fix built-in. Tested both AWS SDK v1 and v2. Below are
screenshots of the topology for various configurations.

**v1 Topology (lambdaA & lambdaB instrumented)**
<img width="1311" alt="Screenshot 2025-02-07 at 11 48 51 AM"
src="https://github.com/user-attachments/assets/48234604-ae4b-49cd-926f-05cdd74038a7"
/>

**v2 Topology (lambdaA & lambdaB instrumented)**
<img width="1222" alt="Screenshot 2025-02-07 at 11 26 34 AM"
src="https://github.com/user-attachments/assets/cf7446f3-888f-4756-8ce0-e5ed1e97c9b5"
/>

We observe the following correct behaviors for topology above:
- Service entity node for `Invoke` call to downstream lambda.
- AWS Resource node for `GetFunction` call to downstream lambda.
- AWS Resource node for `ListBuckets` call to downstream s3.

**v1 Topology (lambdaB not instrumented)**
<img width="965" alt="Screenshot 2025-02-07 at 12 17 59 PM"
src="https://github.com/user-attachments/assets/67c361c0-4b8b-4d54-b1dd-0f21a9eee6ff"
/>

**v2 Topology (lambdaB not instrumented)**
<img width="965" alt="Screenshot 2025-02-07 at 12 17 59 PM"
src="https://github.com/user-attachments/assets/67c361c0-4b8b-4d54-b1dd-0f21a9eee6ff"
/>

We observe the following correct behaviors for topology above:
- Downstream lambda called with `Invoke` is correctly treated as
RemoteService entity when not instrumented

Additionally, I generated the spans locally to verify the lambda
instrumentation patch behaves correctly.

**v1 Invoke**
<img width="1281" alt="Screenshot 2025-02-06 at 10 10 23 PM"
src="https://github.com/user-attachments/assets/8d025453-4658-47c7-8c50-261be8b665f5"
/>

**v2 Invoke**
<img width="1281" alt="Screenshot 2025-02-06 at 10 11 49 PM"
src="https://github.com/user-attachments/assets/46b382d0-9475-4871-9773-ed78e609d4a2"
/>

**v1 GetFunction**
<img width="1281" alt="Screenshot 2025-02-06 at 10 08 53 PM"
src="https://github.com/user-attachments/assets/a59e2de6-d50c-47bf-b9d6-171c1ce7cc02"
/>

**v2 GetFunction**
<img width="1281" alt="Screenshot 2025-02-06 at 10 10 05 PM"
src="https://github.com/user-attachments/assets/bec349e0-92c7-4d80-b778-8fa29f3b1ab2"
/>

We observe the following correct behaviors in the spans above:
- For `Invoke` calls, we see `aws.remote.service` and
`aws.remote.environment` correctly populated in the spans.
- For non-`Invoke` calls (i.e. `GetFunction`), we see AWS Resource
attributes such as `aws.remote.resource.identifier` and
`aws.cloudformation.primary.identifier` correctly populated.


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

Author: yiyuan-he

awsagentprovider/src/main/java/software/amazon/opentelemetry/javaagent/providers/AwsAttributeKeys.java

@@ -32,6 +32,9 @@ private AwsAttributeKeys() {}
   static final AttributeKey<String> AWS_REMOTE_SERVICE =
       AttributeKey.stringKey("aws.remote.service");
 
+  static final AttributeKey<String> AWS_REMOTE_ENVIRONMENT =
+      AttributeKey.stringKey("aws.remote.environment");
+
   static final AttributeKey<String> AWS_REMOTE_OPERATION =
       AttributeKey.stringKey("aws.remote.operation");
 
@@ -64,6 +67,9 @@ private AwsAttributeKeys() {}
   static final AttributeKey<String> AWS_SECRET_ARN =
       AttributeKey.stringKey("aws.secretsmanager.secret.arn");
 
+  static final AttributeKey<String> AWS_LAMBDA_NAME =
+      AttributeKey.stringKey("aws.lambda.function.name");
+
   static final AttributeKey<String> AWS_LAMBDA_ARN =
       AttributeKey.stringKey("aws.lambda.function.arn");
 

awsagentprovider/src/main/java/software/amazon/opentelemetry/javaagent/providers/AwsMetricAttributeGenerator.java

@@ -52,12 +52,15 @@
 import static software.amazon.opentelemetry.javaagent.providers.AwsAttributeKeys.AWS_GUARDRAIL_ARN;
 import static software.amazon.opentelemetry.javaagent.providers.AwsAttributeKeys.AWS_GUARDRAIL_ID;
 import static software.amazon.opentelemetry.javaagent.providers.AwsAttributeKeys.AWS_KNOWLEDGE_BASE_ID;
+import static software.amazon.opentelemetry.javaagent.providers.AwsAttributeKeys.AWS_LAMBDA_ARN;
+import static software.amazon.opentelemetry.javaagent.providers.AwsAttributeKeys.AWS_LAMBDA_NAME;
 import static software.amazon.opentelemetry.javaagent.providers.AwsAttributeKeys.AWS_LAMBDA_RESOURCE_ID;
 import static software.amazon.opentelemetry.javaagent.providers.AwsAttributeKeys.AWS_LOCAL_OPERATION;
 import static software.amazon.opentelemetry.javaagent.providers.AwsAttributeKeys.AWS_LOCAL_SERVICE;
 import static software.amazon.opentelemetry.javaagent.providers.AwsAttributeKeys.AWS_QUEUE_NAME;
 import static software.amazon.opentelemetry.javaagent.providers.AwsAttributeKeys.AWS_QUEUE_URL;
 import static software.amazon.opentelemetry.javaagent.providers.AwsAttributeKeys.AWS_REMOTE_DB_USER;
+import static software.amazon.opentelemetry.javaagent.providers.AwsAttributeKeys.AWS_REMOTE_ENVIRONMENT;
 import static software.amazon.opentelemetry.javaagent.providers.AwsAttributeKeys.AWS_REMOTE_OPERATION;
 import static software.amazon.opentelemetry.javaagent.providers.AwsAttributeKeys.AWS_REMOTE_RESOURCE_IDENTIFIER;
 import static software.amazon.opentelemetry.javaagent.providers.AwsAttributeKeys.AWS_REMOTE_RESOURCE_TYPE;
@@ -518,6 +521,37 @@ private static void setRemoteResourceTypeAndIdentifier(SpanData span, Attributes
                 Optional.ofNullable(escapeDelimiters(span.getAttributes().get(AWS_SECRET_ARN))));
         cloudformationPrimaryIdentifier =
             Optional.ofNullable(escapeDelimiters(span.getAttributes().get(AWS_SECRET_ARN)));
+      } else if (isKeyPresent(span, AWS_LAMBDA_NAME)) {
+        // Handling downstream Lambda as a service vs. an AWS resource:
+        // - If the method call is "Invoke", we treat downstream Lambda as a service.
+        // - Otherwise, we treat it as an AWS resource.
+        //
+        // This addresses a Lambda topology issue in Application Signals.
+        // More context in PR:
+        // https://github.com/aws-observability/aws-otel-python-instrumentation/pull/319
+        //
+        // NOTE: The environment variables LAMBDA_APPLICATION_SIGNALS_REMOTE_ENVIRONMENT was
+        // introduced as part of this fix.
+        // It is optional and allows users to override the default value if needed.
+        if ("Invoke".equals(getRemoteOperation(span, RPC_METHOD))) {
+          Optional<String> remoteService =
+              getLambdaFunctionNameFromArn(
+                  Optional.ofNullable(escapeDelimiters(span.getAttributes().get(AWS_LAMBDA_NAME))));
+          builder.put(AWS_REMOTE_SERVICE, remoteService.get());
+
+          String remoteEnvironment =
+              Optional.ofNullable(System.getenv("LAMBDA_APPLICATION_SIGNALS_REMOTE_ENVIRONMENT"))
+                  .filter(s -> !s.isEmpty())
+                  .orElse("default");
+          builder.put(AWS_REMOTE_ENVIRONMENT, "lambda:" + remoteEnvironment);
+        } else {
+          remoteResourceType = Optional.of(NORMALIZED_LAMBDA_SERVICE_NAME + "::Function");
+          remoteResourceIdentifier =
+              getLambdaFunctionNameFromArn(
+                  Optional.ofNullable(escapeDelimiters(span.getAttributes().get(AWS_LAMBDA_NAME))));
+          cloudformationPrimaryIdentifier =
+              Optional.ofNullable(escapeDelimiters(span.getAttributes().get(AWS_LAMBDA_ARN)));
+        }
       } else if (isKeyPresent(span, AWS_LAMBDA_RESOURCE_ID)) {
         remoteResourceType = Optional.of(NORMALIZED_LAMBDA_SERVICE_NAME + "::EventSourceMapping");
         remoteResourceIdentifier =
@@ -539,6 +573,14 @@ private static void setRemoteResourceTypeAndIdentifier(SpanData span, Attributes
     }
   }
 
+  private static Optional<String> getLambdaFunctionNameFromArn(Optional<String> stringArn) {
+    if (stringArn.isPresent() && stringArn.get().startsWith("arn:aws:lambda:")) {
+      Arn resourceArn = Arn.fromString(stringArn.get());
+      return Optional.of(resourceArn.getResource().toString().split(":")[1]);
+    }
+    return stringArn;
+  }
+
   private static Optional<String> getSecretsManagerResourceNameFromArn(Optional<String> stringArn) {
     Arn resourceArn = Arn.fromString(stringArn.get());
     return Optional.of(resourceArn.getResource().toString().split(":")[1]);

awsagentprovider/src/test/java/software/amazon/opentelemetry/javaagent/providers/AwsMetricAttributeGeneratorTest.java

@@ -26,6 +26,7 @@
 import static software.amazon.opentelemetry.javaagent.providers.AwsAttributeKeys.AWS_DATA_SOURCE_ID;
 import static software.amazon.opentelemetry.javaagent.providers.AwsAttributeKeys.AWS_GUARDRAIL_ID;
 import static software.amazon.opentelemetry.javaagent.providers.AwsAttributeKeys.AWS_KNOWLEDGE_BASE_ID;
+import static software.amazon.opentelemetry.javaagent.providers.AwsAttributeKeys.AWS_LAMBDA_NAME;
 import static software.amazon.opentelemetry.javaagent.providers.AwsAttributeKeys.AWS_LAMBDA_RESOURCE_ID;
 import static software.amazon.opentelemetry.javaagent.providers.AwsAttributeKeys.AWS_LOCAL_OPERATION;
 import static software.amazon.opentelemetry.javaagent.providers.AwsAttributeKeys.AWS_LOCAL_SERVICE;
@@ -874,6 +875,11 @@ public void testSdkClientSpanWithRemoteResourceAttributes() {
     validateRemoteResourceAttributes("AWS::SecretsManager::Secret", "secretName");
     mockAttribute(AWS_SECRET_ARN, null);
 
+    // Validate behaviour of AWS_LAMBDA_NAME, then remove it.
+    mockAttribute(AWS_LAMBDA_NAME, "testLambdaName");
+    validateRemoteResourceAttributes("AWS::Lambda::Function", "testLambdaName");
+    mockAttribute(AWS_LAMBDA_NAME, null);
+
     // Validate behaviour of AWS_LAMBDA_RESOURCE_ID
     mockAttribute(AWS_LAMBDA_RESOURCE_ID, "eventSourceId");
     validateRemoteResourceAttributes("AWS::Lambda::EventSourceMapping", "eventSourceId");