feat(aws-lambda)!: Remove explicit x-ray context in favor of global propagator (open-telemetry#2369)

Co-authored-by: Tyler Benson <[email protected]>
Co-authored-by: Amir Blum <[email protected]>

Author: 3 people

package-lock.json

@@ -54,8 +54,7 @@ In your Lambda function configuration, add or update the `NODE_OPTIONS` environm
 | --- | --- | --- |
 | `requestHook` | `RequestHook` (function) | Hook for adding custom attributes before lambda starts handling the request. Receives params: `span, { event, context }` |
 | `responseHook` | `ResponseHook` (function) | Hook for adding custom attributes before lambda returns the response. Receives params: `span, { err?, res? }` |
-| `disableAwsContextPropagation` | `boolean` | By default, this instrumentation will try to read the context from the `_X_AMZN_TRACE_ID` environment variable set by Lambda, set this to `true` or set the environment variable `OTEL_LAMBDA_DISABLE_AWS_CONTEXT_PROPAGATION=true` to disable this behavior |
-| `eventContextExtractor` | `EventContextExtractor` (function) | Function for providing custom context extractor in order to support different event types that are handled by AWS Lambda (e.g., SQS, CloudWatch, Kinesis, API Gateway). Applied only when `disableAwsContextPropagation` is set to `true`. Receives params: `event, context` |
+| `eventContextExtractor` | `EventContextExtractor` (function) | Function for providing custom context extractor in order to support different event types that are handled by AWS Lambda (e.g., SQS, CloudWatch, Kinesis, API Gateway). |
 | `lambdaHandler` | `string` | By default, this instrumentation automatically determines the Lambda handler function to instrument. This option is used to override that behavior by explicitly specifying the Lambda handler to instrument. See [Specifying the Lambda Handler](#specifying-the-lambda-handler) for additional information. |
 
 ### Hooks Usage Example
@@ -86,6 +85,62 @@ The `lambdaHandler` should be specified as a string in the format `<file>.<handl
 
 One way to determine if the `lambdaHandler` option should be used is to check the handler defined on your Lambda. This can be done by determining the value of the `_HANDLER` environment variable or by viewing the **Runtime Settings** of your Lambda in AWS Console. If the handler is what you expect, then the instrumentation should work without the `lambdaHandler` option. If the handler points to something else, then the `lambdaHandler` option should be used to explicitly specify the handler that should be instrumented.
 
+### Context Propagation
+
+AWS Active Tracing can provide a parent context for the span generated by this instrumentation. Note that the span generated by Active Tracing is always reported only to AWS X-Ray. Therefore, if the OpenTelemetry SDK is configured to export traces to a backend other than AWS X-Ray, this will result in a broken trace.
+
+If you use version `<=0.46.0` of this package, then the Active Tracing context is used as the parent context by default if present. In this case, in order to prevent broken traces, set the `disableAwsContextPropagation` option to `false`.
+Additional propagators can be added in the TracerProvider configuration.
+
+If you use version `>0.46.0`, the Active Tracing context is no longer used by default. In order to enable it, include the [AWSXRayLambdaPropagator](https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/propagators/propagator-aws-xray-lambda) propagator in the list of propagators provided to the TracerProvider via its configuration, or by including `xray-lambda` in the OTEL_PROPAGATORS environment variable (see the example below on using the env variable).
+
+Note that there are two AWS-related propagators: [AWSXRayPropagator](https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/propagators/propagator-aws-xray) and [AWSXRayLambdaPropagator](https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/propagators/propagator-aws-xray-lambda). Here is a guideline for when to use one or the other:
+
+- If you export traces to AWS X-Ray, then use the `AWSXRayLambdaPropagator` or the `xray-lambda` value in the OTEL_PROPAGATORS environment variable. This will handle the active tracing lambda context as well as X-Ray HTTP headers.
+- If you export traces to a backend other than AWS X-Ray, then use the `AWSXrayPropagator` or `xray` in the environment variable. This propagator only handles the X-Ray HTTP headers.
+
+Examples:
+
+1. Active Tracing is enabled and the OpenTelemetry SDK is configured to export traces to AWS X-Ray. In this case, configure the SDK to use the `AWSXRayLambdaPropagator`.
+
+```js
+const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
+const { AWSXRayLambdaPropagator } = require('@opentelemetry/propagator-aws-xray-lambda');
+
+const provider = new NodeTracerProvider();
+provider.register({
+  propagator: new AWSXRayLambdaPropagator()
+});
+```
+
+Alternatively, use the `getPropagators()` function from the [auto-configuration-propagators](https://github.com/open-telemetry/opentelemetry-js-contrib/blob/main/metapackages/auto-configuration-propagators/README.md) package, and set the OTEL_PROPAGATORS environment variable to `xray-lambda`.
+
+```js
+const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
+const { getPropagator } = require('@opentelemetry/auto-configuration-propagators');
+
+const provider = new NodeTracerProvider();
+provider.register({
+  propagator: getPropagator()
+});
+```
+
+1. The OpenTelemetry SDK is configured to export traces to a backend other than AWX X-Ray, but the lambda function is invoked by other AWS services which send the context using the X-Ray HTTP headers. In this case, include the `AWSXRayPropagator`, which extracts context from the HTTP header but not the Lambda Active Tracing context.
+
+```js
+const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
+const { AWSXRayLambdaPropagator } = require('@opentelemetry/propagator-aws-xray-lambda');
+
+const provider = new NodeTracerProvider();
+provider.register({
+  propagator: new AWSXRayPropagator()
+});
+```
+
+Alternatively, use the `auto-configuration-package` as in example #1 and set the OTEL_PROPAGATORS environment variable to `xray`.
+
+For additional information, see the [documentation for lambda semantic conventions](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/faas/aws-lambda.md#aws-x-ray-active-tracing-considerations).
+
 ## Semantic Conventions
 
 This package uses `@opentelemetry/semantic-conventions` version `1.22+`, which implements Semantic Convention [Version 1.7.0](https://github.com/open-telemetry/opentelemetry-specification/blob/v1.7.0/semantic_conventions/README.md)

plugins/node/opentelemetry-instrumentation-aws-lambda/README.md

@@ -45,6 +45,8 @@
   "devDependencies": {
     "@opentelemetry/api": "^1.3.0",
     "@opentelemetry/core": "^1.8.0",
+    "@opentelemetry/propagator-aws-xray": "^1.25.1",
+    "@opentelemetry/propagator-aws-xray-lambda": "^0.52.1",
     "@opentelemetry/sdk-metrics": "^1.8.0",
     "@opentelemetry/sdk-trace-base": "^1.8.0",
     "@opentelemetry/sdk-trace-node": "^1.8.0",
@@ -56,7 +58,6 @@
   },
   "dependencies": {
     "@opentelemetry/instrumentation": "^0.54.0",
-    "@opentelemetry/propagator-aws-xray": "^1.3.1",
     "@opentelemetry/semantic-conventions": "^1.27.0",
     "@types/aws-lambda": "8.10.143"
   },

plugins/node/opentelemetry-instrumentation-aws-lambda/package.json

@@ -35,14 +35,9 @@ import {
   SpanKind,
   SpanStatusCode,
   TextMapGetter,
-  TraceFlags,
   TracerProvider,
   ROOT_CONTEXT,
 } from '@opentelemetry/api';
-import {
-  AWSXRAY_TRACE_ID_HEADER,
-  AWSXRayPropagator,
-} from '@opentelemetry/propagator-aws-xray';
 import {
   SEMATTRS_FAAS_EXECUTION,
   SEMRESATTRS_CLOUD_ACCOUNT_ID,
@@ -60,10 +55,8 @@ import {
 import { AwsLambdaInstrumentationConfig, EventContextExtractor } from './types';
 /** @knipignore */
 import { PACKAGE_NAME, PACKAGE_VERSION } from './version';
-import { env } from 'process';
 import { LambdaModule } from './internal-types';
 
-const awsPropagator = new AWSXRayPropagator();
 const headerGetter: TextMapGetter<APIGatewayProxyEventHeaders> = {
   keys(carrier): string[] {
     return Object.keys(carrier);
@@ -73,26 +66,13 @@ const headerGetter: TextMapGetter<APIGatewayProxyEventHeaders> = {
   },
 };
 
-export const traceContextEnvironmentKey = '_X_AMZN_TRACE_ID';
 export const lambdaMaxInitInMilliseconds = 10_000;
 
 export class AwsLambdaInstrumentation extends InstrumentationBase<AwsLambdaInstrumentationConfig> {
   private _traceForceFlusher?: () => Promise<void>;
   private _metricForceFlusher?: () => Promise<void>;
 
   constructor(config: AwsLambdaInstrumentationConfig = {}) {
-    if (config.disableAwsContextPropagation == null) {
-      if (
-        typeof env['OTEL_LAMBDA_DISABLE_AWS_CONTEXT_PROPAGATION'] ===
-          'string' &&
-        env[
-          'OTEL_LAMBDA_DISABLE_AWS_CONTEXT_PROPAGATION'
-        ].toLocaleLowerCase() === 'true'
-      ) {
-        config = { ...config, disableAwsContextPropagation: true };
-      }
-    }
-
     super(PACKAGE_NAME, PACKAGE_VERSION, config);
   }
 
@@ -230,7 +210,6 @@ export class AwsLambdaInstrumentation extends InstrumentationBase<AwsLambdaInstr
       const parent = AwsLambdaInstrumentation._determineParent(
         event,
         context,
-        config.disableAwsContextPropagation === true,
         config.eventContextExtractor ||
           AwsLambdaInstrumentation._defaultEventContextExtractor
       );
@@ -433,32 +412,8 @@ export class AwsLambdaInstrumentation extends InstrumentationBase<AwsLambdaInstr
   private static _determineParent(
     event: any,
     context: Context,
-    disableAwsContextPropagation: boolean,
     eventContextExtractor: EventContextExtractor
   ): OtelContext {
-    let parent: OtelContext | undefined = undefined;
-    if (!disableAwsContextPropagation) {
-      const lambdaTraceHeader = process.env[traceContextEnvironmentKey];
-      if (lambdaTraceHeader) {
-        parent = awsPropagator.extract(
-          otelContext.active(),
-          { [AWSXRAY_TRACE_ID_HEADER]: lambdaTraceHeader },
-          headerGetter
-        );
-      }
-      if (parent) {
-        const spanContext = trace.getSpan(parent)?.spanContext();
-        if (
-          spanContext &&
-          (spanContext.traceFlags & TraceFlags.SAMPLED) === TraceFlags.SAMPLED
-        ) {
-          // Trace header provided by Lambda only sampled if a sampled context was propagated from
-          // an upstream cloud service such as S3, or the user is using X-Ray. In these cases, we
-          // need to use it as the parent.
-          return parent;
-        }
-      }
-    }
     const extractedContext = safeExecuteInTheMiddle(
       () => eventContextExtractor(event, context),
       e => {
@@ -473,10 +428,6 @@ export class AwsLambdaInstrumentation extends InstrumentationBase<AwsLambdaInstr
     if (trace.getSpan(extractedContext)?.spanContext()) {
       return extractedContext;
     }
-    if (!parent) {
-      // No context in Lambda environment or HTTP headers.
-      return ROOT_CONTEXT;
-    }
-    return parent;
+    return ROOT_CONTEXT;
   }
 }

plugins/node/opentelemetry-instrumentation-aws-lambda/src/instrumentation.ts

@@ -38,7 +38,6 @@ export type EventContextExtractor = (
 export interface AwsLambdaInstrumentationConfig extends InstrumentationConfig {
   requestHook?: RequestHook;
   responseHook?: ResponseHook;
-  disableAwsContextPropagation?: boolean;
   eventContextExtractor?: EventContextExtractor;
   lambdaHandler?: string;
   lambdaStartTime?: number;