No context propagation using aws-xray-propagator with AWS Lambdas and provided or java runtimes

[DSkoufis]

Component(s)

aws-xray-propagator

What happened?

Description

I'm trying to create a Quarkus native lambda and enable XRay tracing. We're making use of the quarkus-opentelemetry extension which offers some automatic instrumentation of various Quarkus features, i.e @WithSpan.

We've enabled the Active tracing mode in the Lambda but we are unable to correctly propagate the XRay context to that extension. It was suggested to use the io.opentelemetry.contrib:opentelemetry-aws-xray-propagator but unfortunately this didn't help. And looking at the implementation I think this is because of the AwsXrayPropagator not selecting the correct header:

opentelemetry-java-contrib/aws-xray-propagator/src/main/java/io/opentelemetry/contrib/awsxray/propagator/AwsXrayPropagator.java

Line 48 in 6a44e04

static final String TRACE_HEADER_KEY = "X-Amzn-Trace-Id";

In the AWS docs it looks like that variable is not populated in the provided runtimes (and debugging the lambda confirms that):

_X_AMZN_TRACE_ID – The X-Ray tracing header. This environment variable changes with each invocation.

• This environment variable is not defined for OS-only runtimes (the provided runtime family). You can set _X_AMZN_TRACE_ID for custom runtimes using the Lambda-Runtime-Trace-Id response header from the Next invocation.
• For Java runtime versions 17 and later, this environment variable is not used. Instead, Lambda stores tracing information in the com.amazonaws.xray.traceHeader system property.

Steps to Reproduce

A reproducer is provided here: https://github.com/DSkoufis/quarkus-otel-lambda/tree/main

Expected Result

Trace context to be set correctly using XRay's properties

Actual Result

The propagator is unable to extract the context meaning that a new trace is created:

START RequestId: 09603df8-91a8-4098-8a37-c6d4e15d03fc Version: $LATEST
15:41:04 INFO  traceId=e1cb9f4b723b5f6c0d33dbdc26af86d2, parentId=, spanId=b2671df3374f3863, sampled=true [or.ac.op.la.LambdaService] (Lambda Thread (NORMAL)) Request ID: value1
15:41:04 INFO  traceId=e1cb9f4b723b5f6c0d33dbdc26af86d2, parentId=b2671df3374f3863, spanId=7fcf32d3009c543d, sampled=true [or.ac.op.la.LambdaService] (Lambda Thread (NORMAL)) Here
END RequestId: 09603df8-91a8-4098-8a37-c6d4e15d03fc

Component version

1.32.0-alpha

Log output

No response

Additional context

While debugging the quarkus native lambda I noticed that we can still get the headers through using the com.amazonaws.xray.traceHeader.

The most relevant example I've found is this: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/aws-lambda/aws-lambda-core-1.0/library/src/main/java/io/opentelemetry/instrumentation/awslambdacore/v1_0/internal/AwsXrayEnvSpanLinksExtractor.java#L29

through the io.opentelemetry.instrumentation:opentelemetry-aws-lambda-core-1.0 module.

[wangzlei]

https://github.com/DSkoufis/quarkus-otel-lambda/blob/main/terraform/lambda.tf#L22
It is an ADOT lambda layer specific issue. I will cc ADOT layer layer owner. As a workaround you can build your private Lambda layer from scratch from this repo https://github.com/open-telemetry/opentelemetry-lambda, or using AWS Lambda java 11 Runtime instead.