How to propagate xray trace context to AWS Lambda for OpenTelemetry?

[DSkoufis]

We're using the io.quarkus:quarkus-amazon-lambda extension to build an AWS Lambda. In the lambda if we enable the Active tracing mode we can see traces sent across to XRay with every request.

What we can't configure is the span linking function for manual instrumentation, i.e @WithSpan. Looks like AWS Lambda sends the tracing header through a system property in Java17: https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html#configuration-envvars-runtime

We're also using a provided.al2 runtime as we've built a native Quarkus app and along with the ADOT layer the property is set, i.e com.amazonaws.xray.traceHeader=Root=1-65ea0523-7ca6bbb929835fc57bd998e4;Parent=d23df4a462562bfe;Sampled=1;Lineage=66fb198a:0.

Is there a way to read that property and pass it down to io.quarkus:quarkus-opentelemetry's @WithSpan and its auto-instrumentation in order to submit the correct span linking context?

[quarkus-bot[bot]]

/cc @brunobat (opentelemetry,tracing), @matejvasek (amazon-lambda), @patriot1burke (amazon-lambda), @radcortez (opentelemetry,tracing)

[brunobat]

If you are using the quarkus-opentelemetry extension you will need to add the XRay context propagation. See: https://quarkus.io/guides/opentelemetry#propagators
Also, you will need to declare the XRay propagator here: quarkus.otel.propagators=xray,baggage
Instead of using the default W3C tracecontext

[DSkoufis]

Thanks @brunobat. I've tried that but still the context is not propagating. I think this is because of the way the AwsXrayPropagator tries to read the context which is through the env variable X-Amzn-Trace-Id: https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/aws-xray-propagator/src/main/java/io/opentelemetry/contrib/awsxray/propagator/AwsXrayPropagator.java#L48

In the AWS docs it looks like that variable is not used in the provided runtimes (and debugging the lambda confirms that):

_X_AMZN_TRACE_ID – The X-Ray tracing header. This environment variable changes with each invocation.

• This environment variable is not defined for OS-only runtimes (the provided runtime family). You can set _X_AMZN_TRACE_ID for custom runtimes using the Lambda-Runtime-Trace-Id response header from the Next invocation.
• For Java runtime versions 17 and later, this environment variable is not used. Instead, Lambda stores tracing information in the com.amazonaws.xray.traceHeader system property.

But while debugging the quarkus native lambda I noticed that we can still get the context through using the com.amazonaws.xray.traceHeader.
The most relevant example I've found is this: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/aws-lambda/aws-lambda-core-1.0/library/src/main/java/io/opentelemetry/instrumentation/awslambdacore/v1_0/internal/AwsXrayEnvSpanLinksExtractor.java#L29

through the io.opentelemetry.instrumentation:opentelemetry-aws-lambda-core-1.0 module.

but I can't find a way to enable it in the quarkus-opentelmetry extension. Is there any customisation we can do in that extension?

[brunobat]

Maybe the opentelemetry-aws-xray-propagator lib needs to be modified.

[DSkoufis]

Fair enough. I thought I was missing something as I'm not too familiar with the whole thing. I'll start a thread in the https://github.com/open-telemetry/opentelemetry-java-contrib repo then.

Many thanks