`middleware-recursion-detection` logic is not skipped when "X-Amzn-Trace-Id" header is set in lowercase

[jj22ee]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

X-Ray SDK propagates X-Ray Trace Header like: x-amzn-trace-id

OTel SDK propagates X-Ray Trace Header like: X-Amzn-Trace-Id

The current logic in middleware-recursion-detection intends to inject trace ID to request header to detect recursion invocation in Lambda.

• This logic is skipped when X-Amzn-Trace-Id is set in the request headers.
• This logic is not skipped when x-amzn-trace-id is set in the request headers, but it should, given that header checks should be case-insensitive

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/middleware-recursion-detection@3.734.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v20.10.0

Reproduction Steps

1. Have LambdaX and LambdaY, LambdaX invokes LambdaY via Lambda invoke from @aws-sdk/client-lambda, add permissions to LambdaX to use invoke.
2. Enable Lambda service traces in both LambdaX and LambdaY
3. Enable Application Signals in LambdaX both LambdaX and LambdaY
4. Invoke LambdaX function.

LambdaX Sample app: https://github.com/aws-observability/aws-otel-js-instrumentation/tree/main/lambda-layer/sample-apps/aws-sdk
(Replace s3.listbuckets call with lambda.invoke(LambdaY) call)

LambdaY Sample app: https://github.com/aws-observability/aws-otel-js-instrumentation/tree/main/lambda-layer/sample-apps/aws-sdk

Observed Behavior

middleware-recursion-detection inject X-Ray Trace Header X-Amzn-Trace-Id, this overwrites the x-amzn-trace-id header set by OpenTelemetry sometime later since all headers are made case-insensitive/lowercase at some point later.

Expected Behavior

middleware-recursion-detection shouldn't inject X-Ray Trace Header X-Amzn-Trace-Id when x-amzn-trace-id header is set by OpenTelemetry.

Possible Solution

Check for both X-Amzn-Trace-Id and x-amzn-trace-id headers in middleware-recursion-detection.

OR

Have all header logic checks in middleware-recursion-detection to be case-insensitive.

Additional Information/Context

No response

[kuhe]

notes to implementer:

I wish we didn't implement headers as a plain object.
Of course I am tempted to make headers transparently case-insensitive with a getter/setter or Proxy, but I believe that will be over-engineered and potentially introduce more bugs.

Let's just change this line in middleware-recursion-detection/index.ts

request.headers.hasOwnProperty(TRACE_ID_HEADER_NAME)

to

const traceIdHeader = Object.keys(request.headers ?? {})
  .find(h => h.toLowerCase() === TRACE_ID_HEADER_NAME.toLowerCase())
  ?? TRACE_ID_HEADER_NAME;

if (... request.headers.hasOwnProperty(traceIdHeader)) {
  ...

[kuhe]

@jj22ee sorry, I missed your existing PR when we were discussing how to fix this. It would have worked great for this case, and we do accept external (AWS or otherwise) contributions in general, thank you.

The associated fix is expected to be released in a few hours.

update: Available in https://github.com/aws/aws-sdk-js-v3/releases/tag/v3.772.0

[jj22ee]

Hey no problem, thanks for the quick fix you two!

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.