Spans without parentId when using splunk otel layer with ESM lambdas

[Travispersson]

Hey!

We are using Splunk Otel lambda layers for auto instrumenting our nodejs lambdas.
For CommonJS lambdas, everything seems to be working fine, but for ESM lambdas spans seem to be missing a parent Id, which results in each of these spans (that should be a part of the same trace/request) becoming their own separate trace as visible in APM - Splunk Observabilit Cloud.

Is there some issue with how trace context is being propagated on ESM lambdas? Maybe we can do something manually in code to force a parent span.. but this layer should do these things automatically as it works perfectly fine with CJS lambdas.

We are using this layer:
arn:aws:lambda:eu-west-1:254067382080:layer:splunk-apm-arm:44 (same issue exists on previous versions)
and the following (example) ENV vars:

AWS_LAMBDA_EXEC_WRAPPER: /opt/nodejs-otel-handler
OTEL_LOG_LEVEL: warn
OTEL_RESOURCE_ATTRIBUTES: service.name=service-name,service.version=abcd,deployment.environment=dev
OTEL_SERVICE_NAME: service-name
OTEL_TRACES_EXPORTER: otlp

SPLUNK_ACCESS_TOKEN: ...
SPLUNK_REALM: ...

Everything else should be handled by the layer.

Other related information that can help.

• Lambda is a LambdaUrl. And is invoked like this Client -> Cloudfront -> LambdaUrl
• Active Tracing is disabled
• ...

Maybe related issues?

• Patch aws-lambda instrumentation to support ESM aws-observability/aws-otel-js-instrumentation#101 PR that adds ESM support to AWS own otel layer.
• OpenTelemetry AWS Lambda Layer Not Propagating Parent Span ID in TypeScript Lambdas open-telemetry/opentelemetry-lambda#1742

Let me know if you need any more information, thanks!