-
Notifications
You must be signed in to change notification settings - Fork 17
Description
The AWSOpenTelemetryDistroJs Lambda layer ignores the generic OTEL_EXPORTER_OTLP_ENDPOINT environment variable and defaults to UDP export (X-Ray). Only the signal-specific OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is recognized as a custom endpoint.
This behavior deviates from the OpenTelemetry OTLP Exporter specification, which states that OTEL_EXPORTER_OTLP_ENDPOINT should be used as a fallback when signal-specific endpoints are not set.
Environment
- AWS Region: eu-central-1
- Lambda Runtime: Node.js 22.x (ARM64)
- ADOT Layer:
arn:aws:lambda:eu-central-1:615299751070:layer:AWSOpenTelemetryDistroJs:10 - Target: Custom ADOT Collector behind ALB
Configuration That Does NOT Work
environment: {
AWS_LAMBDA_EXEC_WRAPPER: "/opt/otel-instrument",
OTEL_SERVICE_NAME: "my-lambda",
OTEL_TRACES_EXPORTER: "otlp",
// ❌ Generic endpoint - IGNORED in Lambda environment
OTEL_EXPORTER_OTLP_ENDPOINT: "http://my-collector:4318",
OTEL_EXPORTER_OTLP_PROTOCOL: "http/protobuf",
}Result: Traces are sent via UDP to X-Ray instead of the specified OTLP endpoint.
Configuration That WORKS
environment: {
AWS_LAMBDA_EXEC_WRAPPER: "/opt/otel-instrument",
OTEL_SERVICE_NAME: "my-lambda",
OTEL_TRACES_EXPORTER: "otlp",
// ✅ Signal-specific endpoint - recognized as custom endpoint
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: "http://my-collector:4318/v1/traces",
OTEL_EXPORTER_OTLP_PROTOCOL: "http/protobuf",
}Result: Traces are successfully sent to the custom OTLP endpoint.
Root Cause
In aws-opentelemetry-configurator.ts, the hasCustomOtlpTraceEndpoint() function only checks for OTEL_EXPORTER_OTLP_TRACES_ENDPOINT:
function hasCustomOtlpTraceEndpoint() {
return process.env['OTEL_EXPORTER_OTLP_TRACES_ENDPOINT'] !== undefined;
}This check is used in configureOtlp() to decide whether to use UDP export in Lambda:
static configureOtlp(): SpanExporter {
// ...
if (!hasCustomOtlpTraceEndpoint() && isLambdaEnvironment()) {
protocol = 'udp';
}
// ...
}When users set OTEL_EXPORTER_OTLP_ENDPOINT (the generic endpoint), the code doesn't recognize it as a custom endpoint and defaults to UDP export.
Workaround
Use OTEL_EXPORTER_OTLP_TRACES_ENDPOINT instead of OTEL_EXPORTER_OTLP_ENDPOINT. Note that when using the signal-specific endpoint, you must include the full path (/v1/traces).