Skip to content

ADOT Lambda Layer ignores OTEL_EXPORTER_OTLP_ENDPOINT in favor of OTEL_EXPORTER_OTLP_TRACES_ENDPOINT #297

@garysassano

Description

@garysassano

The AWSOpenTelemetryDistroJs Lambda layer ignores the generic OTEL_EXPORTER_OTLP_ENDPOINT environment variable and defaults to UDP export (X-Ray). Only the signal-specific OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is recognized as a custom endpoint.

This behavior deviates from the OpenTelemetry OTLP Exporter specification, which states that OTEL_EXPORTER_OTLP_ENDPOINT should be used as a fallback when signal-specific endpoints are not set.

Environment

  • AWS Region: eu-central-1
  • Lambda Runtime: Node.js 22.x (ARM64)
  • ADOT Layer: arn:aws:lambda:eu-central-1:615299751070:layer:AWSOpenTelemetryDistroJs:10
  • Target: Custom ADOT Collector behind ALB

Configuration That Does NOT Work

environment: {
  AWS_LAMBDA_EXEC_WRAPPER: "/opt/otel-instrument",
  OTEL_SERVICE_NAME: "my-lambda",
  OTEL_TRACES_EXPORTER: "otlp",
  // ❌ Generic endpoint - IGNORED in Lambda environment
  OTEL_EXPORTER_OTLP_ENDPOINT: "http://my-collector:4318",
  OTEL_EXPORTER_OTLP_PROTOCOL: "http/protobuf",
}

Result: Traces are sent via UDP to X-Ray instead of the specified OTLP endpoint.

Configuration That WORKS

environment: {
  AWS_LAMBDA_EXEC_WRAPPER: "/opt/otel-instrument",
  OTEL_SERVICE_NAME: "my-lambda",
  OTEL_TRACES_EXPORTER: "otlp",
  // ✅ Signal-specific endpoint - recognized as custom endpoint
  OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: "http://my-collector:4318/v1/traces",
  OTEL_EXPORTER_OTLP_PROTOCOL: "http/protobuf",
}

Result: Traces are successfully sent to the custom OTLP endpoint.

Root Cause

In aws-opentelemetry-configurator.ts, the hasCustomOtlpTraceEndpoint() function only checks for OTEL_EXPORTER_OTLP_TRACES_ENDPOINT:

function hasCustomOtlpTraceEndpoint() {
  return process.env['OTEL_EXPORTER_OTLP_TRACES_ENDPOINT'] !== undefined;
}

This check is used in configureOtlp() to decide whether to use UDP export in Lambda:

static configureOtlp(): SpanExporter {
  // ...
  if (!hasCustomOtlpTraceEndpoint() && isLambdaEnvironment()) {
    protocol = 'udp';
  }
  // ...
}

When users set OTEL_EXPORTER_OTLP_ENDPOINT (the generic endpoint), the code doesn't recognize it as a custom endpoint and defaults to UDP export.

Workaround

Use OTEL_EXPORTER_OTLP_TRACES_ENDPOINT instead of OTEL_EXPORTER_OTLP_ENDPOINT. Note that when using the signal-specific endpoint, you must include the full path (/v1/traces).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions