RFC: OpenTelemetry Integration for Powertools for AWS Lambda - Tracer only

[leandrodamascena]

Note

DISCLAIMER

This RFC is open for discussion and we're actively seeking community feedback to shape OpenTelemetry support in Powertools for AWS Lambda.
At this stage we have not committed to implementing it and our plans will vary based on your feedback.

The feature set described here is not final and represents our current preferred approach. We encourage all interested parties to provide feedback both in this RFC or via email at [email protected]

We'll use Python in all the code samples throughout the RFC, but the general ideas and design choices apply to all Powertools for AWS languages. Class and method names are subject to change.

RFC: OpenTelemetry Integration for Powertools for AWS Lambda - Tracer only

Summary

The OpenTelemetry integration for Powertools for AWS Lambda provides a standardized, vendor-agnostic approach to observability via distributed tracing. This utility enables seamless instrumentation of AWS Lambda functions, allowing developers to trace requests across distributed systems while maintaining Powertools for AWS Lambda commitment to developer experience.

The Powertools for AWS Lambda OpenTelemetry Tracer implements the familiar interface as the existing Powertools for AWS Lambda X-Ray tracer, allowing easy migrations while opening up integration with a broader ecosystem of observability providers through the OpenTelemetry standard.

Powertools for AWS Lambda OpenTelemetry Tracer supports three primary patterns:

1. Auto-instrumentation with the ADOT Lambda Layer - least effort to implement, using AWS Distro for OpenTelemetry
2. Manual instrumentation with Powertools for AWS Lambda defaults - balanced approach with AWS Lambda-optimized settings
3. Customized instrumentation with direct OpenTelemetry SDK access - most flexible, for advanced observability needs

Use case

Powertools for AWS Lambda customers have been asking for OpenTelemetry support to provide flexibility in their observability strategy. While AWS X-Ray offers robust tracing capabilities, many organizations have standardized on OpenTelemetry, and might use multiple observability providers that support the OpenTelemetry protocol.

The Powertools for AWS Lambda OpenTelemetry Tracer allows developers to:

• Instrument AWS Lambda functions with minimal code changes
• Send telemetry data to any OpenTelemetry-compatible observability provider
• Maintain distributed trace context across service boundaries
• Follow OpenTelemetry semantic conventions for consistent observability data
• Leverage both auto-instrumentation and manual instrumentation approaches

Note

Interested in learning more about OpenTelemetry terms and concepts? Read this gist: https://gist.github.com/leandrodamascena/5693744f7fcf4ccf1994a7ff67a2549d

OTEL Instrumentation using Powertools for AWS Lambda

Powertools for AWS Lambda will support both auto-instrumentation, using the AWS Distro for OpenTelemetry (ADOT) Lambda Layer and manual instrumentation approaches, providing developers with flexibility while maintaining the focus on developer experience.

For auto-instrumentation, Powertools for AWS Lambda will integrate with AWS Distro for OpenTelemetry (ADOT) Lambda Layer, enabling developers to automatically instrument their Lambda functions with minimal configuration. This approach will capture traces from AWS SDK calls, HTTP requests, database operations, and other common interactions without requiring code changes.

For manual instrumentation, Powertools for AWS Lambda will provide opinionated methods that implement best practices for AWS Lambda environments, optimizing performance and configurability, while providing simplified higher level abstractions built on top of the OpenTelemetry SDK. At the same time, we'll expose the underlying OpenTelemetry SDK interfaces for power users who need direct access to the full range of OpenTelemetry capabilities.

This dual approach ensures that developers can choose the right level of observability granularity for their needs, from zero-code instrumentation to highly customized tracing, while benefiting from Powertools' Lambda-optimized defaults and simplified APIs.

Auto-Instrumentation with ADOT Lambda Layer

When using AWS Distro for OpenTelemetry (ADOT) Lambda Layer, developers benefit from a pre-packaged environment that includes the OpenTelemetry Python SDK and a pre-configured collector. This AWS Lambda Extension automatically patches all available libraries in the OpenTelemetry contrib repository, providing comprehensive instrumentation without requiring manual setup.

Powertools for AWS Lambda provides a streamlined interface to leverage this auto-instrumentation capability:

from aws_lambda_powertools.tracing.otel import TracerOpenTelemetry

tracer = TracerOpenTelemetry(mode="auto")

@tracer.capture_method
def example():
    return 1

@tracer.capture_lambda_handler
def lambda_handler(event, context):
    with tracer.add_span("example") as span:
        span.set_attribute("example", example())

With mode="auto", Powertools for AWS Lambda configures itself to work alongside AWS Distro for OpenTelemetry (ADOT) Lambda Layer auto-instrumentation capabilities, detecting its presence and optimizing the integration.

Powertools for AWS Lambda provides opinionated methods like add_span, which wraps OpenTelemetry's start_as_current_span with Lambda-optimized defaults. For instance, add_span automatically configures record_exception to ensure exceptions are properly captured in the trace.

Opinionated Auto-instrumentation

Powertools for AWS Lambda also offers opinionated configurations for common instrumentation scenarios. For example, you can easily exclude specific endpoints from being traced when using requests library:

ENDPOINT = os.getenv("PAYMENT_API", "")
IGNORE_URLS = ["/collect", "/refund"]

tracer = TracerOpenTelemetry(mode="auto")

tracer.instrument_requests(ignore_urls=IGNORE_URLS)

This streamlined interface simplifies the configuration of instrumentation filters in OpenTelemetry, making it straightforward to exclude health checks, monitoring endpoints, or other high-volume, low-value traces that could increase costs.

Powertools for AWS Lambda also enhances specific instrumentations, like the requests library and AWS SDK library, providing additional context and ensuring AWS Lambda-specific attributes are properly recorded. This ensures developers get better quality telemetry data without additional configuration:

from aws_lambda_powertools.tracing.otel import TracerOpenTelemetry
import requests

tracer = TracerOpenTelemetry(mode="auto")

@tracer.capture_lambda_handler
def lambda_handler(event, context):
    # HTTP calls are automatically traced with enhanced Lambda context
    response = requests.get("https://api.example.com/data")
    
    with tracer.add_span("processing") as span:
        # Process the response
        result = process_data(response.json())
        span.set_attribute("result_size", len(result))
        
        return {"statusCode": 200, "body": result}

Manual instrumentation with Powertools for AWS Lambda

While auto-instrumentation provides a quick way to get started with tracing, manual instrumentation offers more control and customization. Powertools for AWS Lambda supports a flexible approach to manual instrumentation, allowing developers to either provide their own pre-configured OpenTelemetry components or rely on Powertools for AWS Lambda recommended defaults.

The following configurations will be supported:

• Custom TracerProvider - Bring your own fully configured TracerProvider with complete control over Resources, Processors, and Exporters
• Custom Exporter configurations - Send telemetry data directly to specific observability providers, while using custom default configurations with Powertools for AWS Lambda.
• Custom Sampling strategies - Implement specialized sampling logic based on business rules, error conditions, or trace attributes
• Custom span processors - Configure batch processing behavior with custom queue sizes, export intervals, and retry logic
• Propagator customization - Support specialized context propagation formats or enhance the default W3C/X-Ray propagators

Bring your own TracerProvider instance

When developers need granular control over their OpenTelemetry configuration, they can create and fully customize their own TracerProvider before passing it to Powertools for AWS Lambda. This approach allows developers to define every aspect of their observability pipeline - from custom resources and sampling strategies to specialized exporters and processors - while still using Powertools' Lambda-optimized instrumentation helpers:

from aws_lambda_powertools.tracing.otel import TracerOpenTelemetry
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

# Create custom resource
resource = Resource.create({
    "service.name": "payment-service",
    "service.version": "1.2.3",
    "deployment.environment": "production"
})

# Create custom provider with resource
provider = TracerProvider(resource=resource)

# Configure custom exporter
exporter = OTLPSpanExporter(
    endpoint="https://my-collector.example.com:4317",
    headers={"api-key": "my-api-key"}
)

# Set up custom processor with specific batch settings
processor = BatchSpanProcessor(
    exporter,
    max_export_batch_size=10,
    schedule_delay_millis=5000,
    max_queue_size=100
)
provider.add_span_processor(processor)

# Use custom TracerProvider with Powertools
tracer = TracerOpenTelemetry(mode="manual", tracer_provider=provider)

@tracer.capture_lambda_handler
def lambda_handler(event, context):
    with tracer.add_span("processing") as span:
        # Your code here
        return {"statusCode": 200}

Creating a default TracerProvider with Powertools for AWS Lambda

For simple use cases, Powertools for AWS will create a default TracerProvider with Lambda-optimized settings.

The default configuration includes:

• TracerProvider with comprehensive AWS Lambda resource attributes (function name, version, runtime, etc.)
• BatchSpanProcessor with parameters optimized for AWS Lambda's execution model (smaller batch sizes, faster export scheduling during shutdown)
• OpenTelemetry protocol (OTLP) Exporter pre-configured to work with AWS Lambda environments
• AWS X-Ray Propagator for seamless integration with AWS services
• Automatic AWS Lambda cold-start detection

This approach enables production-ready tracing with zero configuration, while respecting AWS Lambda's environment constraints.

from aws_lambda_powertools.tracing.otel import TracerOpenTelemetry

# Powertools creates a default TracerProvider with appropriate settings
tracer = TracerOpenTelemetry(mode="manual")

@tracer.capture_lambda_handler
def lambda_handler(event, context):
    with tracer.add_span("processing") as span:
        span.set_attribute("event_size", len(str(event)))
        result = process_data(event)
        return {"statusCode": 200, "body": result}

The default configuration includes these AWS Lambda-optimized settings:

• Lambda-aware Resource Detection: Automatically captures critical AWS Lambda context as resource attributes including:
  • service.name: Defaults to the AWS Lambda function name
  • cloud.provider: Set to "aws"
  • cloud.region: Extracted from the AWS Lambda environment
  • cloud.account.id: Extracted from the AWS Lambda function ARN
  • faas.name: AWS Lambda Function name
  • faas.version: AWS Lambda Function version/alias
  • faas.instance: AWS Lambda Instance ID for cold-start tracking
  • faas.max_memory: AWS Lambda Configured memory limit
• Optimized BatchSpanProcessor: Configured with AWS Lambda-friendly batch settings:
  • max_export_batch_size: Set to 10 (smaller than standard defaults) to optimize for AWS Lambda's limited concurrency
  • schedule_delay_millis: Set to 1000ms with more frequent flushing during AWS Lambda Shutdown to ensure traces are exported before AWS Lambda function termination
  • max_queue_size: Limited to 100 to prevent excessive memory usage in the constrained AWS Lambda environment
  • export_timeout_millis: Reduced to 3000ms to prevent blocking AWS Lambda execution for too long
• Efficient OTLP Exporter: Pre-configured for local collector communication:
  • Default endpoint: http://localhost:4317 (standard collector port within AWS Lambda when using AWS Distro for OpenTelemetry (ADOT) Lambda Layer)
• AWS-optimized Context Propagation:
  • AWS X-Ray trace context propagator enabled by default for seamless integration with AWS services
  • W3C TraceContext propagator included for standards compliance and third-party integration
• AWS Lambda-specific Instrumentation Features:
  • Cold-start detection and automatic span annotation with faas.coldstart
  • Trace context extraction from common AWS Lambda event sources (API Gateway, SQS, etc.)
  • AWS SDK span enrichment with AWS Lambda context

Setting a custom exporter with default configuration

The Powertools' Lambda-optimized TracerProvider enables developers to send their telemetry data to a specific observability provider. This hybrid approach allows developers to benefit from the default configuration, while customizing the destination for their traces - ideal for sending data in scenarios when AWS Lambda Layers/Extensions (i.e. AWS Distro for OpenTelemetry (ADOT) Lambda Layer) are not used.

from aws_lambda_powertools.tracing.otel import TracerOpenTelemetry
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

# Create custom exporter for a specific backend
exporter = OTLPSpanExporter(
    endpoint="https://api.honeycomb.io/v1/traces",
    headers={
        "x-honeycomb-team": "your-api-key",
        "x-honeycomb-dataset": "lambda-traces"
    }
)

# Use default provider but with custom exporter
tracer = TracerOpenTelemetry(mode="manual", exporter=exporter)

@tracer.capture_lambda_handler
def lambda_handler(event, context):
    return process_event(event)

@tracer.capture_method
def process_event(event):
    # Processing logic
    return {"processed": True}

Additional TracerProvider customizations

For fine-tuned control over specific aspects of your tracing configuration, you can selectively customize individual components while letting Powertools for AWS Lambda manage the rest. This targeted customization approach is ideal when you need specialized behavior - such as custom sampling logic, enhanced resource attributes, or specific trace processors - while still benefiting from Powertools for AWS Lambda optimised defaults for the remaining components:

from aws_lambda_powertools.tracing.otel import TracerOpenTelemetry
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased
from opentelemetry.sdk.trace import Span
from opentelemetry.sdk.resources import Resource
from typing import Optional, Sequence

# Custom sampler that samples 25% of requests and all errors
class ErrorAwareSampler(TraceIdRatioBased):
    def should_sample(
        self, 
        parent_context,
        trace_id: int,
        name: str,
        kind: Span.Kind = None,
        attributes: Optional[dict] = None,
        links: Optional[Sequence] = None,
        trace_state: Optional[dict] = None
    ):
        # Always sample if there's an error attribute
        if attributes and attributes.get("error", False):
            return True, None, None
        
        # Otherwise use ratio-based sampling
        return super().should_sample(parent_context, trace_id, name, kind, attributes, links, trace_state)

# Custom resource with additional attributes
resource = Resource.create({
    "team.name": "payments",
    "cost.center": "cc123",
    "deployment.region": "us-west-2"
})

# Create tracer with custom components but default provider and exporter
tracer = TracerOpenTelemetry(
    mode="manual",
    resource=resource,
    sampler=ErrorAwareSampler(0.25)
)

@tracer.capture_lambda_handler
def lambda_handler(event, context):
    # Your code here
    return {"statusCode": 200}

Multiple exporters

For specific observability architectures, you can configure your tracing setup to simultaneously send telemetry data to multiple destinations. This capability enables advanced patterns such as sending production traces to your primary observability platform while routing a copy to specialized analysis tools, maintaining a local debug stream, or implementing graceful degradation paths for your telemetry pipeline:

from aws_lambda_powertools.tracing.otel import TracerOpenTelemetry
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace.export import ConsoleSpanExporter

# Create provider
provider = TracerProvider()

# Add processor for production collector
collector_exporter = OTLPSpanExporter(endpoint="http://collector:4317")
collector_processor = BatchSpanProcessor(collector_exporter)
provider.add_span_processor(collector_processor)

# Add console exporter for debugging
console_processor = BatchSpanProcessor(ConsoleSpanExporter())
provider.add_span_processor(console_processor)

# Use provider with multiple exporters
tracer = TracerOpenTelemetry(mode="manual", tracer_provider=provider)

@tracer.capture_lambda_handler
def lambda_handler(event, context):
    with tracer.add_span("business_logic") as span:
        # Your code here
        return {"status": "success"}

Resource Attributes and OpenTelemetry Semantic Conventions

Powertools for AWS Lambda strictly follows OpenTelemetry's semantic conventions for all resource attributes. This ensures your telemetry data is standardized and compatible with any OpenTelemetry observability provider.

Standard Resource Attributes

When applicable, Powertools for AWS Lambda automatically populate Standard Resource Attributes, according to the OpenTelemetry specification:

• Cloud Provider Attributes:
  • cloud.provider: "aws"
  • cloud.region: AWS region of the AWS Lambda function
  • cloud.account.id: AWS Account ID from the AWS Lambda Function ARN
• Service Attributes:
  • service.name: AWS Lambda Function Name (customizable)
  • service.version: AWS Lambda Function Version or Deployment ID
• Function-as-a-Service Attributes:
  • faas.name: AWS Lambda Function Name
  • faas.version: AWS Lambda Function Version/Alias
  • faas.instance: AWS Lambda Function Instance ID
  • faas.max_memory: AWS Lambda Function Configured Memory Limit
• Telemetry SDK Attributes:
  • telemetry.sdk.name: "opentelemetry"
  • telemetry.sdk.language: AWS Lambda Runtime Language
  • telemetry.distro.name: "powertools-for-aws-lambda"

Custom Resource Extensions

You can extend the default Attributes while maintaining specification compliance:

from aws_lambda_powertools.tracing.otel import TracerOpenTelemetry
from opentelemetry.sdk.resources import Resource

custom_resource = Resource.create({
    "deployment.environment": "production",
    "service.namespace": "payment-processing"
})

tracer = TracerOpenTelemetry(mode="manual", resource=custom_resource)

Opinionated API for integration with OpenTelemetry SDK

Powertools for AWS will provide an opinionated API for OpenTelemetry while incorporating best practices for serverless environments. The opinionated methods abstract away the complexity of OpenTelemetry's lower-level APIs while providing optimised defaults for AWS Lambda functions.

capture_method

The capture_method decorator transforms any function or method in your code into a traced operation, by automatically creating a properly configured span. This decorator applies AWS Lambda-optimized best practices, such as exception tracking, proper span naming, and parent-child relationship:

from aws_lambda_powertools.tracing.otel import TracerOpenTelemetry

# Powertools creates a default TracerProvider with appropriate settings
tracer = TracerOpenTelemetry(mode="manual")

@tracer.capture_method(record_exception=True, set_status_on_exception=True)
def process_payment(payment_id, amount):
    # If an exception occurs, it will be automatically recorded in the span
    # and the span status will be set to ERROR
    result = charge_payment(payment_id, amount)
    return result

# You can also customize per method
@tracer.capture_method(record_exception=True, attributes={"operation.type": "database"})
def get_user_data(user_id):
    # Method execution is tracked with custom attributes
    return database.query(f"SELECT * FROM users WHERE id = {user_id}")

def lambda_handler(event, context):
    process_payment(..., ...)
    get_user_data(..., ....)

capture_lambda_handler

The capture_lambda_handler decorator creates a root span that encompasses the entire AWS Lambda invocation lifecycle. It intelligently integrates with the [opentelemetry-instrumentation-aws-lambda](https://pypi.org/project/opentelemetry-instrumentation-aws-lambda/) package to capture the AWS Lambda execution context, cold-starts, and initialization phases, while automatically extracting and propagating trace context from various event sources.

from aws_lambda_powertools.tracing.otel import TracerOpenTelemetry

tracer = TracerOpenTelemetry(mode="manual")

@tracer.capture_lambda_handler
def lambda_handler(event, context):
    # Lambda invocation is automatically traced with:
    # - Cold-start detection
    # - AWS Lambda Function name, memory, and region attributes
    # - Error handling and proper status codes
    # - Event source-specific attributes when possible
    
    payment_id = event.get("payment_id")
    result = process_payment(payment_id, event.get("amount"))
    
    return {
        "statusCode": 200,
        "body": {"payment": result}
    }

add_span

The add_span method serves as Powertools for AWS Lambda opinionated version of OpenTelemetry's
start_as_current_span, providing applicable default values. This method simplifies span creation, including error handling, automatic status management, and contextual attribute injection:

from aws_lambda_powertools.tracing.otel import TracerOpenTelemetry

tracer = TracerOpenTelemetry(mode="manual")

@tracer.capture_lambda_handler
def lambda_handler(event, context):
    # Create a span with sensible defaults for error handling
    with tracer.add_span("payment_processing", record_exception=True) as span:
        span.set_attribute("payment.id", event.get("payment_id"))
        span.set_attribute("payment.amount", event.get("amount"))
        
        # Business logic
        result = process_payment(event.get("payment_id"), event.get("amount"))
        
        # Add additional context to the span
        span.set_attribute("payment.status", result.get("status"))
        
    return {"statusCode": 200, "body": result}

# You can also use span kinds for semantic meaning
@tracer.capture_lambda_handler
def api_handler(event, context):
    # Create a span with SERVER kind for receiving external requests
    with tracer.add_span(
        "api_request", 
        kind="SERVER", 
        set_status_on_exception=True,
        attributes={"http.method": event.get("httpMethod")}
    ) as span:
        # Process the API request
        result = process_api_request(event)
        
        # Set response attributes
        span.set_attribute("http.status_code", result.get("statusCode"))
        
    return result

Direct Access to OpenTelemetry SDK

For advanced observability scenarios, when fine-grained control is required, Powertools for AWS Lambda provides direct access to the underlying OpenTelemetry SDK. This escape hatch enables instrumentation patterns beyond Powertools for AWS Lambda opinionated APIs, giving developers the flexibility to implement specialized tracing solutions, while still benefiting from Powertools' Lambda-optimized configuration:

from aws_lambda_powertools.tracing.otel import TracerOpenTelemetry
from opentelemetry.trace.status import Status, StatusCode
from opentelemetry.trace import SpanKind, Link

tracer = TracerOpenTelemetry(mode="manual")

@tracer.capture_lambda_handler
def lambda_handler(event, context):
    # Access current span through Powertools or direct OTel API
    current_span = tracer.get_current_span()
    
    try:
        # Add detailed events with timestamps to the span timeline
        current_span.add_event("processing.start", {
            "event.id": event.get("id"),
            "event.type": event.get("type")
        })
        
        # Create child span with direct OTel API access
        with tracer.provider.start_as_current_span(
            "detailed_processing", 
            kind=SpanKind.INTERNAL,
            links=[Link(current_span.get_span_context())]
        ) as processing_span:
            processing_span.set_attribute("priority", event.get("priority", "standard"))
            result = process_complex_event(event)
            processing_span.set_attribute("items.count", len(result.get("items", [])))
        
        # Record completion event with rich metadata
        current_span.add_event("processing.complete", {
            "process.duration_ms": result.get("duration"),
            "items.processed": result.get("item_count"),
            "batch.success_rate": result.get("success_rate", 1.0)
        })
        
        # Set explicit status with OTel's native APIs
        current_span.set_status(Status(StatusCode.OK))
        
        return {"statusCode": 200, "body": result}
    except Exception as e:
        # Use advanced OTel exception handling capabilities
        current_span.record_exception(
            e,
            attributes={
                "error.type": e.__class__.__name__,
                "error.handling": "custom_logic",
                "error.retry_attempted": True
            }
        )
        current_span.set_status(Status(StatusCode.ERROR, str(e)))
        raise

Potential Helper Functions

We're considering adding helper functions to simplify common OpenTelemetry workflows, particularly around context propagation across asynchronous boundaries. For example, we might create utilities to extract and inject trace context into JSON payloads for messaging systems like SQS, SNS, EventBridge, and other message-based services:

from aws_lambda_powertools.tracing.otel import TracerOpenTelemetry
import boto3
import json

tracer = TracerOpenTelemetry()
sqs = boto3.client('sqs')

@tracer.capture_lambda_handler
def lambda_handler(event, context):
    # Process the event
    result = process_event(event)
    
    # Create message with trace context
    message = {
        "data": result,
        "timestamp": "2023-12-01T12:00:00Z"
    }
    
    # Inject current trace context into the message
    context_carrier = tracer.inject_trace_context(message)
    
    # Send message with propagated context
    sqs.send_message(
        QueueUrl="https://sqs.region.amazonaws.com/account/queue",
        MessageBody=json.dumps(context_carrier)
    )
    
    return {"status": "success"}

@tracer.capture_lambda_handler
def consumer_handler(event, context):
    for record in event["Records"]:
        # Parse message body
        message = json.loads(record["body"])
        
        # Extract trace context and create span with proper parent context
        with tracer.create_span_from_context("process_message", carrier=message):
            process_message(message["data"])

However, we're not yet certain if these helper functions should be part of the core implementation or added later based on community feedback.

We welcome input from the community on whether such helpers would be valuable and how they should be designed to maximize practicality while maintaining simplicity.

AWS X-Ray to OpenTelemetry migration reference

For customers migrating from AWS X-Ray to OpenTelemetry with Powertools for AWS Lambda, the following table provides a mapping between concepts and methods:

AWS X-Ray Concept	OpenTelemetry Concept	Powertools X-Ray Method	Powertools OpenTelemetry Method
Segment	Root Span	@tracer.capture_lambda_handler	@tracer.capture_lambda_handler
---	---	---	---
Subsegment	Child Span	with tracer.capture_subsegment(name)	with tracer.add_span(name)
Annotation	Attribute (indexed)	tracer.put_annotation(key, value)	span.set_attribute(key, value)
Metadata	Attribute (non-indexed)	tracer.put_metadata(key, value)	span.set_attribute(key, value)
Trace ID	Trace ID	tracer.current_trace_id	tracer.get_current_span().get_span_context().trace_id
Segment ID	Span ID	tracer.current_entity_id	tracer.get_current_span().get_span_context().span_id
Service	Service.Name Resource Attribute	service parameter	service.name resource attribute
Patch Module	Instrumentation	tracer.patch(["boto3", "requests"])	tracer = TracerOpenTelemetry(auto_instrument=True)
Cold Start	Attribute	Built-in annotation	Built-in attribute faas.coldstart
Ignore Endpoint	Filter Request	tracer.ignore_endpoint(hostname, urls)	tracer.ignore_endpoint(hostname, urls)
Custom Sampler	Sampler	X-Ray specific	sampler=CustomSampler() when creating provider
Subsegment Capture Method	Span Around Method	@tracer.capture_method	@tracer.capture_method

Evolving with the community

At Powertools for AWS Lambda, our approach is deliberately progressive. This initial implementation of OpenTelemetry tracing represents our first step into a broader integration with the OpenTelemetry ecosystem. We recognize that we may not have all the answers today, which is why community feedback will be essential in shaping our path forward.

We're looking ahead to a future where OpenTelemetry becomes a comprehensive observability foundation across all Powertools for AWS Lambda components. We anticipate developing additional RFCs for Metrics and Logging using OpenTelemetry, creating a cohesive observability experience that leverages industry standards while maintaining the developer-friendly experience Powertools for AWS lambda is known for.

We also envision integrating OpenTelemetry with other Powertools for AWS Lambda utilities. For example, a future “BatchProcessing” implementation might include built-in instrumentation:

from aws_lambda_powertools import BatchProcessor
from aws_lambda_powertools.tracing.otel import TracerOpenTelemetry

tracer = TracerOpenTelemetry()

processor = BatchProcessor(
    event_type=EventType.SQS,
    tracer=tracer
)

@tracer.capture_lambda_handler
def lambda_handler(event, context):
    with tracer.add_span("processing_batch"):
        # The BatchProcessor automatically creates spans for each record
        # and tracks processing outcomes, retries, and batch statistics
        `return process_partial_response(event=event, record_handler=record_handler, processor=processor, context=context)`

Out of scope

To maintain focus on delivering high-quality core functionality, the following items are explicitly out of scope for this implementation:

1. ADOT Collector Provisioning: Powertools for AWS Lambda will not create any AWS Lambda Layer/Extension containing the AWS Distro for OpenTelemetry Collector. Developers will need to use existing solutions such as the official AWS Distro for OpenTelemetry (ADOT) Lambda Layer, or configure OpenTelemetry SDK to handle transport themselves.
2. API Extensions Beyond OpenTelemetry: Powertools for AWS Lambda will not create new APIs that extend beyond the scope of OpenTelemetry's standard capabilities. Our focus remains on providing opinionated, AWS Lambda-optimized wrappers for existing OpenTelemetry functionality.
3. Cold-Start/Extension Overhead Mitigation: Addressing cold-start penalties or additional execution time generated by AWS Lambda Layers/Extensions that serve as transport mechanisms for OpenTelemetry signals is out of scope. While our implementation will be optimized for AWS Lambda environments, the overhead introduced by external components like AWS Distro for OpenTelemetry Collector is outside our control.

We need your feedback

This RFC represents an initial proposal for integrating OpenTelemetry with Powertools for AWS Lambda, but we need your feedback to ensure we're building a solution that truly meets your needs.

What are we missing? Are there specific use cases we haven't addressed? How can we further improve the developer experience? Are there particular instrumentation patterns that would make your observability workflows more effective?

Please share your thoughts in this RFC, or if you prefer, via email at [email protected]

[RaphaelManke]

Great initiative
I do have some additional resource attributes which should always be set:

• cloud.platform = aws
• cloud.resource_id = <lambda arn>

Additionally I do se a problem with setting the account id as resource attribute because is not available at cold start (init) time.
A solution for that could be extracting the account id from the AWS credentials.
See this article as reference https://awsteele.com/blog/2020/09/26/aws-access-key-format.html

[leandrodamascena]

Hey @RaphaelManke I think we can make it available when customers using @tracer.capture_lambda_handler because we can intercept the Lambda execution and at this point we will have Lambda ARN + Accountid. Also, if we auto-instrument the function with https://pypi.org/project/opentelemetry-instrumentation-aws-lambda/ this will collect.

But good point, let me think about edge-cases we can have.

[RaphaelManke]

Ok.
The problem with the account id is also a general problem.
So most of the instrumentations are skipping the account id and then add lambda arn to the first span.
But that is quite hard to use because the spans are not processed all together so that backfilling those values as resource attributes often not works.
Resource attributes are not that easy added after the initial initialization.

[RaphaelManke]

Another thing that comes to mind, where the powertools can make a great impact, is the batch processors. In OpenTelemetry there is the concept of span links, which is intended to be used in messaging scenarios.
See

• https://opentelemetry.io/docs/specs/semconv/messaging/sqs/
• https://opentelemetry.io/docs/specs/semconv/messaging/messaging-spans/#examples

The problem is that in messaging scenarios you don't want to have all messaging processors in one giant trace. To mitigate that it can be split into multiple ones and link them.
Current instrumentations cannot do that correctly because its depending on how you process a batch of records. With Powertools we have an opinionated way of processing messages and thereby add this links while processing events.

I did a POC implementation here
https://github.com/RaphaelManke/otel-lambda-nodejs-messaging-example/blob/main/src/patches/powertools/instrumented-batch-processor.ts

[leandrodamascena]

Thanks for sharing the semantic convention links, we will follow them in all spans/attributes we create on customer's behalf.

Hmmm you're saying that the producer create the root span and then the consumers make then part as a Span Link and then you can observe it better in the observability tools? Hmm I really liked this idea, my initial idea was make everything part of the same tracer, but I see your point.

Another point here. I see you created some extractors, can you please explain a little bit more what you do with those and what kind information you extract?

[RaphaelManke]

The extractors are a little similar to the parser from powertools. The essentially encapsulate the extraction of the semantic attributes that are added to the span.

[jeastham1993]

Just to add to this discussion, I think it should be configurable whether you link or propagate. I think it's a bit more nuanced than just defaulting to one or the other. I've got some links to conferences talks I've given that elaborate if you're interested.

[RaphaelManke]

For the parser utility it should also extract and set the relevant semantic convention attributes.
E.g. for http requests, sqs records etc.
So when I use the parser utility the initial span, wrapping the handler, should have matching attributes set after the payload is parsed correctly. This also includes setting the span kind like described here

• https://opentelemetry.io/docs/concepts/signals/traces/#span-kind

[leandrodamascena]

I'm not sure I understand. Could you explain a little more?

[RaphaelManke]

Whenever the parser is called it should not only validate the event structure but also be able to set the correct span attributes based on the semantic conventions. E.g. when a sqs message is parsed then this span might be a CONSUMER span type.

[pcolazurdo]

Just one thought: The integration should have an easy way to fallback to other tracing mechanisms (i.e. X-Ray, Logging) in case the OTEL SDK or the OTEL Layer is not available. Also, what are the strategies you are considering handling tracing in local environments? There should be a mechanism to fallback to a local configuration in case the user is testing the function and they use a remote OTEL endpoint.

[leandrodamascena]

Hey Pablo, I really appreciate your insights here @pcolazurdo!

Just one thought: The integration should have an easy way to fallback to other tracing mechanisms (i.e. X-Ray, Logging) in case the OTEL SDK or the OTEL Layer is not available.

This is something we are targeting: a similar experience to use Powertools with OTEL or XRAY. So, the idea is something like this:

with xray

from aws_lambda_powertools import Tracer

tracer = Tracer()
...
...
tracer.add_segment()

with otel

from aws_lambda_powertools import TracerOtel

tracer = TracerOtel()

...
...
# Keep this name for backwards compatibility.
tracer.add_segment()

Or are you thinking about an automatic fallback? Can you please explain a little bit more?

Also, what are the strategies you are considering handling tracing in local environments? There should be a mechanism to fallback to a local configuration in case the user is testing the function and they use a remote OTEL endpoint.

I think the OTEL implementation covers this well. Everything in OTEL can be set using ENV VAR or programmatically in the SDK, which means that customers using automatic or manual instrumentation will need to have opentelemetry-sdk installed locally in they DEV ENV and will be able to set the ENV VAR for any endpoint they have, including local or remote. Does that make sense?

[leandrodamascena]

Tracing in local environments will most likely be disabled altogether just like we do today with X-Ray. The Tracer utility automatically detects that it's not running in AWS Lambda, and all tracing operations in your code become no-op without you having to change anything in your code.

This is something I hadn't considered, but I think we need to think about it a bit more, @dreamorosi! Using OTEL is a bit different from XRay, as we don't necessarily need to rely on the root span/segment created by Lambda, and customers can use Tracer in their local environment. We already have the POWERTOOLS_TRACE_DISABLED environment variable, so we can tell customers to use it, but not disable tracing since it's outside of the Lambda environment. What do you think?

[dreamorosi]

For the current implementation of Tracer, we disable because in the absence of the X-Ray daemon the AWS X-Ray SDK would essentially crash. If this is not the case for OTEL and there's a safe way to run it without having to do a complex setup then I am not against keeping it enabled, although I don't know what would be the benefit - do we expect/think customers will want to test the data being emitted?

[pcolazurdo]

My thought process was:

1. First to avoid crashes or other weird situations in case there is no access to an otel collector - I've seen cases where, because some libraries auto-instrumented with Otel, any exception code keeps triggering the instrumention which fails and this creates a loop. The implementation should be careful to avoid this.
2. Secondly, it would be nice to have the chance to see the tracing information locally if the user has a local setup. It's pretty easy to create a tracing collector and visualization with Otel/Jaegger in local environment and this can be really useful in some cases.
3. Third, be sure that the tracing implementation is compatible with OTEL in all its forms, and it still works if the user doesn't use the opinionated Lambda Layer.

[dreamorosi]

Thank you for clarifying, that makes total sense and wasn't aware of these nuances for local envs for instrumented packages, we will keep in mind during testing.

[yagrxu]

Some questions from my side:

1. Only trace supported? as spanmetrics connector brings some key metrics for services. I hope metrics can be supported as well. ECS for example supports both trace and metrics with ADOT sidecar.
2. I am not sure if ADOT or OpenTelemetry as ADOT does not support connectors and many components in Otel community.
3. What about logs? any options for log storage other than CloudWatch Logs?
4. Currently Lambda supports powertools, ADOT layer, Active Tracing and Application Signals. What will happen if more than 1 of them are enabled? any conflicts?

[scottgerring]

Cool 🫶 !
Some quick thoughts in no particular order:

API Indirection

Powertools for AWS will provide an opinionated API for OpenTelemetry while incorporating best practices for serverless environments. The opinionated methods abstract away the complexity of OpenTelemetry's lower-level APIs while providing optimised defaults for AWS Lambda functions.

I think it makes sense to have an annotation to capture the handler and whack on all the lambda-specific-request stuff we're going to want:

@tracer.capture_method

... but for the rest I don't really see the value of explicitly coupling on PT, unless this is just a question of preserving the existing interface PT offers?

Said otherwise, I am a bit leery of encouraging users to use a layer of indirection over the OTel-API, which is really intended to be the lingua franca abstraction for instrumenting apps and libraries.

Perhaps you can clarify what problem with the underlying API you are trying to solve. Insofar as this is an RFC for all of PowerTools, this is probably a good point to go look at how this works in other languages too. In java for instance, the annotations are straightforward to use, and you have the option of falling back to the explicit span API if you need. Rust is a bit more verbose because it doesn't have AOP but again the API isn't particularly onerous.

Instrument 3rd party libraries

We're considering adding helper functions to simplify common OpenTelemetry workflows, particularly around context propagation across asynchronous boundaries. For example, we might create utilities to extract and inject trace context into JSON payloads for messaging systems like SQS, SNS, EventBridge, and other message-based services:

I think this is a good idea, where it doesn't already happen. Some auto-instrumentation libraries will handle this for you, so you'll probably want the user to opt into this behaviour. Also FWIW I think the most "otel-native" thing to do would be that the libraries themselves - e.g. the AWS SQS client - would be directly instrumented with the OTel-api. Then everything 🪄 just works 🪄 !

Tracer Creation

In the case where we are auto-instrumenting, what does TracerOpenTelemetry(mode='auto') do?

Manual Instrumentation

When developers need granular control over their OpenTelemetry configuration, they can create and fully customize their own TracerProvider before passing it to Powertools for AWS Lambda.

Why does this need to be explicitly provided to PT?

Batching & Flushing

If a user sets up a batched exporter and a single request runs through, what's to ensure that the batch is sent before the lambda environment is shut down?

Other Signals

Interested in how you see this fitting in with logging & metrics!

[jeastham1993]

To add to Scott's API indirection point, I do agree completely. One of the points of OTEL is to have an agnostic way to instrument application code. If my codebase is littered with PowerTools specific stuff that kind of breaks that model.

I think the value PowerTools could add is a plugin-able way to wrap the Lambda handler and add all the default span attribites (region, provider etc etc). And then also extract trace context from inbound payloads (because we know the structure), making it configurable whether to parent-child or span link.

A user, in their application code, should use the OTEL SDK, right?

[jeastham1993]

Some thoughts from me :)

ADOT layer

Is there any work being done to improve the performance of the ADOT layer? Last time I checked it really pushed up init duration. I understand there's the option to point the functionality at a different OTEL endpoint, but I think the impact to perf should be pointed out.

Flushing

I think you mentioned this, but just wanted to be absoloutely sure, how are you going to guarantee that if the environment shuts down that traces are flushed? Using a BatchExporter makes this a more challenging.

Automatic Propagation

How would it allow users to use any of the existing OTEL SDK extensions? You mention the ability to auto inject, but I think OTEL already supports that for most languages? And then from an extraction perspective, how do you make it configurable? To @RaphaelManke point above, SemConv would recommend using SpanLinks, but I think it would need to be configurable.

Trace/Log Correlation

A couple of others have mentioned logs, but would this include/should this include any work to correlate logs and traces together. E.g. if someone is using PowerTools Logger and Tracers, could it auto-inject the current traceID into all log messages? I think that happens with X-Ray today, but if people aren't using X-Ray I don't know how useful that is.

[SodaDev]

I think it would be great to see utils that automatically instrument metrics related to messaging (https://opentelemetry.io/docs/specs/semconv/messaging/messaging-spans/) attributes based on the type of ESM attached to the handler. Will this be supported?

[SodaDev]

ADOT layer - I believe this layer should be reviewed.

Official lambda layer at https://github.com/open-telemetry/opentelemetry-lambda/blob/main/collector/README.md is right and says that:
"Configuring the Lambda Collector without the decouple processor and batch processor can lead to performance issues. So the OpenTelemetry Lambda Layer automatically adds the decouple processor to the end of the chain if the batch processor is used and the decouple processor is not." None of them is available at the adot collector.

[pcolazurdo]

If this is the case, we should open an Issue with the ADOT team. I'll try to see if we can find the reason for this.

[pcolazurdo]

According to the documentation, the ADOT seem to include opentelemetry-lambda @ 5cc1580 which includes the auto-config to add the decoupler if a batch processor was used by the user. This seems to follow the same pattern than the collector does.
Can you share more details about why you think this is not the case?

[SodaDev]

I recently tried to add batch and decouple processor on one of the latest adot layers and it failed at start time due to missing processor

[SodaDev]

Would be great to add CloudEvents support that have distributed tracing extension as a part of the standard to easily trace events:

https://github.com/cloudevents/spec/blob/main/cloudevents/extensions/distributed-tracing.md

[leandrodamascena]

Would be great to add CloudEvents support that have distributed tracing extension as a part of the standard to easily trace events:

cloudevents/spec@main/cloudevents/extensions/distributed-tracing.md

Hi @SodaDev, thanks for raising this idea. But in CloudEvents, you can set the tracingid to propagate it, right? I'm not sure I understand how we can integrate this beyond providing the tracerid. Can you explain your idea a little better?