OTEL Exporter performance bottleneck when exporting to Grafana from AWS Lambda #1123
Description
name: Bug report
about: Create a report to help us improve
title: "OTEL Exporter performance bottleneck when exporting to Grafana from AWS Lambda"
labels: bug
assignees: ''
Describe the bug
When using the OTEL exporter in AWS Lambda to send traces, metrics, and logs to Grafana, we are experiencing significant performance bottlenecks. The exporter appears to block and delay responses from Lambda back to API Gateway, resulting in increased request latency and degraded user experience.
Steps to reproduce
- Instrument an AWS Lambda function with the OTEL DOT layer configured to export traces, metrics, and logs to Grafana.
- Trigger the Lambda via API Gateway.
- Observe response time and function execution duration.
- Compare with baseline (without OTEL exporter enabled).
What did you expect to see?
The OTEL exporter should send telemetry data asynchronously, with minimal impact on Lambda response time and no noticeable delay for API Gateway responses regardless of if the target OTEL endpoint encounters issues.
What did you see instead?
Lambda responses are delayed, and function execution time is noticeably increased. The OTEL exporter seems to block execution and causes slow exports to Grafana, directly impacting API Gateway latency. Issue was found due to the Grafana endpoint being slow but we expect OTEL not to block the lambda.
What version of collector/language SDK version did you use?
Version: (e.g., v0.58.0, v1.11.0)
(arn:aws:lambda::901920570463:layer:aws-otel-nodejs--ver-1-30-2:1)
arn:aws:lambda::901920570463:layer:aws-otel-collector--ver-0-117-0:1
What language layer did you use?
Config: (e.g., Java, Python, etc)
Javascript
.NET
Additional context
- We are considering migrating to export telemetry data directly from AWS account to Grafana, bypassing the OTEL DOT layer due to these performance issues.
- Are there any recommended best practices or configuration changes to improve exporter performance in AWS Lambda environments?
- Is there guidance on asynchronous/non-blocking export for OTEL in Lambda?
- Any known issues or workarounds for this scenario?