SigV4 Authentication Support for OTLP HTTP Logs Exporter #358

liustve · 2025-05-02T22:12:07Z

Background
Supporting ADOT auto instrumentation to automatically inject SigV4 authentication headers for outgoing export log requests to the allow exporting to the AWS Logs OTLP endpoint. Users will need to configure the following environment variables in order to enable and properly run this exporter:

OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://logs.[AWS-REGION].amazonaws.com/v1/logs; required
OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=[CW-LOG-GROUP-NAME],x-aws-log-stream=[CW-LOG-STREAM-NAME] required
OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true required
OTEL_LOGS_EXPORTER=otlp required or do not set env variable
OTEL_EXPORTER_OTLP_LOGS_PROTOCOL=http/protobuf required or do not set env variable
OTEL_METRICS_EXPORTER=none

Description of changes:

Add new AwsAuthSession class to inject Sigv4 headers directly into the sessions object used by the upstream exporter.
https://github.com/srprash/aws-otel-sigv4-auth/tree/main
The ADOT auto instrumentation is now configured to automatically detect if a user is exporting to CW Logs OTLP Logs endpoint by checking if the environment variable OTEL_EXPORTER_OTLP_LOGS_ENDPOINT is configured to match this url pattern: https://logs.[AWS-REGION].amazonaws.com/v1/logs

Testing:

E2E test done in an empty EC2 environment without configuring .aws credentials config file or setting AWS credentials in the environment variable
Manual testing was done by configuring the above environment variables and setting up the sample app locally with ADOT auto instrumentation and verified the spans in CW Logs.
The sample app was run and rerun 30 times and confirmed no issues with exporting the logs to the endpoint
Unit tests were added to verify functionality of OtlpAwsLogsExporter

Further testing will be done with the Release tests.

Example of a log exported using this exporter:

{
    "resource": {
        "attributes": {
            "aws.local.service": "test",
            "service.name": "test",
            "cloud.region": "us-west-2",
            "host.type": "c5.4xlarge",
            "cloud.availability_zone": "us-west-2c",
            "telemetry.sdk.name": "opentelemetry",
            "telemetry.sdk.language": "python",
            "cloud.provider": "aws",
            "cloud.account.id": "571600841604",
            "telemetry.sdk.version": "1.27.0",
            "host.name": "ip-172-31-7-29.us-west-2.compute.internal",
            "cloud.platform": "aws_ec2",
            "host.id": "i-0b04d6affbae7d629",
            "telemetry.auto.version": "0.9.0.dev0-aws"
        }
    },
    "scope": {
        "name": "opentelemetry.sdk._logs._internal"
    },
    "timeUnixNano": 1746221090210187520,
    "observedTimeUnixNano": 1746221090210645309,
    "severityNumber": 9,
    "severityText": "INFO",
    "body": "\u001b[31m\u001b[1mWARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.\u001b[0m\n * Running on http://127.0.0.1:8082",
    "attributes": {
        "code.filepath": "/home/ec2-user/aws-otel-python-instrumentation/auto_instrumentation/lib64/python3.11/site-packages/werkzeug/_internal.py",
        "otelTraceSampled": false,
        "code.function": "_log",
        "code.lineno": 97,
        "otelTraceID": "0",
        "otelSpanID": "0",
        "otelServiceName": "test"
    },
    "traceId": "",
    "spanId": ""
}

yiyuan-he · 2025-05-02T22:50:50Z

Thanks Steve - Code structure looks clean overall with good separation of concerns. Did some E2E testing as well and can confirm functionality.

My only concern is the approach to use multiple inheritance:

Python's method resolution order can be unintuitive when methods are overriden in multiple parent classes.
The code manually calls both parent __init__ methods in a specific order. This is fragile and could break if either parent class implementation changes.
When upstream makes changes to parent class, the multiple inheritance can make tracking changes more difficult.

Perhaps we could consider another approach where we simplify by just extracting the common SigV4 auth into a utility function or class.

For example:

# For the span exporter
class OTLPAwsSpanExporter(OTLPSpanExporter):
    def _export(self, serialized_data: bytes):
        from amazon.opentelemetry.distro.exporter.otlp.aws.common.sigv4_auth import apply_sigv4_auth
        apply_sigv4_auth(self._endpoint, self._session, "xray", serialized_data)
        return super()._export(serialized_data)

# For the logs exporter
class OTLPAwsLogExporter(OTLPLogExporter):
    def _export(self, serialized_data: bytes):
        from amazon.opentelemetry.distro.exporter.otlp.aws.common.sigv4_auth import apply_sigv4_auth
        apply_sigv4_auth(self._endpoint, self._session, "logs", serialized_data)
        return super()._export(serialized_data)

aws-opentelemetry-distro/src/amazon/opentelemetry/distro/aws_opentelemetry_configurator.py

aws-opentelemetry-distro/tests/amazon/opentelemetry/distro/test_aws_auth_session.py

srprash

Also, you would want to update the PR description since it is no longer accurate. :)

aws-opentelemetry-distro/src/amazon/opentelemetry/distro/aws_opentelemetry_configurator.py

…le gzip by default

*Background:* #358 The above PR got rid of OtlpAwsSpanExporter and OtlpAwsLogExporter as to use the default upstream exporters as that was a cleaner approach. However, we need the OtlpAwsSpanExporter and OtlpAwsLogExporter classes to support later requirements for Gen AI as we need to override the export method. *Description of changes:* This PR reintroduces the OtlpAwsSpanExporter and OtlpAwsLogExporter classes to support later requirements for Gen AI as we need to override the export method. This change does not introduce anything new and the Sigv4 span + logs exporter still work as intended: Logs: ``` { "resource": { "attributes": { "aws.local.service": "test", "service.name": "test", "cloud.region": "us-west-2", "host.type": "c5.4xlarge", "cloud.availability_zone": "us-west-2c", "telemetry.sdk.name": "opentelemetry", "telemetry.sdk.language": "python", "cloud.provider": "aws", "cloud.account.id": "571600841604", "telemetry.sdk.version": "1.27.0", "host.name": "ip-172-31-7-29.us-west-2.compute.internal", "cloud.platform": "aws_ec2", "host.id": "i-0b04d6affbae7d629", "telemetry.auto.version": "0.9.0.dev0-aws" } }, "scope": { "name": "opentelemetry.sdk._logs._internal" }, "timeUnixNano": 1747074906326769664, "observedTimeUnixNano": 1747074906326822815, "severityNumber": 9, "severityText": "INFO", "body": "127.0.0.1 - - [12/May/2025 18:35:06] \"GET /server_request HTTP/1.1\" 200 -", "attributes": { "code.filepath": "/home/ec2-user/.local/lib/python3.9/site-packages/werkzeug/_internal.py", "otelTraceSampled": false, "code.function": "_log", "code.lineno": 97, "otelTraceID": "0", "otelSpanID": "0", "otelServiceName": "test" }, "traceId": "", "spanId": "" } ``` Spans: ``` { "resource": { "attributes": { "aws.local.service": "test", "service.name": "test", "cloud.region": "us-west-2", "host.type": "c5.4xlarge", "cloud.availability_zone": "us-west-2c", "telemetry.sdk.name": "opentelemetry", "telemetry.sdk.language": "python", "cloud.provider": "aws", "cloud.account.id": "571600841604", "telemetry.sdk.version": "1.27.0", "host.name": "ip-172-31-7-29.us-west-2.compute.internal", "cloud.platform": "aws_ec2", "host.id": "i-0b04d6affbae7d629", "telemetry.auto.version": "0.9.0.dev0-aws" } }, "scope": { "name": "opentelemetry.instrumentation.flask", "version": "0.48b0" }, "traceId": "68223f2d375733237e24512171012437", "spanId": "34dd5a89d7a21fdf", "flags": 256, "name": "GET /", "kind": "SERVER", "startTimeUnixNano": 1747074861988123056, "endTimeUnixNano": 1747074861988911516, "durationNano": 788460, "attributes": { "net.host.name": "localhost:8082", "aws.local.service": "test", "net.peer.port": 57356, "telemetry.extended": "true", "http.target": "/", "http.flavor": "1.1", "net.peer.ip": "127.0.0.1", "http.host": "localhost:8082", "aws.local.environment": "ec2:default", "http.status_code": 404, "aws.local.operation": "GET /", "aws.span.kind": "LOCAL_ROOT", "http.server_name": "127.0.0.1", "http.user_agent": "curl/8.5.0", "net.host.port": 8082, "PlatformType": "AWS::EC2", "http.method": "GET", "http.response.status_code": 404, "http.scheme": "http" }, "status": { "code": "UNSET" } } ``` By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

implement sigv4 support for logs exporter

dd37193

liustve requested a review from a team as a code owner May 2, 2025 22:12

liustve marked this pull request as draft May 2, 2025 22:12

liustve requested review from majanjua-amzn, srprash and yiyuan-he May 2, 2025 22:13

add otlp logs exporter sigv4 support

4613dbf

liustve force-pushed the sigv4_logs branch from 02e72f8 to 4613dbf Compare May 2, 2025 23:41

liustve added 9 commits May 4, 2025 18:25

deleted base otlp aws class and added aws auth session

52a32a8

removed print

329007f

add unit testing + formatting

529079c

linting fix

ec855cc

lint fix

cde87a6

linting fix

efafc4b

linting fix

a6f4a0b

added configurator unit tests

8241661

lint fix

69d225c

liustve marked this pull request as ready for review May 5, 2025 22:03

srprash reviewed May 6, 2025

View reviewed changes

mxiamxia mentioned this pull request May 6, 2025

feat: handle LLOs in TracerProvider pipeline [WIP] #357

Closed

removed aws span and logs exporter

03877f1

mxiamxia reviewed May 7, 2025

View reviewed changes

aws-opentelemetry-distro/tests/amazon/opentelemetry/distro/test_aws_auth_session.py Outdated Show resolved Hide resolved

liustve added 2 commits May 7, 2025 22:00

default to gzip compression

4ad4435

default to gzip compression

6c396db

liustve force-pushed the sigv4_logs branch from 23940f3 to 6c396db Compare May 8, 2025 23:56

srprash reviewed May 9, 2025

View reviewed changes

liustve added 3 commits May 9, 2025 20:36

add underscore for private helper methods and comment for why we enab…

97b2bb5

…le gzip by default

lint fix

4f4dae0

Merge branch 'main' into sigv4_logs

701aaec

srprash approved these changes May 9, 2025

View reviewed changes

liustve merged commit cb3a1b0 into aws-observability:main May 9, 2025
12 checks passed

liustve deleted the sigv4_logs branch May 9, 2025 22:17

liustve mentioned this pull request May 12, 2025

Add OtlpAwsLogExporter and OtlpAwsSpanExporter #360

Merged

thpierce mentioned this pull request Jun 23, 2025

Add AwsBatchLogProcessor and OtlpAwsLogExporter Logs Pipeline #402

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

SigV4 Authentication Support for OTLP HTTP Logs Exporter #358

SigV4 Authentication Support for OTLP HTTP Logs Exporter #358

Uh oh!

liustve commented May 2, 2025 •

edited

Loading

Uh oh!

yiyuan-he commented May 2, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

srprash left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

SigV4 Authentication Support for OTLP HTTP Logs Exporter #358

SigV4 Authentication Support for OTLP HTTP Logs Exporter #358

Uh oh!

Conversation

liustve commented May 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yiyuan-he commented May 2, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

srprash left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

liustve commented May 2, 2025 •

edited

Loading