Skip to content

Conversation

srprash
Copy link
Contributor

@srprash srprash commented Aug 7, 2024

Description

Implementing a UDP exporter for OTLP spans and metrics. Few notes:

  • If AppSignals is enabled in Lambda, the OTel spans and Application Signals metrics are always exported via this UdpExporter. Currently it can't be configured to use any other exporter.
  • Custom OTel metrics can still be exported (if enabled and requires using ADOT Collector layer) via user-specified exporter.
  • OTel spans and Appsignals metrics are first protobuf encoded, then base64 encoded, then prefixed with T1 or M1 respectively, and finally appended to the '{"format":"json","version":1}\n' string before being exported to the address provided in the AWS_XRAY_DAEMON_ADDRESS or default 127.0.0.1:2000.

Testing

  • Test script test_lambda.py to generate 2 nested spans:
    if __name__ == "__main__":
    tracer = trace.get_tracer("test-tracer")
    with tracer.start_as_current_span("parent"):
        # Attach a new child and update the current span
        with tracer.start_as_current_span("child"):
            # sleep for 1s
            sleep(1)
        # Close child span, set parent as current
    # Close parent span, set default span as current
  • Command to run the script with auto-instrumentation:
    OTEL_TRACES_EXPORTER=otlp \
    OTEL_METRICS_EXPORTER=none \
    OTEL_LOGS_EXPORTER=none \
    OTEL_AWS_APPLICATION_SIGNALS_ENABLED=true \
    OTEL_PYTHON_DISTRO=aws_distro \
    OTEL_PYTHON_CONFIGURATOR=aws_configurator \
    OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \ # this config will be ignored since appsignals is enabled in lambda
    OTEL_TRACES_SAMPLER=always_on \
    OTEL_RESOURCE_ATTRIBUTES="service.name=lambda_test" \
    AWS_LAMBDA_FUNCTION_NAME="lambda_func_name" \
    opentelemetry-instrument python test_lambda.py
  • Printed spans in JSON format
    Raw spans:
{
    "name": "child",
    "context": {
        "trace_id": "0x66bbf01064511ad55e9fe435a1326167",
        "span_id": "0x0915f345ead6d194",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": "0xbd128e94b0caa171",
    "start_time": "2024-08-13T23:45:20.930139Z",
    "end_time": "2024-08-13T23:45:21.935234Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {
        "aws.local.operation": "InternalOperation"
    },
    "events": [],
    "links": [],
    "resource": {
        "attributes": {
            "telemetry.sdk.language": "python",
            "telemetry.sdk.name": "opentelemetry",
            "telemetry.sdk.version": "1.25.0",
            "service.name": "lambda_test",
            "telemetry.auto.version": "0.3.0.dev0-aws"
        },
        "schema_url": ""
    }
}
{
    "name": "parent",
    "context": {
        "trace_id": "0x66bbf01064511ad55e9fe435a1326167",
        "span_id": "0xbd128e94b0caa171",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": null,
    "start_time": "2024-08-13T23:45:20.930107Z",
    "end_time": "2024-08-13T23:45:21.935353Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {
        "aws.local.operation": "InternalOperation",
        "aws.local.service": "lambda_test",
        "aws.span.kind": "LOCAL_ROOT"
    },
    "events": [],
    "links": [],
    "resource": {
        "attributes": {
            "telemetry.sdk.language": "python",
            "telemetry.sdk.name": "opentelemetry",
            "telemetry.sdk.version": "1.25.0",
            "service.name": "lambda_test",
            "telemetry.auto.version": "0.3.0.dev0-aws"
        },
        "schema_url": ""
    }
}

Encoded trace data from SDK:

"T1CvcDCrkBCiIKFnRlbGVtZXRyeS5zZGsubGFuZ3VhZ2USCAoGcHl0aG9uCiUKEnRlbGVtZXRyeS5zZGsubmFtZRIPCg1vcGVudGVsZW1ldHJ5CiEKFXRlbGVtZXRyeS5zZGsudmVyc2lvbhIICgYxLjI1LjAKHQoMc2VydmljZS5uYW1lEg0KC2xhbWJkYV90ZXN0CioKFnRlbGVtZXRyeS5hdXRvLnZlcnNpb24SEAoOMC4zLjAuZGV2MC1hd3MSuAIKDQoLdGVzdC10cmFjZXISdQoQZrvwEGRRGtVen+Q1oTJhZxIICRXzRerW0ZQiCL0SjpSwyqFxKgVjaGlsZDABOXhrfZxybusXQdDzZdhybusXSioKE2F3cy5sb2NhbC5vcGVyYXRpb24SEwoRSW50ZXJuYWxPcGVyYXRpb256AIUBAAEAABKvAQoQZrvwEGRRGtVen+Q1oTJhZxIIvRKOlLDKoXEqBnBhcmVudDABOXjufJxybusXQajEZ9hybusXSioKE2F3cy5sb2NhbC5vcGVyYXRpb24SEwoRSW50ZXJuYWxPcGVyYXRpb25KIgoRYXdzLmxvY2FsLnNlcnZpY2USDQoLbGFtYmRhX3Rlc3RKHQoNYXdzLnNwYW4ua2luZBIMCgpMT0NBTF9ST09UegCFAQABAAA="

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@srprash srprash force-pushed the lambda_exporter branch 2 times, most recently from 040fcb4 to 8a4d47a Compare August 7, 2024 19:52
Copy link
Contributor

@wangzlei wangzlei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you help me understand how to configure span_exporter to be UDP exporter?

@srprash srprash marked this pull request as ready for review August 14, 2024 06:04
@srprash srprash requested a review from a team as a code owner August 14, 2024 06:04
Copy link
Contributor

@wangzlei wangzlei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Look good to me!

@vastin vastin merged commit 1d0aedd into main Aug 14, 2024
11 checks passed
@vastin vastin deleted the lambda_exporter branch August 14, 2024 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants