Skip to content

Conversation

@mxiamxia
Copy link
Member

@mxiamxia mxiamxia commented Jun 5, 2025

This PR introduces the core structure for the CloudWatch EMF exporter for sending custom metrics. So for some specific customers they can send their own custom OTel metrics to CloudWatch metrics backend without requiring any Collector or Agent installed at this moment.

This PR includes:
Changes:

  • Basic EMF log structure creation
  • Gauge metric conversion and export
  • Unit mapping from OTel to CloudWatch units
  • Metric grouping by attributes and timestamps
  • Supports DELTA temporality for CloudWatch compatibility

Future PRs will add:

  • Support for Sum, Histogram, and ExponentialHistogram metrics
  • Advanced batching and CloudWatch Logs constraints
  • Enhanced error handling and validation

Testing:

  • Comprehensive unit tests for core functionality
  • Mock-based testing to avoid AWS dependencies
  • Tests for initialization, conversion, and basic export flow
cpu_usage_gauge = meter.create_gauge(
        name="system_cpu_usage_percent",
        description="Current CPU usage percentage",
        unit="percent"
    )
cpu_usage_gauge.set(0.2, {"host": "server-01", "region": "us-east-1"})
image
{
    "_aws": {
        "Timestamp": 1749774096539,
        "CloudWatchMetrics": [
            {
                "Namespace": "MyApplication1",
                "Dimensions": [
                    [
                        "host",
                        "region"
                    ]
                ],
                "Metrics": [
                    {
                        "Name": "system_cpu_usage_percent",
                        "Unit": "percent"
                    }
                ]
            }
        ]
    },
    "Version": "1",
    "otel.resource.telemetry.sdk.language": "python",
    "otel.resource.telemetry.sdk.name": "opentelemetry",
    "otel.resource.telemetry.sdk.version": "1.27.0",
    "otel.resource.service.name": "my-service",
    "otel.resource.service.version": "0.1.0",
    "otel.resource.deployment.environment": "production",
    "system_cpu_usage_percent": 0.2,
    "host": "server-01",
    "region": "us-east-1"
}

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@mxiamxia mxiamxia requested a review from a team as a code owner June 5, 2025 04:12
@mxiamxia mxiamxia force-pushed the pr1-emf-exporter-foundation branch 2 times, most recently from bc2ce38 to 8491507 Compare June 5, 2025 04:19
@mxiamxia mxiamxia force-pushed the pr1-emf-exporter-foundation branch 12 times, most recently from 22ab59b to 5b6d407 Compare June 10, 2025 23:40
This PR introduces the core structure for the CloudWatch EMF (Embedded Metric Format)
exporter for OpenTelemetry Python. This foundation includes:

Core Features:
- CloudWatchEMFExporter class with full initialization
- Basic EMF log structure creation
- Gauge metric conversion and export
- Unit mapping from OTel to CloudWatch units
- CloudWatch Logs integration with log group/stream management
- Metric grouping by attributes and timestamps
- Basic error handling and logging

Architecture:
- Follows OTel MetricExporter interface
- Uses boto3 for CloudWatch Logs integration
- Implements proper resource attribute handling
- Supports DELTA temporality for CloudWatch compatibility

Testing:
- Comprehensive unit tests for core functionality
- Mock-based testing to avoid AWS dependencies
- Tests for initialization, conversion, and basic export flow

Future PRs will add:
- Support for Sum, Histogram, and ExponentialHistogram metrics
- Advanced batching and CloudWatch Logs constraints
- Enhanced error handling and validation
@mxiamxia mxiamxia force-pushed the pr1-emf-exporter-foundation branch from 5b6d407 to 1b34c6f Compare June 11, 2025 00:00
Copy link
Contributor

@thpierce thpierce left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did not review _create_emf_log, _send_log_event, export, or unit tests. Will try to take another crack at this this week.

@mxiamxia mxiamxia force-pushed the pr1-emf-exporter-foundation branch 3 times, most recently from f44f3be to 34a406c Compare June 13, 2025 00:44
@mxiamxia mxiamxia force-pushed the pr1-emf-exporter-foundation branch from 34a406c to 7de65e4 Compare June 13, 2025 00:50
@mxiamxia mxiamxia force-pushed the pr1-emf-exporter-foundation branch from 195e95a to a0da19f Compare June 14, 2025 00:45
Copy link
Contributor

@srprash srprash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did not look at the unit tests. Will do so in the next pass.

@srprash
Copy link
Contributor

srprash commented Jun 19, 2025

LGTM. Thanks for addressing the comments. :)

@mxiamxia mxiamxia merged commit f7cd7b6 into aws-observability:main Jun 19, 2025
13 checks passed
jj22ee added a commit to aws-observability/aws-otel-js-instrumentation that referenced this pull request Jul 1, 2025
)

*Issue #, if available:*
JS Equivalent of:
-
aws-observability/aws-otel-python-instrumentation#382
-
aws-observability/aws-otel-python-instrumentation#409
-
aws-observability/aws-otel-python-instrumentation#410

*Description of changes:*


By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants