feat: [Java] EMF Exporter Implementation #1209

liustve · 2025-09-19T22:32:49Z

Description of changes:
Java Version of these PRs:

Add CloudWatch EMF exporter foundation with basic Gauge support aws-otel-python-instrumentation#382
Extend CloudWatch EMF exporter with Sum and Histogram support aws-otel-python-instrumentation#409
Add CloudWatch EMF exporter integration aws-otel-python-instrumentation#410
Fix EMF exporter to generate EMF metrics with dimension empty aws-otel-python-instrumentation#434

This PR introduces the complete CloudWatch EMF (Embedded Metric Format) exporter implementation for sending OpenTelemetry metrics directly to CloudWatch without requiring a Collector or Agent.

In order to enable this exporter, users MUST set the following environment variables:

OTEL_METRICS_EXPORTER=awsemf -
OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=<log-group-name>,x-aws-log-stream=<log-stream-name>, x-aws-metric-namespace=<namespace>
AWS_REGION=<region> OR AWS_DEFAULT_REGION=<region>

This PR includes:

EMF MetricRecord translation for for unified representation of all OTel metric types with log creation with unit mapping from OpenTelemetry to CloudWatch-compatible units
Automatic log group and stream creation with retry logic
Supported CloudWatch Logs integration with batching and constraint handling (256KB event limit, 1MB request limit, timestamp limits)
Support for metric grouping by attributes and timestamps for EMF log generation
Support for Gauge, Sum, Histogram, and ExponentialHistogram metric types

TODO:

On the next PR, will integrate the Console EMF exporter into AWS Lambda environments to validate EMF log formatting and ensure consistent behavior in Lambda runtime: Support export OTel metrics to be EMF in starndard output aws-otel-python-instrumentation#437

Testing:

Added unit tests to validate EMF exporter configuration scenarios, including parameterized tests for both valid configurations (supporting AWS_REGION and AWS_DEFAULT_REGION) and invalid configurations (missing headers, wrong exporter type, missing region). The tests ensure the EMF exporter is correctly enabled only when all required environment variables are properly configured.
Manual end to end testing with the following environment variables to ensure the EMF logs show up:
- AWS_REGION=us-east-1
- OTEL_METRICS_EXPORTER=awsemf
- OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=test,x-aws-log-stream=default,x-aws-metric-namespace=testNamespace
- OTEL_RESOURCE_ATTRIBUTES=service.name=testService,aws.log.group.names=test,cloud.resource_id=agent-12345
- OTEL_LOGS_EXPORTER=none
- OTEL_TRACES_EXPORTER=none

Example EMF log emitted:

{
    "otel.resource.process.command_args": "[/Library/Java/JavaVirtualMachines/amazon-corretto-21.jdk/Contents/Home/bin/java, -javaagent:/Users/liustve/aws-otel-java-instrumentation/otelagent/build/libs/aws-opentelemetry-agent-2.18.0-SNAPSHOT.jar, -Dfile.encoding=UTF-8, -Dsun.stdout.encoding=UTF-8, -Dsun.stderr.encoding=UTF-8, -jar, /Users/liustve/aws-otel-java-instrumentation/sample-apps/springboot/build/libs/springboot-2.11.0-SNAPSHOT.jar]",
    "otel.resource.host.arch": "aarch64",
    "otel.resource.host.name": "7cf34dd812df",
    "otel.resource.service.instance.id": "a0399d3c-b856-43ae-b374-fe66dee41ce8",
    "otel.resource.aws.log.group.names": "test",
    "jvm.class.unloaded": 1,
    "otel.resource.service.name": "testSErvice",
    "_aws": {
        "CloudWatchMetrics": [
            {
                "Metrics": [
                    {
                        "Unit": "Count",
                        "Name": "jvm.cpu.count"
                    },
                    {
                        "Unit": "Count",
                        "Name": "jvm.class.loaded"
                    },
                    {
                        "Name": "jvm.cpu.recent_utilization"
                    },
                    {
                        "Unit": "Seconds",
                        "Name": "jvm.cpu.time"
                    },
                    {
                        "Unit": "Count",
                        "Name": "jvm.class.count"
                    },
                    {
                        "Unit": "Count",
                        "Name": "jvm.class.unloaded"
                    }
                ],
                "Namespace": "testNamespace"
            }
        ],
        "Timestamp": 1758761023497
    },
    "otel.resource.cloud.resource_id": "agent-12345",
    "jvm.class.count": 14264,
    "Version": "1",
    "otel.resource.process.pid": "46822",
    "otel.resource.os.description": "Mac OS X 15.6.1",
    "otel.resource.telemetry.distro.name": "opentelemetry-java-instrumentation",
    "otel.resource.os.type": "darwin",
    "otel.resource.telemetry.sdk.name": "opentelemetry",
    "otel.resource.telemetry.distro.version": "2.18.0-aws-SNAPSHOT",
    "otel.resource.process.runtime.description": "Amazon.com Inc. OpenJDK 64-Bit Server VM 21.0.8+9-LTS",
    "otel.resource.process.runtime.version": "21.0.8+9-LTS",
    "jvm.cpu.recent_utilization": 0,
    "otel.resource.process.executable.path": "/Library/Java/JavaVirtualMachines/amazon-corretto-21.jdk/Contents/Home/bin/java",
    "otel.resource.telemetry.sdk.version": "1.52.0",
    "jvm.cpu.count": 12,
    "jvm.class.loaded": 14265,
    "otel.resource.process.runtime.name": "OpenJDK Runtime Environment",
    "otel.resource.telemetry.sdk.language": "java",
    "jvm.cpu.time": 7.756965
}

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…ws-otel-java-instrumentation into emf-exporter

wangzlei · 2025-09-27T02:22:58Z

...ftware/amazon/opentelemetry/javaagent/providers/AwsApplicationSignalsCustomizerProvider.java

+      Optional<String> awsRegion = getAwsRegionFromConfig(configProps);
+
+      if (awsRegion.isPresent()) {
+        if (headers.containsKey(AWS_OTLP_LOGS_GROUP_HEADER)


Does customer have to set all 3 headers?

It looks like from Python implementation they don't need namespace header:
https://github.com/aws-observability/aws-otel-python-instrumentation/blob/main/aws-opentelemetry-distro/src/amazon/opentelemetry/distro/aws_opentelemetry_configurator.py#L127

Removed validation for namespace header.

…eader, and consolidated unit tests

wangzlei · 2025-12-04T00:33:06Z

...java/software/amazon/opentelemetry/javaagent/providers/AwsApplicationSignalsConfigUtils.java

+   * @return Optional containing string with "awsemf" removed if the original OTEL_METRICS_EXPORTER
+   *     contains "awsemf", otherwise empty Optional if "awsemf" is not found
+   */
+  static Optional<String> removeEmfExporterIfEnabled(ConfigProperties configProps) {


think about simplify the logic, improve readability

added some comments to clarify the method logic

mxiamxia · 2025-12-04T19:52:16Z

...re/amazon/opentelemetry/javaagent/providers/exporter/aws/metrics/common/BaseEmfExporter.java

+
+      for (MetricData metric : metricsData) {
+        Data<? extends PointData> metricData = metric.getData();
+        if (metricData == null || metricData.getPoints().isEmpty()) {


can metricData.getPoints() be null and the for loop exits with the NPE exception?

Based on their interface Java doc, no:
https://github.com/open-telemetry/opentelemetry-java/blob/ef17aa5e2bffd3cbee4fac92b43400cdcd0ce60f/sdk/metrics/src/main/java/io/opentelemetry/sdk/metrics/data/Data.java#L17-L23

https://github.com/open-telemetry/opentelemetry-java/blob/ef17aa5e2bffd3cbee4fac92b43400cdcd0ce60f/sdk/metrics/src/main/java/io/opentelemetry/sdk/metrics/data/MetricData.java#L1

mxiamxia · 2025-12-04T21:43:19Z

...re/amazon/opentelemetry/javaagent/providers/exporter/aws/metrics/common/BaseEmfExporter.java

+          MetricRecord firstRecord = records.get(0);
+          // Get resource from first metric in the collection
+          MetricData firstMetric = metricsData.iterator().next();


Can you escalate this into the createEmfLog method as part of EMF log creation logic?

Fixed a bug so that the grouping logic to also based on resource attributes and scope

initial implmentation

bcf5eb2

liustve requested a review from a team as a code owner September 19, 2025 22:32

liustve added 14 commits September 19, 2025 15:50

formatting

f6ba80a

Merge branch 'main' into emf-exporter

0b2c14c

add emf exporter to changelog

cd1475d

add emf exporter to changelog

5276407

merge main

3ec3aa5

modify message

7cd68df

add further unit tests

e6a5283

Merge branch 'main' into emf-exporter

8067773

refactoring + more unit tests for all exporters

b686dc1

Merge branch 'main' into emf-exporter

af3b570

add unit tests for configuring emf exporter in adot

48e9453

Merge branch 'main' into emf-exporter

0e3e77a

Merge branch 'emf-exporter' of https://github.com/aws-observability/a…

48ecda1

…ws-otel-java-instrumentation into emf-exporter

delete duplicate customizer test

fcbd7a0

liustve changed the title ~~[WIP][Java] EMF Exporter Implementation~~ feat: [Java] EMF Exporter Implementation Sep 25, 2025

add further emf + auto instrumentation unit tests and better java docs

746e989

liustve force-pushed the emf-exporter branch from 2cf0524 to 746e989 Compare September 25, 2025 01:36

wangzlei reviewed Sep 27, 2025

View reviewed changes

wangzlei previously approved these changes Sep 27, 2025

View reviewed changes

move emf exporter files, removed need to set x-aws-metric-namespace h…

4236220

…eader, and consolidated unit tests

liustve dismissed wangzlei’s stale review via 4236220 September 27, 2025 17:05

liustve added 7 commits September 27, 2025 13:12

update CHANGELOG.md

b7d0506

plug emf and compact console log exporter to lambda

82e0b46

Merge branch 'main' into emf-exporter

369501a

Merge branch 'main' into emf-exporter

84dbe92

Merge branch 'main' into emf-exporter

216841e

Merge branch 'main' into emf-exporter

d1ccad6

Merge branch 'main' into emf-exporter

a95a598

liustve added 4 commits October 29, 2025 16:10

Merge branch 'main' into emf-exporter

178085c

Merge branch 'main' into emf-exporter

64e6202

Merge branch 'main' into emf-exporter

8923525

Merge branch 'main' into emf-exporter

05ad879

wangzlei reviewed Dec 4, 2025

View reviewed changes

mxiamxia reviewed Dec 4, 2025

View reviewed changes

liustve added 2 commits December 4, 2025 15:50

fix emf metric grouping logic

07a0a80

add comment for removeEmfIfEnabled logic

12a070a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: [Java] EMF Exporter Implementation #1209

feat: [Java] EMF Exporter Implementation #1209

Uh oh!

liustve commented Sep 19, 2025 •

edited

Loading

Uh oh!

wangzlei Sep 27, 2025

Uh oh!

liustve Sep 27, 2025

Uh oh!

liustve Sep 27, 2025

Uh oh!

wangzlei Dec 4, 2025

Uh oh!

liustve Dec 5, 2025

Uh oh!

mxiamxia Dec 4, 2025

Uh oh!

liustve Dec 4, 2025

Uh oh!

mxiamxia Dec 4, 2025

Uh oh!

liustve Dec 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat: [Java] EMF Exporter Implementation #1209

Are you sure you want to change the base?

feat: [Java] EMF Exporter Implementation #1209

Uh oh!

Conversation

liustve commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

liustve commented Sep 19, 2025 •

edited

Loading