[Python] Custom Metrics E2E Test #471

Miqueasher · 2025-10-16T21:28:18Z

This PR adds custom metrics to the existing sample app to verify public documentation is correct and functionality continues to work as changes happen in the future (Regression testing). Specifically for OpenTelemetry metrics using the CloudWatch Agent.

The changes made in this PR are:

2 New imports to views.py (sample app's frontend service for the 'main' instance)
Global meter, and histogram/counter added to views.py from OTEL SDK Metrics page
Update to environment variables in main.tf
Update to jobs & env sections in python-ec2-default-test
Update from aws region us-west-2 to us-east-1
Update to Cloudwatch agent config to include otlp ports
Update to variables.tf to include custom metrics enabled
For loop expansion for easier Debugging logs (used in testing & for future)
New Validator file to call updated predefined template file
Update to PredefinedExpectedTemplate to include new mustache file
New mustache file for validation of custom metrics

IMPORTANT
Updates to views.py effect 'python-sample-app.zip which exists in S3 bucket. Proper Zip protocol must be followed for it properly deploy and run. If not it will fail.

Rollback procedure:
A git revert to the most recent SHA (4ca5d60) will be easy to revert back to if necessary.

Passing test runs:
Test 1
Test 2

References:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AppSignals-CustomMetrics.html#AppSignals-CustomMetrics-OpenTelemetry
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-OpenTelemetry-metrics.html
https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#exporter-selection
https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/#otel_exporter_otlp_metrics_endpoint
https://opentelemetry.io/docs/specs/otel/metrics/sdk/

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…ervability/aws-application-signals-test-framework into otel-custom-metrics-test

This reverts commit 72c88c5.

This reverts commit e2e01a8.

majanjua-amzn · 2025-10-16T21:29:18Z

.github/workflows/dotnet-ec2-adot-sigv4-test.yml


 env:
-  E2E_TEST_AWS_REGION: 'us-west-2'
+  E2E_TEST_AWS_REGION: 'us-east-1'


Don't change stuff for unrelated files

Will be updated in the next push

majanjua-amzn · 2025-10-16T21:29:23Z

.github/workflows/java-ec2-adaptive-sampling-test.yml


 env:
-  E2E_TEST_AWS_REGION: us-west-2
+  E2E_TEST_AWS_REGION: us-east-1


Will be updated in the next push

majanjua-amzn · 2025-10-16T21:29:30Z

.github/workflows/java-ec2-adot-sigv4-test.yml


 env:
-  E2E_TEST_AWS_REGION: 'us-west-2' # Test uses us-west-2 in the us-east-1 accoun
+  E2E_TEST_AWS_REGION: 'us-east-1' # Test uses us-east-1 in the us-east-1 account


Will be updated in the next push

majanjua-amzn · 2025-10-16T21:29:36Z

.github/workflows/node-ec2-adot-sigv4-test.yml


 env:
-  E2E_TEST_AWS_REGION: 'us-west-2'
+  E2E_TEST_AWS_REGION: 'us-east-1'


Will be updated in the next push

majanjua-amzn · 2025-10-16T21:29:42Z

.github/workflows/python-ec2-adot-sigv4-test.yml


 env:
-  E2E_TEST_AWS_REGION: 'us-west-2'
+  E2E_TEST_AWS_REGION: 'us-east-1'


Will be updated in the next push

majanjua-amzn · 2025-10-16T21:34:46Z

terraform/python/ec2/default/main.tf

      # Copy in CW Agent configuration
      agent_config='${replace(replace(file("./amazon-cloudwatch-agent.json"), "/\\s+/", ""), "$REGION", var.aws_region)}'
-      echo $agent_config > amazon-cloudwatch-agent.json
+      echo "$agent_config" > amazon-cloudwatch-agent.json


agent_config goes through the replace function and the quotes are just to prevent word splitting from breaking it.

majanjua-amzn · 2025-10-16T21:35:13Z

validator/src/main/java/com/amazon/aoc/fileconfigs/PredefinedExpectedTemplate.java

+/** Python EC2 Default Custom Metrics Test Case Validations */
+  PYTHON_EC2_DEFAULT_AWS_OTEL_CUSTOM_METRIC("/expected-data-template/python/ec2/default/aws-otel-custom-metrics.mustache"),


Suggested change

/** Python EC2 Default Custom Metrics Test Case Validations */

PYTHON_EC2_DEFAULT_AWS_OTEL_CUSTOM_METRIC("/expected-data-template/python/ec2/default/aws-otel-custom-metrics.mustache"),

/** Python EC2 Default Custom Metrics Test Case Validations */

PYTHON_EC2_DEFAULT_AWS_OTEL_CUSTOM_METRIC("/expected-data-template/python/ec2/default/aws-otel-custom-metrics.mustache"),

Will be updated in the next push

majanjua-amzn · 2025-10-16T21:36:26Z

...rc/main/resources/expected-data-template/python/ec2/default/aws-otel-custom-metrics.mustache

This file should be moved to .../python/ec2/custom-metrics since it's a unique test case, even though it's using the same workflow file. Same with the other file added under .../resources

Will be updated in the next push

majanjua-amzn · 2025-10-16T21:37:42Z

...rc/main/resources/expected-data-template/python/ec2/default/aws-otel-custom-metrics.mustache

+      value: {{serviceName}}
+    -
+      name: cloud.account.id
+      value: ANY_VALUE


We know the values in some cases, so we should put it in. In this case, we know the account ID so we should be able to validate it fully.

Will be updated in the next push

majanjua-amzn · 2025-10-16T21:38:11Z

...rc/main/resources/expected-data-template/python/ec2/default/aws-otel-custom-metrics.mustache

+      value: opentelemetry
+    -
+      name: telemetry.sdk.version
+      value: 1.33.1


This value can change in the future, don't hardcode it.

Will be updated in the next push

Miqueasher added 30 commits October 7, 2025 12:52

Create test.yml

f02dcb3

updating custom metrics test branch

383791c

Merge branch 'otel-custom-metrics-test' of https://github.com/aws-obs…

c74f03f

…ervability/aws-application-signals-test-framework into otel-custom-metrics-test

Update test.yml

d89c43b

fixing variables.tf typo

3e1be9b

Merge branch 'otel-custom-metrics-test' of https://github.com/aws-obs…

f5b6b0e

…ervability/aws-application-signals-test-framework into otel-custom-metrics-test

updated bash syntax error

c5f28fd

updating tf files

3455277

updating syntax error

ddd2993

updating main service to connect to local host only if CM is enabled

446bd08

updating main.tf

75c47a2

updating default instance

0c75867

setting custom metrics default to true

e285993

fixing cw agent

dd9e778

making custom metrics conditional

a3fedbc

setting custom metrics to false

81500c6

separating metric validation files

72c88c5

Revert "separating metric validation files"

b0e450f

This reverts commit 72c88c5.

adding debugger for sql pws

0e3aff2

adding dummy MYSQL data to prevent KeyError

d070aab

setting custom metrics to true

67329dc

updating zip,validation templates, variables, and test region

59b9f12

fixing typo

284c65e

updating otlp-proto dependency version

761a7ed

updating grpc dependency and sample app

02e58d5

fixing typo in main.tf

bc4a37a

testing new grpc/http endpoints

f3dbdf4

updating cwa

3807ca9

removing cwagent and updated views

2883c2f

updating env variables

c826fa0

Miqueasher added 28 commits October 14, 2025 09:45

reverting back to last passing commit

92e8620

adding telemetry.status to resource

effd5d6

updating namespace

ce893f2

removing custom export pipeline

c3b25d0

updating views

f83f427

updating traffic generator to hit custom endpoints

79f9a09

updating according to public docs.

29c32d6

testing new cwa config

7153eaf

reverting cwagent and adding logging to metric validator

1c5f429

adding more log debuggers

710b4c3

adjusting otlp metrics end point for test

0a29d7e

making grpc unecrypted

e2e01a8

Revert "making grpc unecrypted"

5d84db6

This reverts commit e2e01a8.

attempt to revert back to test id-18514948781-1549-2067

9a649bc

adding validator for CWAgent

a4ba51d

testing out single validator file

dd6c12c

reverting back to separate validation files

f0f890d

updating dimension value of custom_export_counter

b6bab9e

updating validation dimensions

34087f4

adding debug loggers

1f74499

updating validation template

db6fda5

updating service.name attribute to Service

ea96f6e

updating service.name attribute to Service

f360d97

updating val templates with temp values for testing

523e41b

removing debuggers/loggers

9149f84

removing dead code

567e84c

deleting 'delete-me' file and removing test.yml file

fc77459

Merge branch 'main' into otel-custom-metrics-test

e3034a5

majanjua-amzn reviewed Oct 16, 2025

View reviewed changes

majanjua-amzn changed the title ~~Custom metrics test~~ [Python] Custom Metrics E2E Test Oct 16, 2025

		/** Python EC2 Default Custom Metrics Test Case Validations */
		PYTHON_EC2_DEFAULT_AWS_OTEL_CUSTOM_METRIC("/expected-data-template/python/ec2/default/aws-otel-custom-metrics.mustache"),

[Python] Custom Metrics E2E Test #471

Are you sure you want to change the base?

[Python] Custom Metrics E2E Test #471

Uh oh!

Conversation

Miqueasher commented Oct 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants