Skip to content

Commit fb173f2

Browse files
MiqueasherMiqueasher
authored
[Python] Custom Metrics E2E Test (#471)
1 parent f0b1f6e commit fb173f2

File tree

11 files changed

+238
-10
lines changed

11 files changed

+238
-10
lines changed

‎.github/workflows/java-ec2-adot-sigv4-test.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ permissions:
3434
contents: read
3535

3636
env:
37-
E2E_TEST_AWS_REGION: 'us-west-2' # Test uses us-west-2 in the us-east-1 accoun
37+
E2E_TEST_AWS_REGION: 'us-west-2' # Test uses us-west-2 in the us-east-1 account
3838
CALLER_WORKFLOW_NAME: ${{ inputs.caller-workflow-name }}
3939
METRIC_NAMESPACE: ApplicationSignals
4040
JAVA_VERSION: ${{ inputs.java-version }}

‎.github/workflows/python-ec2-default-test.yml‎

Lines changed: 22 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -172,7 +172,7 @@ jobs:
172172
-var="language_version=${{ env.PYTHON_VERSION }}" \
173173
-var="cpu_architecture=${{ env.CPU_ARCHITECTURE }}" \
174174
|| deployment_failed=$?
175-
175+
176176
if [ $deployment_failed -eq 1 ]; then
177177
echo "Terraform deployment was unsuccessful. Will attempt to retry deployment."
178178
fi
@@ -252,9 +252,28 @@ jobs:
252252
--instance-id ${{ env.MAIN_SERVICE_INSTANCE_ID }}
253253
--rollup'
254254

255+
# Creating a separate validation for custom metrics
256+
- name: Validate custom metrics
257+
id: cwagent-metric-validation
258+
if: (success() || steps.log-validation.outcome == 'failure' || steps.metric-validation.outcome == 'failure') && !cancelled()
259+
run: ./gradlew validator:run --args='-c python/ec2/default/custom-metric-validation.yml
260+
--testing-id ${{ env.TESTING_ID }}
261+
--account-id ${{ env.ACCOUNT_ID }}
262+
--endpoint http://${{ env.MAIN_SERVICE_ENDPOINT }}
263+
--remote-service-deployment-name ${{ env.REMOTE_SERVICE_IP }}:8001
264+
--region ${{ env.E2E_TEST_AWS_REGION }}
265+
--metric-namespace CWAgent
266+
--log-group ${{ env.LOG_GROUP_NAME }}
267+
--service-name python-sample-application-${{ env.TESTING_ID }}
268+
--remote-service-name python-sample-remote-application-${{ env.TESTING_ID }}
269+
--query-string ip=${{ env.REMOTE_SERVICE_IP }}
270+
--instance-ami ${{ env.EC2_INSTANCE_AMI }}
271+
--instance-id ${{ env.MAIN_SERVICE_INSTANCE_ID }}
272+
--rollup'
273+
255274
- name: Validate generated traces
256275
id: trace-validation
257-
if: (success() || steps.log-validation.outcome == 'failure' || steps.metric-validation.outcome == 'failure') && !cancelled()
276+
if: (success() || steps.log-validation.outcome == 'failure' || steps.metric-validation.outcome == 'failure' || steps.cwagent-metric-validation.outcome == 'failure') && !cancelled()
258277
run: ./gradlew validator:run --args='-c python/ec2/default/trace-validation.yml
259278
--testing-id ${{ env.TESTING_ID }}
260279
--endpoint http://${{ env.MAIN_SERVICE_ENDPOINT }}
@@ -281,7 +300,7 @@ jobs:
281300
if: always()
282301
id: validation-result
283302
run: |
284-
if [ "${{ steps.log-validation.outcome }}" = "success" ] && [ "${{ steps.metric-validation.outcome }}" = "success" ] && [ "${{ steps.trace-validation.outcome }}" = "success" ]; then
303+
if [ "${{ steps.log-validation.outcome }}" = "success" ] && [ "${{ steps.cwagent-metric-validation.outcome }}" = "success" ] && [ "${{ steps.metric-validation.outcome }}" = "success" ] && [ "${{ steps.trace-validation.outcome }}" = "success" ]; then
285304
echo "validation-result=success" >> $GITHUB_OUTPUT
286305
else
287306
echo "validation-result=failure" >> $GITHUB_OUTPUT

‎sample-apps/python/django_frontend_service/frontend_service_app/views.py‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,17 +5,24 @@
55
import base64
66
import threading
77
import time
8-
8+
import random
99
import boto3
1010
import pymysql
1111
import requests
1212
import schedule
1313
from django.http import HttpResponse, JsonResponse
14-
from opentelemetry import trace
14+
from opentelemetry import trace, metrics
1515
from opentelemetry.trace.span import format_trace_id
1616

1717
logger = logging.getLogger(__name__)
1818

19+
#python equivalent of Meter meter = GlobalOpenTelemetry.getMeter("myMeter"); for custom metrics
20+
meter = metrics.get_meter("myMeter")
21+
agent_based_counter = meter.create_counter("agent_based_counter", unit="1", description="agent export counter")
22+
agent_based_histogram = meter.create_histogram("agent_based_histogram", description="agent export histogram")
23+
agent_based_gauge = meter.create_up_down_counter("agent_based_gauge", unit="1", description="agent export gauge")
24+
25+
1926
should_send_local_root_client_call = False
2027
lock = threading.Lock()
2128
def run_local_root_client_call_recurring_service():
@@ -50,6 +57,12 @@ def healthcheck(request):
5057
return HttpResponse("healthcheck")
5158

5259
def aws_sdk_call(request):
60+
61+
# Increment counter/histogram
62+
agent_based_counter.add(1, {"Operation": "counter"})
63+
agent_based_histogram.record(random.randint(100, 1000), {"Operation": "histogram"})
64+
agent_based_gauge.add(random.randint(-10, 10), {"Operation": "gauge"})
65+
5366
bucket_name = "e2e-test-bucket-name"
5467

5568
# Add a unique test ID to bucketname to associate buckets to specific test runs

‎terraform/python/ec2/default/README.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# CLOUDWATCH AGENT CONFIG.
2+
This cloudwatch-agent.json file is a version of the amazon-cloudwatch-agent fitted to the needs of the python sample app using custom metrics. Shown in public docs: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AppSignals-CustomMetrics.html#AppSignals-CustomMetrics-OpenTelemetry
3+
4+
It handles the custom metrics through the addition of the otlp ports:
5+
"otlp": {
6+
"grpc_endpoint": "0.0.0.0:4317",
7+
"http_endpoint": "0.0.0.0:4318"
8+
}

‎terraform/python/ec2/default/amazon-cloudwatch-agent.json‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,11 @@
1010
},
1111
"logs": {
1212
"metrics_collected": {
13-
"application_signals": {}
13+
"application_signals": {},
14+
"otlp": {
15+
"grpc_endpoint": "0.0.0.0:4317",
16+
"http_endpoint": "0.0.0.0:4318"
17+
}
1418
}
1519
}
1620
}

‎terraform/python/ec2/default/main.tf‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -170,15 +170,18 @@ resource "null_resource" "main_service_setup" {
170170
export DJANGO_SETTINGS_MODULE="django_frontend_service.settings"
171171
export OTEL_PYTHON_DISTRO="aws_distro"
172172
export OTEL_PYTHON_CONFIGURATOR="aws_configurator"
173-
export OTEL_METRICS_EXPORTER=none
173+
export OTEL_METRICS_EXPORTER=otlp
174174
export OTEL_TRACES_EXPORTER=otlp
175175
export OTEL_AWS_APPLICATION_SIGNALS_ENABLED=true
176176
export OTEL_AWS_APPLICATION_SIGNALS_EXPORTER_ENDPOINT=http://localhost:4315
177177
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4315
178178
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=grpc
179179
export OTEL_EXPORTER_OTLP_METRICS_PROTOCOL=grpc
180-
export OTEL_SERVICE_NAME=python-sample-application-${var.test_id}
180+
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=localhost:4317
181+
export OTEL_EXPORTER_OTLP_METRICS_INSECURE=true
181182
export OTEL_TRACES_SAMPLER=always_on
183+
export OTEL_RESOURCE_ATTRIBUTES="service.name=python-sample-application-${var.test_id},deployment.environment.name=ec2:default"
184+
export AWS_REGION='${var.aws_region}'
182185
python${var.language_version} manage.py migrate
183186
nohup opentelemetry-instrument python${var.language_version} manage.py runserver 0.0.0.0:8000 --noreload &
184187
@@ -276,7 +279,7 @@ resource "null_resource" "remote_service_setup" {
276279
277280
# Copy in CW Agent configuration
278281
agent_config='${replace(replace(file("./amazon-cloudwatch-agent.json"), "/\\s+/", ""), "$REGION", var.aws_region)}'
279-
echo $agent_config > amazon-cloudwatch-agent.json
282+
echo "$agent_config" > amazon-cloudwatch-agent.json
280283
281284
# Get and run CW agent rpm
282285
${var.get_cw_agent_rpm_command}

‎validator/src/main/java/com/amazon/aoc/fileconfigs/PredefinedExpectedTemplate.java‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -210,6 +210,9 @@ public enum PredefinedExpectedTemplate implements FileConfig {
210210
PYTHON_EC2_DEFAULT_CLIENT_CALL_METRIC("/expected-data-template/python/ec2/default/client-call-metric.mustache"),
211211
PYTHON_EC2_DEFAULT_CLIENT_CALL_TRACE("/expected-data-template/python/ec2/default/client-call-trace.mustache"),
212212

213+
/** Python EC2 Default Custom Metrics Test Case Validations */
214+
PYTHON_EC2_DEFAULT_AWS_OTEL_CUSTOM_METRIC("/expected-data-template/python/ec2/default/aws-otel-custom-metrics.mustache"),
215+
213216
/** Python EC2 Asg Test Case Validations */
214217
PYTHON_EC2_ASG_OUTGOING_HTTP_CALL_LOG("/expected-data-template/python/ec2/asg/outgoing-http-call-log.mustache"),
215218
PYTHON_EC2_ASG_OUTGOING_HTTP_CALL_METRIC("/expected-data-template/python/ec2/asg/outgoing-http-call-metric.mustache"),

‎validator/src/main/java/com/amazon/aoc/services/CloudWatchService.java‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,9 @@ public class CloudWatchService {
3636
public static final String SERVICE_DIMENSION = "Service";
3737
public static final String REMOTE_SERVICE_DIMENSION = "RemoteService";
3838
public static final String REMOTE_TARGET_DIMENSION = "RemoteTarget";
39+
public static final String CUSTOM_SERVICE_DIMENSION = "service.name";
40+
public static final String CUSTOM_ENVIRONMENT_DIMENSION = "deployment.environment.name";
41+
3942

4043
private static final int MAX_QUERY_PERIOD = 60;
4144
private static final String REQUESTER = "integrationTest";

‎validator/src/main/java/com/amazon/aoc/validators/CWMetricValidator.java‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,16 @@ public void validate() throws Exception {
133133
remoteTargetNames,
134134
expectedMetricList,
135135
actualMetricList);
136+
addMetrics(
137+
CloudWatchService.CUSTOM_SERVICE_DIMENSION,
138+
serviceNames,
139+
expectedMetricList,
140+
actualMetricList);
141+
addMetrics(
142+
CloudWatchService.CUSTOM_ENVIRONMENT_DIMENSION,
143+
Lists.newArrayList("ec2:default"),
144+
expectedMetricList,
145+
actualMetricList);
136146

137147
// remove the skip dimensions
138148
log.info("dimensions to be skipped in validation: {}", skippedDimensionNameList);

‎validator/src/main/resources/expected-data-template/python/ec2/default/aws-otel-custom-metrics.mustache‎

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
# OpenTelemetry Custom Metrics Validation Templates - AWS SDK Call Only
2+
# ANY_VALUE defines a string to = 'ANY_VALUE' to pass validation testing
3+
# Custom export templates
4+
-
5+
metricName: agent_based_counter
6+
namespace: {{metricNamespace}}
7+
dimensions:
8+
-
9+
name: deployment.environment.name
10+
value: ec2:default
11+
-
12+
name: aws.local.service
13+
value: {{serviceName}}
14+
-
15+
name: cloud.region
16+
value: {{region}}
17+
-
18+
name: service.name
19+
value: {{serviceName}}
20+
-
21+
name: Operation
22+
value: counter
23+
-
24+
name: host.type
25+
value: ANY_VALUE
26+
-
27+
name: cloud.availability_zone
28+
value: ANY_VALUE
29+
-
30+
name: telemetry.sdk.name
31+
value: opentelemetry
32+
-
33+
name: telemetry.sdk.language
34+
value: python
35+
-
36+
name: cloud.provider
37+
value: aws
38+
-
39+
name: cloud.account.id
40+
value: {{accountId}}
41+
-
42+
name: host.name
43+
value: ANY_VALUE
44+
-
45+
name: telemetry.sdk.version
46+
value: ANY_VALUE
47+
-
48+
name: host.id
49+
value: ANY_VALUE
50+
-
51+
name: telemetry.auto.version
52+
value: ANY_VALUE
53+
-
54+
name: cloud.platform
55+
value: aws_ec2
56+
-
57+
metricName: agent_based_histogram
58+
namespace: {{metricNamespace}}
59+
dimensions:
60+
-
61+
name: deployment.environment.name
62+
value: ec2:default
63+
-
64+
name: aws.local.service
65+
value: {{serviceName}}
66+
-
67+
name: cloud.region
68+
value: {{region}}
69+
-
70+
name: service.name
71+
value: {{serviceName}}
72+
-
73+
name: Operation
74+
value: histogram
75+
-
76+
name: host.type
77+
value: ANY_VALUE
78+
-
79+
name: cloud.availability_zone
80+
value: ANY_VALUE
81+
-
82+
name: telemetry.sdk.name
83+
value: opentelemetry
84+
-
85+
name: telemetry.sdk.language
86+
value: python
87+
-
88+
name: cloud.provider
89+
value: aws
90+
-
91+
name: cloud.account.id
92+
value: {{accountId}}
93+
-
94+
name: host.name
95+
value: ANY_VALUE
96+
-
97+
name: telemetry.sdk.version
98+
value: ANY_VALUE
99+
-
100+
name: host.id
101+
value: ANY_VALUE
102+
-
103+
name: telemetry.auto.version
104+
value: ANY_VALUE
105+
-
106+
name: cloud.platform
107+
value: aws_ec2
108+
-
109+
metricName: agent_based_gauge
110+
namespace: {{metricNamespace}}
111+
dimensions:
112+
-
113+
name: deployment.environment.name
114+
value: ec2:default
115+
-
116+
name: aws.local.service
117+
value: {{serviceName}}
118+
-
119+
name: cloud.region
120+
value: {{region}}
121+
-
122+
name: service.name
123+
value: {{serviceName}}
124+
-
125+
name: Operation
126+
value: gauge
127+
-
128+
name: host.type
129+
value: ANY_VALUE
130+
-
131+
name: cloud.availability_zone
132+
value: ANY_VALUE
133+
-
134+
name: telemetry.sdk.name
135+
value: opentelemetry
136+
-
137+
name: telemetry.sdk.language
138+
value: python
139+
-
140+
name: cloud.provider
141+
value: aws
142+
-
143+
name: cloud.account.id
144+
value: {{accountId}}
145+
-
146+
name: host.name
147+
value: ANY_VALUE
148+
-
149+
name: telemetry.sdk.version
150+
value: ANY_VALUE
151+
-
152+
name: host.id
153+
value: ANY_VALUE
154+
-
155+
name: telemetry.auto.version
156+
value: ANY_VALUE
157+
-
158+
name: cloud.platform
159+
value: aws_ec2

0 commit comments

Comments
 (0)