Skip to content

Commit 124c0ca

Browse files
khanhntdkhanhntd
authored
Support testcase for ec2 based on ecs when using otlp to send metrics to amprometheus xray cloudwatch (#412)
* add ecs export trace metrics to amp, xray and cloudwatch * add terraform validation for otlp when sending metrics to cloudwatch,amp,xray * delete some debug lon info * change ecs validation to spark sdk ecs expected trace and add ecs validation later * change default private repo to public repo of adot collector * support trace for aws container and also oltp * change back some values for debug * change back some values for debug again * fix nit pick * change back to use private repo image to avoid CI failing * change back to use private repo image to avoid CI failing * change back to normal format * change back to normal format * change back to normal format * change back to normal format * change back to normal format * change back pipielines name for test case * finalize the unit test with cw,amp,xray Co-authored-by: Ameen [email protected] * change back to private repo to avoid failing Co-authored-by: Ameen [email protected]
1 parent 75d04ae commit 124c0ca

File tree

4 files changed

+170
-0
lines changed

4 files changed

+170
-0
lines changed

‎terraform/ecs/main.tf‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -328,6 +328,7 @@ module "validator" {
328328
account_id = data.aws_caller_identity.current.account_id
329329

330330
ecs_context_json = jsonencode({
331+
ecsClusterName : module.ecs_cluster.cluster_name
331332
ecsClusterArn : module.ecs_cluster.cluster_arn
332333
ecsTaskDefFamily : aws_ecs_task_definition.aoc[0].family
333334
ecsTaskDefVersion : aws_ecs_task_definition.aoc[0].revision

‎terraform/testcases/otlp_grpc_exporter_cw_amp_xray_ecs/otconfig.tpl‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
extensions:
2+
pprof:
3+
endpoint: 0.0.0.0:1777
4+
receivers:
5+
awsecscontainermetrics:
6+
otlp:
7+
protocols:
8+
grpc:
9+
endpoint: 0.0.0.0:${grpc_port}
10+
11+
processors:
12+
batch:
13+
send_batch_size: 50
14+
15+
resourcedetection:
16+
detectors: [ env, ecs ]
17+
timeout: 2s
18+
override: false
19+
20+
#Only include some metrics relating to the container's metrics
21+
#Documentation: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor
22+
filter:
23+
metrics:
24+
include:
25+
match_type: strict
26+
metric_names:
27+
- ecs.task.memory.reserved
28+
- ecs.task.memory.utilized
29+
- ecs.task.cpu.reserved
30+
- ecs.task.cpu.utilized
31+
- ecs.task.network.rate.rx
32+
- ecs.task.network.rate.tx
33+
- ecs.task.storage.read_bytes
34+
- ecs.task.storage.write_bytes
35+
36+
#Update metrics' name to be equal to the expected container metrics in mustache
37+
#Expected metric: https://github.com/aws-observability/aws-otel-test-framework/blob/terraform/validator/src/main/resources/expected-data-template/ecsContainerExpectedMetric.mustache
38+
metricstransform:
39+
transforms:
40+
- metric_name: ecs.task.memory.reserved
41+
action: update
42+
new_name: ecs.task.memory.reserved_${testing_id}
43+
- metric_name: ecs.task.memory.utilized
44+
action: update
45+
new_name: ecs.task.memory.utilized_${testing_id}
46+
- metric_name: ecs.task.cpu.reserved
47+
action: update
48+
new_name: ecs.task.cpu.reserved_${testing_id}
49+
- metric_name: ecs.task.cpu.utilized
50+
action: update
51+
new_name: ecs.task.cpu.utilized_${testing_id}
52+
- metric_name: ecs.task.network.rate.rx
53+
action: update
54+
new_name: ecs.task.network.rate.rx_${testing_id}
55+
- metric_name: ecs.task.network.rate.tx
56+
action: update
57+
new_name: ecs.task.network.rate.tx_${testing_id}
58+
- metric_name: ecs.task.storage.read_bytes
59+
action: update
60+
new_name: ecs.task.storage.read_bytes_${testing_id}
61+
- metric_name: ecs.task.storage.write_bytes
62+
action: update
63+
new_name: ecs.task.storage.write_bytes_${testing_id}
64+
#Delete some metrics' dimension exporting to the cloudwatch or amp
65+
#Documentation: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourceprocessor
66+
resource:
67+
attributes:
68+
- key: aws.ecs.docker.name
69+
action: delete
70+
- key: aws.ecs.task.pull_started_at
71+
action: delete
72+
- key: aws.ecs.task.pull_stopped_at
73+
action: delete
74+
- key: cloud.zone
75+
action: delete
76+
- key: aws.ecs.task.launch_type
77+
action: delete
78+
- key: cloud.region
79+
action: delete
80+
- key: cloud.account.id
81+
action: delete
82+
- key: container.id
83+
action: delete
84+
- key: container.name
85+
action: delete
86+
- key: container.image.name
87+
action: delete
88+
- key: aws.ecs.container.image.id
89+
action: delete
90+
- key: aws.ecs.container.exit_code
91+
action: delete
92+
- key: aws.ecs.container.created_at
93+
action: delete
94+
- key: aws.ecs.container.started_at
95+
action: delete
96+
- key: aws.ecs.container.finished_at
97+
action: delete
98+
- key: container.image.tag
99+
action: delete
100+
- key: aws.ecs.container.know_status
101+
action: delete
102+
- key: aws.ecs.task.known_status
103+
action: delete
104+
105+
exporters:
106+
awsemf:
107+
namespace: '${otel_service_namespace}/${otel_service_name}'
108+
region: '${region}'
109+
resource_to_telemetry_conversion:
110+
enabled: true
111+
awsprometheusremotewrite:
112+
endpoint: https://aps-workspaces.us-west-2.amazonaws.com/workspaces/ws-cd278045-1d6e-4550-a207-c5046a1b40b8/api/v1/remote_write
113+
resource_to_telemetry_conversion:
114+
enabled: true
115+
aws_auth:
116+
region: ${region}
117+
service: "aps"
118+
awsxray:
119+
local_mode: true
120+
region: '${region}'
121+
logging:
122+
loglevel: debug
123+
124+
service:
125+
pipelines:
126+
metrics/container/cw:
127+
receivers: [awsecscontainermetrics]
128+
processors: [ filter, metricstransform, resource, batch]
129+
exporters: [awsemf,logging]
130+
metrics/container/amp:
131+
receivers: [ awsecscontainermetrics ]
132+
processors: [ filter, metricstransform, resource, batch ]
133+
exporters: [ awsprometheusremotewrite,logging ]
134+
metrics/application/cw:
135+
receivers: [ otlp ]
136+
processors: [ resourcedetection, batch ]
137+
exporters: [ awsemf,logging ]
138+
metrics/application/amp:
139+
receivers: [ otlp ]
140+
processors: [ resourcedetection, batch ]
141+
exporters: [ awsprometheusremotewrite,logging]
142+
traces/application/xray:
143+
receivers: [ otlp ]
144+
processors: [ resourcedetection, batch ]
145+
exporters: [ awsxray ]
146+
extensions: [pprof]
147+
telemetry:
148+
logs:
149+
level: debug

‎terraform/testcases/otlp_grpc_exporter_cw_amp_xray_ecs/parameters.tfvars‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
validation_config = "ecs-container-cw-amp-xray-validation.yml"
2+
3+
# data type will be emitted. Possible values: metric or trace
4+
soaking_data_mode = "trace"
5+
6+
7+

‎validator/src/main/resources/validations/ecs-container-cw-amp-xray-validation.yml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
-
2+
validationType: "trace"
3+
httpPath: "/outgoing-http-call"
4+
httpMethod: "get"
5+
callingType: "http"
6+
expectedTraceTemplate: "SPARK_SDK_ECS_EXPECTED_TRACE"
7+
-
8+
validationType: "cw-metric"
9+
callingType: "none"
10+
expectedMetricTemplate: "ECS_CONTAINER_EXPECTED_METRIC"
11+
12+
13+

0 commit comments

Comments
 (0)