Merge branch 'terraform' into merge-conflict-trace-validator

Author: bhautikpip

README.md

@@ -37,8 +37,6 @@ This is optional item, only do it if your goal is to add a component or fix a bu
 
 4. find out the version number, click into the workflow page, click `e2etest-preparation` step, and click `Versioning for testing`, record the version number. Ex(v0.1.12-299946851).
 
-5. provide your testing aws account id to Ying(wangmyin@), so that he will give permission to your aws account to fetch the testing image from a private ECR. [todo, once we migrate to dockerhub, you don't need this step] 
-
 
 ## 2. Run ECS Test
 
@@ -48,7 +46,7 @@ This is optional item, only do it if your goal is to add a component or fix a bu
 cd terraform/ecs && terraform init && terraform apply -var-file="../testing-suites/statsd-ecs.tfvars"
 ```
 
-### 2.1 run with a specific aoc version
+### 2.2 run with a specific aoc version if you have done 1.4
 
 ```shell
 cd terraform/ecs && terraform init && terraform apply -var-file="../testing-suites/statsd-ecs.tfvars" -var="aoc_version={the version you got from workflow}"
@@ -61,35 +59,35 @@ cd terraform/ecs && terraform destory"
 ```
 ## 3. Run EC2
 
-### 3.1 run with the testing suite [only amazonlinux2 is supported at this moment]
+### 3.1 run with the testing suite [support amazonlinux2, ubuntu16, windows2019]
 
 ```shell
-cd terraform/ec2 && terraform init && terraform apply -var="sshkey_s3_bucket={the bucket name you set in setup}" -var-file="../testing-suites/statsd-ec2.tfvars"
+cd terraform/ec2 && terraform init && terraform apply -var="testing_ami=amazonlinux2" -var="sshkey_s3_bucket={the bucket name you set in setup}" -var-file="../testing-suites/statsd-ec2.tfvars"
 ```
 
-### 3.2 run with a specfic aoc version
+### 3.2 run with a specfic aoc version if you have done 1.4
 
 
 ```shell
 cd terraform/ec2 && terraform init && terraform apply -var="sshkey_s3_bucket={the bucket name you set in setup}" -var-file="../testing-suites/statsd-ec2.tfvars" -var="aoc_version={the version you got from workflow}"
 ```
 
-### 3.3 don't forget to clean the resources
+### 3.3 don't forget to clean the resources 
 ```shell
-cd terraform/ec2 && terraform destory"
+cd terraform/ec2 && terraform destory
 ```
 
 ## 4. Run EKS
 
-create a eks cluster in your account before run below command
+please note you are required to create a eks cluster in your account before running below command
 
 ### 4.1 run with the testing suite
 
 ```shell
 cd terraform/eks && terraform init && terraform apply -var="eks_cluster_name={the eks cluster name in your account}" -var-file="../testing-suites/statsd-eks.tfvars"
 ```
 
-### 4.2 run with a specfic aoc version
+### 4.2 run with a specfic aoc version if you have done 1.4
 
 ```shell
 cd terraform/eks && terraform init && terraform apply -var="eks_cluster_name={the eks cluster name in your account}" -var-file="../testing-suites/statsd-eks.tfvars" -var="aoc_version={the version you got from workflow}"

integ-test/src/main/java/com/amazon/aocagent/validators/MetricValidator.java

@@ -42,7 +42,7 @@
 @Log4j2
 public class MetricValidator implements IValidator {
   private static int MAX_RETRY_COUNT = 60;
-  private static final String DEFAULT_DIMENSION_NAME = "OTLib";
+  private static final String DEFAULT_DIMENSION_NAME = "OTelLib";
   private static final String DEFAULT_DIMENSION_VALUE = "cloudwatch-otel";
 
   private MustacheHelper mustacheHelper = new MustacheHelper();

terraform/README.md

@@ -1,8 +1,36 @@
-## 1. Add a testing suite
+# How to add a testing suite to the framework
 
-please add a new tfvars file under `terraform/testing-suites` folder, each tfvar file represent a testing suite.
+## 1. Basic concepts
 
-### 1.1 add an ecs testing suite
+There're some concepts you might want to learn before starting to add your testing suite. 
+
+### 1.1 the workflow of the testing framework
+
+the testing framework is built based on terraform, every time you run your testing suite, it will
+
+1. **Create resources** Create the related resources in your aws account, such as, ec2 instance/ecs cluster.
+2. **Install softwares** Install AWS Otel Collector/sample apps onto it and start it base on the configurations you provide. 
+3. **Validate data** Validate the data by fetching them from backend(CloudWatch, XRay) base on the validation config you provide. 
+
+
+### 1.2 Testing Suite
+
+each file with suffix ".tfvar" under folder `terraform/testing-suites` will be treated as a testing suite, each testing suite defines different otconfig, validation_config, and deployment config based on different platform.Ex(taskdef for ECS) 
+
+### 1.3 AOC Version
+
+`aoc_version` is a parameter you can use while running the testing-suite, which tells the testing framework to fetch the AWS Otel Collector with this version, this parameter is useful especially when you create a PR to AWS Otel Collector, which will build a version to you with your unmerged code, so that you can test your "new code" with your new testing suite.
+
+### 1.4 Testing Id
+ 
+`testing_id` is a placeholder that you can use in your configuration, which is a unique id representing each run of the testing suite. You can use this testing id as part of your metric name, or dimension name, to ensure the metric emitted from each run of the testing suite is different, so that the validation will always validate on your new metric.
+
+
+## 2. Add a testing suite
+
+please add a new tfvars file under `terraform/testing-suites` folder, each tfvar file represent a testing suite. 
+
+### 2.1 add an ecs testing suite
 
 specify below config in the tfvars file
 1. otconfig_path, please put a new otconfig file under `terraform/templates/otconfig` folder and specify the path in the tfvars file.
@@ -13,24 +41,47 @@ specify below config in the tfvars file
 6. [optional] aoc_image_repo, if you have an aoc image you just built with the new component, then set its repo name here
 7. [optional] aoc_version, if you have an aoc image you just built with the new component, then set it as its tag name.
 
-### 1.2 add an ec2 testing suite
+an example here:
+
+```shell
+sample_app_callable = false
+
+otconfig_path="../template/otconfig/statsd_otconfig.tpl"
+ecs_taskdef_path="../template/ecstaskdef/statsd_taskdef.tpl"
+
+# this file is defined in validator/src/main/resources/validations
+validation_config="statsd-metric-validation.yml"
+
+data_emitter_image="alpine/socat:latest"
+```
+
+### 2.2 add an ec2 testing suite
 
 specify below config in the tfvars file
 1. otconfig_path, please put a new otconfig file under `terraform/templates/otconfig` folder and specify the path in the tfvars file.
 2. docker_compose_path, please put a new docker compose file under `terraform/templates/ec2-docker-compose-config` folder and specify the path in the tfvars file.
 3. validation_config, please put a new validation config file under `validator/src/main/resources/validations` folder and specify the filename in the tfvars file.
 4. [optional] sample_app_callable, by default it's true, only set it to false when you don't have a web application sample app image
 5. [optional] data_emitter_image, if you have a sample app then set its image name here
-6. [optional] package_s3_bucket, if you have an aoc rpm/deb/msi built with the new component, and uploaded it to s3, then set its s3 bucket name here.
 7. [optional] aoc_version, if you have an aoc rpm/dev/msi you just built with the new component, then set it as its tag name.
 
-please note the rpm/dev/msi s3 object path should be 
+an example here:
 
 ```shell
-/amazon_linux/amd64/${aoc_version}/aws-observability-collector.rpm/deb/msi
+sample_app_callable = false
+
+otconfig_path="../template/otconfig/statsd_otconfig.tpl"
+
+docker_compose_path="../template/ec2-docker-compose-config"
+
+# this file is defined in validator/src/main/resources/validations
+validation_config="statsd-metric-validation.yml"
+
+data_emitter_image="alpine/socat:latest"
 ```
 
-### 1.3 add an eks testing suite
+
+### 2.3 add an eks testing suite
 
 specify below config in the tfvars file
 1. otconfig_path, please put a new otconfig file under `terraform/templates/otconfig` folder and specify the path in the tfvars file.
@@ -41,9 +92,23 @@ specify below config in the tfvars file
 6. [optional] aoc_image_repo, if you have an aoc image you just built with the new component, then set its repo name here
 7. [optional] aoc_version, if you have an aoc image you just built with the new component, then set it as its tag name.
 
-### 1.4 how to write the configurations?
+```shell
+sample_app_callable = false
+
+otconfig_path="../template/otconfig/statsd_otconfig.tpl"
+eks_pod_config_path="../template/eks-pod-config/statsd-eks-config.tpl"
+
+# this file is defined in validator/src/main/resources/validations
+validation_config="statsd-metric-validation.yml"
+
+data_emitter_image="alpine/socat:latest"
+```
+
+### 3 how to write the configurations?
+
+You are able to use placeholders in your configuration files, the testing framework will replace the placeholders with its runtime value when you run the testing suite.
 
-#### 1.4.1 otconfig
+#### 3.1 otconfig
 
 Below are the placeholders you can use in the otconfig
 
@@ -71,7 +136,7 @@ service:
       exporters: [logging, awsemf]
 ```
 
-#### 1.4.2 ecs task definition 
+#### 3.2 ecs task definition 
 
 Below are the placeholders you can use in the ecs task def.
 
@@ -185,7 +250,7 @@ an example:
 ]
 ```
 
-#### 1.4.3 Docker compose file
+#### 3.3 Docker compose file
 
 Below are the placeholders you can use in the docker compose file
 
@@ -218,7 +283,7 @@ services:
       start_period: 10s
 ```
 
-#### 1.4.4 Eks Config
+#### 3.4 Eks Config
 
 Below are the placeholders you can use in the EKS config.
 
@@ -237,7 +302,7 @@ sample_app:
   args: []
 ```
 
-#### 1.4.5 Validation config.
+#### 3.5 Validation config.
 
 An example for validation config.
 
@@ -269,6 +334,10 @@ Below are the placeholders you can use in the expected data pattern.
 
 * metricNamespace
 * testingId
+* ecsContext.ecsClusterName
+* ecsContext.ecsTaskArn
+* ecsContext.ecsTaskDefFamily
+* ecsContext.ecsTaskDefVersion
 
 an example: 
 
@@ -278,7 +347,7 @@ an example:
   namespace: {{metricNamespace}}
   dimensions:
     -
-      name: OTLib
+      name: OTelLib
       value: cloudwatch-otel
     -
       name: apiName
@@ -291,7 +360,7 @@ an example:
   namespace: {{metricNamespace}}
   dimensions:
     -
-      name: OTLib
+      name: OTelLib
       value: cloudwatch-otel
     -
       name: apiName
@@ -304,7 +373,7 @@ an example:
   namespace: {{metricNamespace}}
   dimensions:
     -
-      name: OTLib
+      name: OTelLib
       value: cloudwatch-otel
     -
       name: apiName
@@ -317,7 +386,7 @@ an example:
   namespace: {{metricNamespace}}
   dimensions:
     -
-      name: OTLib
+      name: OTelLib
       value: cloudwatch-otel
     -
       name: apiName
@@ -328,7 +397,7 @@ an example:
 ```
 
 
-## 2. Build Sample App
+## 4. Build Sample App
 
 For any testing suite related with sdk, you are required to build a sample app. 
 
@@ -355,7 +424,7 @@ Environment Variable: There will be some env vars which will be set while runnin
 
 * Keep “/” accessible with response code 200.: This “/” will be used for the load balancer health check.
 
-## 3. configure the testing suite in github workflow
+## 5. configure the testing suite in github workflow
 
 below is an example in the aoc repo workflow, you can also configure it in your sdk repos.
 

terraform/basic_components/main.tf

@@ -1,3 +1,18 @@
+# ------------------------------------------------------------------------
+# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License").
+# You may not use this file except in compliance with the License.
+# A copy of the License is located at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# or in the "license" file accompanying this file. This file is distributed
+# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+# express or implied. See the License for the specific language governing
+# permissions and limitations under the License.
+# -------------------------------------------------------------------------
+
 module "common" {
   source = "../common"
 }

terraform/basic_components/outputs.tf

@@ -1,3 +1,18 @@
+# ------------------------------------------------------------------------
+# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License").
+# You may not use this file except in compliance with the License.
+# A copy of the License is located at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# or in the "license" file accompanying this file. This file is distributed
+# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+# express or implied. See the License for the specific language governing
+# permissions and limitations under the License.
+# -------------------------------------------------------------------------
+
 output "aoc_vpc_id" {
   value = data.aws_vpc.aoc_vpc.id
 }

terraform/basic_components/variables.tf

@@ -1,3 +1,18 @@
+# ------------------------------------------------------------------------
+# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License").
+# You may not use this file except in compliance with the License.
+# A copy of the License is located at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# or in the "license" file accompanying this file. This file is distributed
+# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+# express or implied. See the License for the specific language governing
+# permissions and limitations under the License.
+# -------------------------------------------------------------------------
+
 variable "region" {
   default = "us-west-2"
 }

terraform/common/main.tf

@@ -1,3 +1,18 @@
+# ------------------------------------------------------------------------
+# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License").
+# You may not use this file except in compliance with the License.
+# A copy of the License is located at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# or in the "license" file accompanying this file. This file is distributed
+# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+# express or implied. See the License for the specific language governing
+# permissions and limitations under the License.
+# -------------------------------------------------------------------------
+
 # generate a testing_id whenever people want to use, for example, use it as a ecs cluster to prevent cluster name conflict
 resource "random_id" "testing_id" {
   byte_length = 8

terraform/common/outputs.tf

@@ -1,3 +1,18 @@
+# ------------------------------------------------------------------------
+# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License").
+# You may not use this file except in compliance with the License.
+# A copy of the License is located at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# or in the "license" file accompanying this file. This file is distributed
+# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+# express or implied. See the License for the specific language governing
+# permissions and limitations under the License.
+# -------------------------------------------------------------------------
+
 output "testing_id" {
   value = random_id.testing_id.hex
 }