Skip to content

Commit 180b6fb

Browse files
ryota-sakamotoryota-sakamoto
authored
feat: Support CloudWatch as a metric provider (argoproj#1338)
Signed-off-by: Ryota Sakamoto <[email protected]>
1 parent ebd6cbd commit 180b6fb

18 files changed

+1382
-0
lines changed

‎docs/analysis/cloudwatch.md

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
# CloudWatch Metrics
2+
3+
!!! important
4+
Available since v1.1.0
5+
6+
A [CloudWatch](https://aws.amazon.com/cloudwatch/) using [GetMetricData](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html) can be used to obtain measurements for analysis.
7+
8+
## Setup
9+
10+
You can use CloudWatch Metrics if you have used to EKS or not. This analysis is required IAM permission for `cloudwatch:GetMetricData` and you need to define `AWS_REGION` in Deployment for `argo-rollouts`.
11+
12+
### EKS
13+
14+
If you create new cluster on EKS, you can attach [cluster IAM role](https://docs.aws.amazon.com/eks/latest/userguide/service_IAM_role.html) or attach [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html).
15+
If you have already cluster on EKS, you can attach [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html).
16+
17+
### not EKS
18+
19+
You need to define access key and secret key.
20+
21+
```yaml
22+
apiVersion: v1
23+
kind: Secret
24+
metadata:
25+
name: cloudwatch-secret
26+
type: Opaque
27+
stringData:
28+
AWS_ACCESS_KEY_ID: <aws-access-key-id>
29+
AWS_SECRET_ACCESS_KEY: <aws-secret-access-key>
30+
AWS_REGION: <aws-region>
31+
```
32+
33+
```yaml
34+
apiVersion: apps/v1
35+
kind: Deployment
36+
metadata:
37+
name: argo-rollouts
38+
spec:
39+
template:
40+
spec:
41+
containers:
42+
- name: argo-rollouts
43+
env:
44+
- name: AWS_ACCESS_KEY_ID
45+
valueFrom:
46+
secretKeyRef:
47+
name: cloudwatch-secret
48+
key: AWS_ACCESS_KEY_ID
49+
- name: AWS_SECRET_ACCESS_KEY
50+
valueFrom:
51+
secretKeyRef:
52+
name: cloudwatch-secret
53+
key: AWS_SECRET_ACCESS_KEY
54+
- name: AWS_REGION
55+
valueFrom:
56+
secretKeyRef:
57+
name: cloudwatch-secret
58+
key: AWS_REGION
59+
```
60+
61+
## Configuration
62+
63+
- `metricDataQueries` - GetMetricData query: [MetricDataQuery](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDataQuery.html)
64+
- `interval` - optional interval, e.g. 30m, default: 5m
65+
66+
```yaml
67+
apiVersion: argoproj.io/v1alpha1
68+
kind: AnalysisTemplate
69+
metadata:
70+
name: success-rate
71+
spec:
72+
metrics:
73+
- name: success-rate
74+
interval: 1m
75+
successCondition: "len(result[0].Values) >= 5 and all(result[0].Values, {# <= 0.01})"
76+
failureLimit: 3
77+
provider:
78+
cloudWatch:
79+
interval: 30m
80+
metricDataQueries:
81+
- {
82+
"id": "rate",
83+
"expression": "errors / requests"
84+
}
85+
- {
86+
"id": "errors",
87+
"metricStat": {
88+
"metric": {
89+
"namespace": "app",
90+
"metricName": "errors"
91+
},
92+
"period": 300,
93+
"stat": "Sum",
94+
"unit": "Count"
95+
},
96+
"returnData": false
97+
}
98+
- {
99+
"id": "requests",
100+
"metricStat": {
101+
"metric": {
102+
"namespace": "app",
103+
"metricName": "requests"
104+
},
105+
"period": 300,
106+
"stat": "Sum",
107+
"unit": "Count"
108+
},
109+
"returnData": false
110+
}
111+
```
112+
113+
## debug
114+
115+
You can confirm the results value in `AnalysisRun`.
116+
117+
```bash
118+
$ kubectl get analysisrun/rollouts-name-xxxxxxxxxx-xx -o yaml
119+
(snip)
120+
status:
121+
metricResults:
122+
- count: 2
123+
failed: 1
124+
measurements:
125+
- finishedAt: "2021-09-08T17:29:14Z"
126+
phase: Failed
127+
startedAt: "2021-09-08T17:29:13Z"
128+
value: '[[0.0029476787030213707 0.006100422336931018 0.01020408163265306 0.007932573128408527
129+
0.00589622641509434 0.006339144215530904]]'
130+
- finishedAt: "2021-09-08T17:30:14Z"
131+
phase: Successful
132+
startedAt: "2021-09-08T17:30:14Z"
133+
value: '[[0.004484304932735426 0.0058374494836102376 0.006736068585425597 0.008444444444444444
134+
0.006859756097560976 0.0045385779122541605]]'
135+
name: success-rate
136+
phase: Running
137+
successful: 1
138+
phase: Running
139+
startedAt: "2021-09-08T17:29:14Z"
140+
```

‎go.mod

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ require (
88
github.com/argoproj/pkg v0.9.0
99
github.com/aws/aws-sdk-go-v2/config v1.0.0
1010
github.com/aws/aws-sdk-go-v2/internal/ini v1.2.1 // indirect
11+
github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.5.0
1112
github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2 v1.6.1
1213
github.com/blang/semver v3.5.1+incompatible
1314
github.com/docker/spdystream v0.0.0-20181023171402-6480d4af844c // indirect

‎go.sum

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -177,6 +177,7 @@ github.com/aws/aws-sdk-go v1.33.16/go.mod h1:5zCpMtNQVjRREroY7sYe8lOMRSxkhG6MZve
177177
github.com/aws/aws-sdk-go v1.35.24/go.mod h1:tlPOdRjfxPBpNIwqDj61rmsnA85v9jc0Ps9+muhnW+k=
178178
github.com/aws/aws-sdk-go-v2 v0.18.0/go.mod h1:JWVYvqSMppoMJC0x5wdwiImzgXTI9FuZwxzkQq9wy+g=
179179
github.com/aws/aws-sdk-go-v2 v1.0.0/go.mod h1:smfAbmpW+tcRVuNUjo3MOArSZmW72t62rkCzc2i0TWM=
180+
github.com/aws/aws-sdk-go-v2 v1.7.0/go.mod h1:tb9wi5s61kTDA5qCkcDbt3KRVV74GGslQkl/DRdX/P4=
180181
github.com/aws/aws-sdk-go-v2 v1.8.1 h1:GcFgQl7MsBygmeeqXyV1ivrTEmsVz/rdFJaTcltG9ag=
181182
github.com/aws/aws-sdk-go-v2 v1.8.1/go.mod h1:xEFuWz+3TYdlPRuo+CqATbeDWIWyaT5uAPwPaWtgse0=
182183
github.com/aws/aws-sdk-go-v2/config v1.0.0 h1:x6vSFAwqAvhYPeSu60f0ZUlGHo3PKKmwDOTL8aMXtv4=
@@ -187,13 +188,16 @@ github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.0.0 h1:lO7fH5n7Q1dKcDBpuTmwJylD
187188
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.0.0/go.mod h1:wpMHDCXvOXZxGCRSidyepa8uJHY4vaBGfY2/+oKU/Bc=
188189
github.com/aws/aws-sdk-go-v2/internal/ini v1.2.1 h1:IkqRRUZTKaS16P2vpX+FNc2jq3JWa3c478gykQp4ow4=
189190
github.com/aws/aws-sdk-go-v2/internal/ini v1.2.1/go.mod h1:Pv3WenDjI0v2Jl7UaMFIIbPOBbhn33RmmAmGgkXDoqY=
191+
github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.5.0 h1:XO1uX7dQKWfD0WzycEfz+bL/7rl0SsQ05VJwLPWGzGM=
192+
github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.5.0/go.mod h1:acH3+MQoiMzozT/ivU+DbRg7Ooo2298RdRaWcOv+4vM=
190193
github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2 v1.6.1 h1:mGc8UvJS4XJv8Tp7Doxlx2p3vfwPx46K9zg+9s9szPE=
191194
github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2 v1.6.1/go.mod h1:lGKz4aJbqGX+pgyXG47ZBAJPjwrlA5+TJsAuJ2+aE2g=
192195
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.0.0 h1:IAutMPSrynpvKOpHG6HyWHmh1xmxWAmYOK84NrQVqVQ=
193196
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.0.0/go.mod h1:3jExOmpbjgPnz2FJaMOfbSk1heTkZ66aD3yNtVhnjvI=
194197
github.com/aws/aws-sdk-go-v2/service/sts v1.0.0 h1:6XCgxNfE4L/Fnq+InhVNd16DKc6Ue1f3dJl3IwwJRUQ=
195198
github.com/aws/aws-sdk-go-v2/service/sts v1.0.0/go.mod h1:5f+cELGATgill5Pu3/vK3Ebuigstc+qYEHW5MvGWZO4=
196199
github.com/aws/smithy-go v1.0.0/go.mod h1:EzMw8dbp/YJL4A5/sbhGddag+NPT7q084agLbB9LgIw=
200+
github.com/aws/smithy-go v1.5.0/go.mod h1:SObp3lf9smib00L/v3U2eAKG8FyQ7iLrJnQiAmR5n+E=
197201
github.com/aws/smithy-go v1.7.0 h1:+cLHMRrDZvQ4wk+KuQ9yH6eEg6KZEJ9RI2IkDqnygCg=
198202
github.com/aws/smithy-go v1.7.0/go.mod h1:SObp3lf9smib00L/v3U2eAKG8FyQ7iLrJnQiAmR5n+E=
199203
github.com/aybabtme/rgbterm v0.0.0-20170906152045-cc83f3b3ce59/go.mod h1:q/89r3U2H7sSsE2t6Kca0lfwTK8JdoNGS/yzM/4iH5I=

‎manifests/crds/analysis-run-crd.yaml

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,59 @@ spec:
9797
type: string
9898
provider:
9999
properties:
100+
cloudWatch:
101+
properties:
102+
interval:
103+
type: string
104+
metricDataQueries:
105+
items:
106+
properties:
107+
expression:
108+
type: string
109+
id:
110+
type: string
111+
label:
112+
type: string
113+
metricStat:
114+
properties:
115+
metric:
116+
properties:
117+
dimensions:
118+
items:
119+
properties:
120+
name:
121+
type: string
122+
value:
123+
type: string
124+
type: object
125+
type: array
126+
metricName:
127+
type: string
128+
namespace:
129+
type: string
130+
type: object
131+
period:
132+
anyOf:
133+
- type: integer
134+
- type: string
135+
x-kubernetes-int-or-string: true
136+
stat:
137+
type: string
138+
unit:
139+
type: string
140+
type: object
141+
period:
142+
anyOf:
143+
- type: integer
144+
- type: string
145+
x-kubernetes-int-or-string: true
146+
returnData:
147+
type: boolean
148+
type: object
149+
type: array
150+
required:
151+
- metricDataQueries
152+
type: object
100153
datadog:
101154
properties:
102155
interval:

‎manifests/crds/analysis-template-crd.yaml

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,59 @@ spec:
9292
type: string
9393
provider:
9494
properties:
95+
cloudWatch:
96+
properties:
97+
interval:
98+
type: string
99+
metricDataQueries:
100+
items:
101+
properties:
102+
expression:
103+
type: string
104+
id:
105+
type: string
106+
label:
107+
type: string
108+
metricStat:
109+
properties:
110+
metric:
111+
properties:
112+
dimensions:
113+
items:
114+
properties:
115+
name:
116+
type: string
117+
value:
118+
type: string
119+
type: object
120+
type: array
121+
metricName:
122+
type: string
123+
namespace:
124+
type: string
125+
type: object
126+
period:
127+
anyOf:
128+
- type: integer
129+
- type: string
130+
x-kubernetes-int-or-string: true
131+
stat:
132+
type: string
133+
unit:
134+
type: string
135+
type: object
136+
period:
137+
anyOf:
138+
- type: integer
139+
- type: string
140+
x-kubernetes-int-or-string: true
141+
returnData:
142+
type: boolean
143+
type: object
144+
type: array
145+
required:
146+
- metricDataQueries
147+
type: object
95148
datadog:
96149
properties:
97150
interval:

‎manifests/crds/cluster-analysis-template-crd.yaml

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,59 @@ spec:
9292
type: string
9393
provider:
9494
properties:
95+
cloudWatch:
96+
properties:
97+
interval:
98+
type: string
99+
metricDataQueries:
100+
items:
101+
properties:
102+
expression:
103+
type: string
104+
id:
105+
type: string
106+
label:
107+
type: string
108+
metricStat:
109+
properties:
110+
metric:
111+
properties:
112+
dimensions:
113+
items:
114+
properties:
115+
name:
116+
type: string
117+
value:
118+
type: string
119+
type: object
120+
type: array
121+
metricName:
122+
type: string
123+
namespace:
124+
type: string
125+
type: object
126+
period:
127+
anyOf:
128+
- type: integer
129+
- type: string
130+
x-kubernetes-int-or-string: true
131+
stat:
132+
type: string
133+
unit:
134+
type: string
135+
type: object
136+
period:
137+
anyOf:
138+
- type: integer
139+
- type: string
140+
x-kubernetes-int-or-string: true
141+
returnData:
142+
type: boolean
143+
type: object
144+
type: array
145+
required:
146+
- metricDataQueries
147+
type: object
95148
datadog:
96149
properties:
97150
interval:

0 commit comments

Comments
 (0)