Skip to content

Commit 641478c

Browse files
tristanburgessthepalbiDuologic
authored
cloudprovider: add support for AWS Accounts and CloudWatch Scrape Jobs (#1882)
* Cloud Observability AWS Accounts scaffolding (#1597) * initial commit for AWS accounts for Cloud Observability * mv cloudobservability package and start impl ops * start adding tests * get initial tests passing * solidify testing and example data scaffolding * update CODEOWNERS * update docs * fix linting * PR feedback * update docs * fix examples * cloud provider API client initial pass (#1649) * initial commit * get creates working * Cloud Provider API client impl * revert tmp testing fixes * fix examples test check * fix docs * tf fmt and update docs * PR feedback * terraform fmt * document id format explicitly * Cloud Provider AWS CloudWatch Scrape Jobs scaffolding (#1662) * initial commit * flesh out schema and examples * add data sources skeletons * data source examples * get initial testing working * start converting to new TF plugin framework * finished migrating aws accounts to Terraform Framework * more progress on conversion * finish migrating to TF framework and tests passing again * finalize-ish testing * update docs * shorten struct names * update docs * add terraform-side validation of some scrape job properties * revert tmp testing change * fix memory aliasing in convertScrapeJobClientModelToTFModel() * revert tmp testing change * fix jobs list datasource schema * pr feedback * tf fmt * revert tmp testing change * simplify examples * update docs * remove unneeded field from examples * CW Jobs API Calls for cloudprovider resources (#1780) * initial commit of CW jobs API calls * get create and delete working, minor refactors * start prepare other endpoints * fix tests * lint errors * fix tag filters for scrape jobs * add support for export tags and disabled reason settings * revert testing changes * many small tweaks * revert tmp testing changes * reduce ID computations * pr feedback * pr feedback * cloudprovider: fix update calls for accounts and scrape jobs (#1789) * cloudprovider: fix update * reduce plan noise on update * cw scrape job: fix modifyplan when creating a new resource (#1804) * cw scrape job: fix modifyplan * add error when client is not configured in provider * cloudprovider: ensure empty optional collections persist uniformly (#1808) * fix some state wonkiness * prepare for acceptance test ticket * Add docs for setting up cloud provider provider section (#1816) * simplify scrape job examples a bit * generate docs with examples * added pre docs * revert changes * Apply suggestions from code review Co-authored-by: Tristan <[email protected]> * tell user to use previous token * added sub sections * more pr corrections --------- Co-authored-by: Tristan <[email protected]> * Add common provider headers to cloud-provider-client (#1828) * Add provider headers * Apply suggestions from code review Co-authored-by: Jeroen Op 't Eynde <[email protected]> --------- Co-authored-by: Jeroen Op 't Eynde <[email protected]> * cloudprovider: update description for CW scrape job regions field (#1829) * cloudprovider: update regions behavior for cloudwatch jobs (#1846) * update regions behavior for cloudwatch jobs * match data source example with resource example * fix docs * pr feedback * cloudprovider: aws acceptance tests (#1881) * initial commit - aws account tests * update secret paths * fix account tests * set up cw scrape job tests * fix tests * data source lookup dependencies * moar test fixes * test config common function * pr feedback * fix tabs * small doc tweak to clarify scrape job field usagge * update CODEOWNERS --------- Co-authored-by: Pablo <[email protected]> Co-authored-by: Jeroen Op 't Eynde <[email protected]>
1 parent 64da3ae commit 641478c

40 files changed

+3589
-26
lines changed

‎.github/CODEOWNERS‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,8 @@
22

33
/internal/resources/grafana/*_alerting_* @grafana/platform-monitoring @grafana/alerting-squad
44
/internal/resources/cloud/* @grafana/platform-monitoring @grafana/grafana-com-maintainers
5-
/internal/resources/connections/* @grafana/terraform-provider @grafana/middleware-apps
5+
/internal/resources/cloudprovider/* @grafana/platform-monitoring @grafana/middleware-apps
6+
/internal/resources/connections/* @grafana/platform-monitoring @grafana/middleware-apps
67
/internal/resources/machinelearning/* @grafana/platform-monitoring @grafana/machine-learning
78
/internal/resources/oncall/* @grafana/platform-monitoring @grafana/grafana-oncall
89
/internal/resources/slo/* @grafana/platform-monitoring @grafana/slo-squad

‎.github/workflows/acc-tests.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,11 @@ jobs:
4343
GRAFANA_SM_ACCESS_TOKEN=cloud-instance-tests:sm-token
4444
GRAFANA_SM_URL=cloud-instance-tests:sm-url
4545
GRAFANA_URL=cloud-instance-tests:url
46+
GRAFANA_CLOUD_PROVIDER_URL=cloudprovider-tests:url
47+
GRAFANA_CLOUD_PROVIDER_TEST_AWS_ACCOUNT_RESOURCE_ID=cloudprovider-tests:test-aws-account-resource-id
48+
GRAFANA_CLOUD_PROVIDER_ACCESS_TOKEN=cloudprovider-tests:access-token
49+
GRAFANA_CLOUD_PROVIDER_AWS_ROLE_ARN=cloudprovider-tests:aws-role-arn
50+
GRAFANA_CLOUD_PROVIDER_TEST_STACK_ID=cloudprovider-tests:test-stack-id
4651
- uses: iFaxity/wait-on-action@a7d13170ec542bdca4ef8ac4b15e9c6aa00a6866 # v1.2.1
4752
with:
4853
resource: ${{ env.GRAFANA_URL }}

‎GNUmakefile‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,5 +67,8 @@ golangci-lint:
6767
--workdir "/src" \
6868
golangci/golangci-lint:v1.61.0 golangci-lint run ./... -v
6969

70+
docs:
71+
go generate ./...
72+
7073
linkcheck:
7174
docker run --rm --entrypoint sh -v "$$PWD:$$PWD" -w "$$PWD" python:3.11-alpine -c "pip3 install linkchecker && linkchecker --config .linkcheckerrc docs"

‎docs/data-sources/cloud_provider_aws_account.md‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
---
2+
# generated by https://github.com/hashicorp/terraform-plugin-docs
3+
page_title: "grafana_cloud_provider_aws_account Data Source - terraform-provider-grafana"
4+
subcategory: "Cloud Provider"
5+
description: |-
6+
7+
---
8+
9+
# grafana_cloud_provider_aws_account (Data Source)
10+
11+
12+
13+
## Example Usage
14+
15+
```terraform
16+
data "grafana_cloud_stack" "test" {
17+
slug = "gcloudstacktest"
18+
}
19+
20+
resource "grafana_cloud_provider_aws_account" "test" {
21+
stack_id = data.grafana_cloud_stack.test.id
22+
role_arn = data.aws_iam_role.test.arn
23+
regions = [
24+
"us-east-2",
25+
"eu-west-3"
26+
]
27+
}
28+
29+
data "grafana_cloud_provider_aws_account" "test" {
30+
stack_id = data.grafana_cloud_stack.test.id
31+
resource_id = grafana_cloud_provider_aws_account.test.resource_id
32+
}
33+
```
34+
35+
<!-- schema generated by tfplugindocs -->
36+
## Schema
37+
38+
### Required
39+
40+
- `resource_id` (String) The ID given by the Grafana Cloud Provider API to this AWS Account resource.
41+
- `stack_id` (String) The StackID of the Grafana Cloud instance. Part of the Terraform Resource ID.
42+
43+
### Read-Only
44+
45+
- `id` (String) The Terraform Resource ID. This has the format "{{ stack_id }}:{{ resource_id }}".
46+
- `regions` (Set of String) A set of regions that this AWS Account resource applies to.
47+
- `role_arn` (String) An IAM Role ARN string to represent with this AWS Account resource.

‎docs/data-sources/cloud_provider_aws_cloudwatch_scrape_job.md‎

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
---
2+
# generated by https://github.com/hashicorp/terraform-plugin-docs
3+
page_title: "grafana_cloud_provider_aws_cloudwatch_scrape_job Data Source - terraform-provider-grafana"
4+
subcategory: "Cloud Provider"
5+
description: |-
6+
7+
---
8+
9+
# grafana_cloud_provider_aws_cloudwatch_scrape_job (Data Source)
10+
11+
12+
13+
## Example Usage
14+
15+
```terraform
16+
data "grafana_cloud_stack" "test" {
17+
slug = "gcloudstacktest"
18+
}
19+
20+
data "aws_iam_role" "test" {
21+
name = "my-role"
22+
}
23+
24+
resource "grafana_cloud_provider_aws_account" "test" {
25+
stack_id = data.grafana_cloud_stack.test.id
26+
role_arn = data.aws_iam_role.test.arn
27+
regions = [
28+
"us-east-1",
29+
"us-east-2",
30+
"us-west-1"
31+
]
32+
}
33+
34+
35+
resource "grafana_cloud_provider_aws_cloudwatch_scrape_job" "test" {
36+
stack_id = data.grafana_cloud_stack.test.id
37+
name = "my-cloudwatch-scrape-job"
38+
aws_account_resource_id = grafana_cloud_provider_aws_account.test.resource_id
39+
export_tags = true
40+
41+
service {
42+
name = "AWS/EC2"
43+
metric {
44+
name = "CPUUtilization"
45+
statistics = ["Average"]
46+
}
47+
metric {
48+
name = "StatusCheckFailed"
49+
statistics = ["Maximum"]
50+
}
51+
scrape_interval_seconds = 300
52+
resource_discovery_tag_filter {
53+
key = "k8s.io/cluster-autoscaler/enabled"
54+
value = "true"
55+
}
56+
tags_to_add_to_metrics = ["eks:cluster-name"]
57+
}
58+
59+
custom_namespace {
60+
name = "CoolApp"
61+
metric {
62+
name = "CoolMetric"
63+
statistics = ["Maximum", "Sum"]
64+
}
65+
scrape_interval_seconds = 300
66+
}
67+
}
68+
69+
70+
data "grafana_cloud_provider_aws_cloudwatch_scrape_job" "test" {
71+
stack_id = data.grafana_cloud_stack.test.id
72+
name = grafana_cloud_provider_aws_cloudwatch_scrape_job.test.name
73+
}
74+
```
75+
76+
<!-- schema generated by tfplugindocs -->
77+
## Schema
78+
79+
### Required
80+
81+
- `name` (String) The name of the CloudWatch Scrape Job. Part of the Terraform Resource ID.
82+
- `stack_id` (String) The Stack ID of the Grafana Cloud instance. Part of the Terraform Resource ID.
83+
84+
### Read-Only
85+
86+
- `aws_account_resource_id` (String) The ID assigned by the Grafana Cloud Provider API to an AWS Account resource that should be associated with this CloudWatch Scrape Job. This can be provided by the `id` attribute of the `grafana_cloud_provider_aws_account` resource.
87+
- `custom_namespace` (Block List) Zero or more configuration blocks to configure custom namespaces for the CloudWatch Scrape Job to scrape. Each block must have a distinct `name` attribute. When accessing this as an attribute reference, it is a list of objects. (see [below for nested schema](#nestedblock--custom_namespace))
88+
- `disabled_reason` (String) When the CloudWatch Scrape Job is disabled, this will show the reason that it is in that state.
89+
- `enabled` (Boolean) Whether the CloudWatch Scrape Job is enabled or not.
90+
- `export_tags` (Boolean) When enabled, AWS resource tags are exported as Prometheus labels to metrics formatted as `aws_<service_name>_info`.
91+
- `id` (String) The Terraform Resource ID. This has the format "{{ stack_id }}:{{ name }}".
92+
- `regions` (Set of String) The set of AWS region names that this CloudWatch Scrape Job is configured to scrape.
93+
- `regions_subset_override_used` (Boolean) When true, the `regions` attribute will be the set of regions configured in the override. When false, the `regions` attribute will be the set of regions belonging to the AWS Account resource that is associated with this CloudWatch Scrape Job.
94+
- `role_arn` (String) The AWS ARN of the IAM role associated with the AWS Account resource that is being used by this CloudWatch Scrape Job.
95+
- `service` (Block List) One or more configuration blocks to dictate what this CloudWatch Scrape Job should scrape. Each block must have a distinct `name` attribute. When accessing this as an attribute reference, it is a list of objects. (see [below for nested schema](#nestedblock--service))
96+
97+
<a id="nestedblock--custom_namespace"></a>
98+
### Nested Schema for `custom_namespace`
99+
100+
Read-Only:
101+
102+
- `metric` (Block List) One or more configuration blocks to configure metrics and their statistics to scrape. Each block must represent a distinct metric name. When accessing this as an attribute reference, it is a list of objects. (see [below for nested schema](#nestedblock--custom_namespace--metric))
103+
- `name` (String) The name of the custom namespace to scrape.
104+
- `scrape_interval_seconds` (Number) The interval in seconds to scrape the custom namespace.
105+
106+
<a id="nestedblock--custom_namespace--metric"></a>
107+
### Nested Schema for `custom_namespace.metric`
108+
109+
Read-Only:
110+
111+
- `name` (String) The name of the metric to scrape.
112+
- `statistics` (Set of String) A set of statistics to scrape.
113+
114+
115+
116+
<a id="nestedblock--service"></a>
117+
### Nested Schema for `service`
118+
119+
Read-Only:
120+
121+
- `metric` (Block List) One or more configuration blocks to configure metrics and their statistics to scrape. Each block must represent a distinct metric name. When accessing this as an attribute reference, it is a list of objects. (see [below for nested schema](#nestedblock--service--metric))
122+
- `name` (String) The name of the service to scrape. See https://grafana.com/docs/grafana-cloud/monitor-infrastructure/aws/cloudwatch-metrics/services/ for supported services, metrics, and their statistics.
123+
- `resource_discovery_tag_filter` (Block List) One or more configuration blocks to configure tag filters applied to discovery of resource entities in the associated AWS account. When accessing this as an attribute reference, it is a list of objects. (see [below for nested schema](#nestedblock--service--resource_discovery_tag_filter))
124+
- `scrape_interval_seconds` (Number) The interval in seconds to scrape the service. See https://grafana.com/docs/grafana-cloud/monitor-infrastructure/aws/cloudwatch-metrics/services/ for supported scrape intervals.
125+
- `tags_to_add_to_metrics` (Set of String) A set of tags to add to all metrics exported by this scrape job, for use in PromQL queries.
126+
127+
<a id="nestedblock--service--metric"></a>
128+
### Nested Schema for `service.metric`
129+
130+
Read-Only:
131+
132+
- `name` (String) The name of the metric to scrape.
133+
- `statistics` (Set of String) A set of statistics to scrape.
134+
135+
136+
<a id="nestedblock--service--resource_discovery_tag_filter"></a>
137+
### Nested Schema for `service.resource_discovery_tag_filter`
138+
139+
Read-Only:
140+
141+
- `key` (String) The key of the tag filter.
142+
- `value` (String) The value of the tag filter.

‎docs/data-sources/cloud_provider_aws_cloudwatch_scrape_jobs.md‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
---
2+
# generated by https://github.com/hashicorp/terraform-plugin-docs
3+
page_title: "grafana_cloud_provider_aws_cloudwatch_scrape_jobs Data Source - terraform-provider-grafana"
4+
subcategory: "Cloud Provider"
5+
description: |-
6+
7+
---
8+
9+
# grafana_cloud_provider_aws_cloudwatch_scrape_jobs (Data Source)
10+
11+
12+
13+
## Example Usage
14+
15+
```terraform
16+
data "grafana_cloud_stack" "test" {
17+
slug = "gcloudstacktest"
18+
}
19+
20+
data "grafana_cloud_provider_aws_cloudwatch_scrape_jobs" "test" {
21+
stack_id = data.grafana_cloud_stack.test.id
22+
}
23+
```
24+
25+
<!-- schema generated by tfplugindocs -->
26+
## Schema
27+
28+
### Required
29+
30+
- `stack_id` (String) The Stack ID of the Grafana Cloud instance. Part of the Terraform Resource ID.
31+
32+
### Read-Only
33+
34+
- `id` (String) The Terraform Resource ID. This has the format "{{ stack_id }}".
35+
- `scrape_job` (Block List) A list of AWS CloudWatch Scrape Job objects associated with the given StackID. (see [below for nested schema](#nestedblock--scrape_job))
36+
37+
<a id="nestedblock--scrape_job"></a>
38+
### Nested Schema for `scrape_job`
39+
40+
Read-Only:
41+
42+
- `aws_account_resource_id` (String) The ID assigned by the Grafana Cloud Provider API to an AWS Account resource that should be associated with this CloudWatch Scrape Job. This can be provided by the `id` attribute of the `grafana_cloud_provider_aws_account` resource.
43+
- `custom_namespace` (Block List) Zero or more configuration blocks to configure custom namespaces for the CloudWatch Scrape Job to scrape. Each block must have a distinct `name` attribute. When accessing this as an attribute reference, it is a list of objects. (see [below for nested schema](#nestedblock--scrape_job--custom_namespace))
44+
- `disabled_reason` (String) When the CloudWatch Scrape Job is disabled, this will show the reason that it is in that state.
45+
- `enabled` (Boolean) Whether the CloudWatch Scrape Job is enabled or not.
46+
- `export_tags` (Boolean) When enabled, AWS resource tags are exported as Prometheus labels to metrics formatted as `aws_<service_name>_info`.
47+
- `id` (String) The Terraform Resource ID. This has the format "{{ stack_id }}:{{ name }}".
48+
- `name` (String) The name of the CloudWatch Scrape Job. Part of the Terraform Resource ID.
49+
- `regions` (Set of String) The set of AWS region names that this CloudWatch Scrape Job is configured to scrape.
50+
- `regions_subset_override_used` (Boolean) When true, the `regions` attribute will be the set of regions configured in the override. When false, the `regions` attribute will be the set of regions belonging to the AWS Account resource that is associated with this CloudWatch Scrape Job.
51+
- `role_arn` (String) The AWS ARN of the IAM role associated with the AWS Account resource that is being used by this CloudWatch Scrape Job.
52+
- `service` (Block List) One or more configuration blocks to dictate what this CloudWatch Scrape Job should scrape. Each block must have a distinct `name` attribute. When accessing this as an attribute reference, it is a list of objects. (see [below for nested schema](#nestedblock--scrape_job--service))
53+
- `stack_id` (String) The Stack ID of the Grafana Cloud instance. Part of the Terraform Resource ID.
54+
55+
<a id="nestedblock--scrape_job--custom_namespace"></a>
56+
### Nested Schema for `scrape_job.custom_namespace`
57+
58+
Read-Only:
59+
60+
- `metric` (Block List) One or more configuration blocks to configure metrics and their statistics to scrape. Each block must represent a distinct metric name. When accessing this as an attribute reference, it is a list of objects. (see [below for nested schema](#nestedblock--scrape_job--custom_namespace--metric))
61+
- `name` (String) The name of the custom namespace to scrape.
62+
- `scrape_interval_seconds` (Number) The interval in seconds to scrape the custom namespace.
63+
64+
<a id="nestedblock--scrape_job--custom_namespace--metric"></a>
65+
### Nested Schema for `scrape_job.custom_namespace.metric`
66+
67+
Read-Only:
68+
69+
- `name` (String) The name of the metric to scrape.
70+
- `statistics` (Set of String) A set of statistics to scrape.
71+
72+
73+
74+
<a id="nestedblock--scrape_job--service"></a>
75+
### Nested Schema for `scrape_job.service`
76+
77+
Read-Only:
78+
79+
- `metric` (Block List) One or more configuration blocks to configure metrics and their statistics to scrape. Each block must represent a distinct metric name. When accessing this as an attribute reference, it is a list of objects. (see [below for nested schema](#nestedblock--scrape_job--service--metric))
80+
- `name` (String) The name of the service to scrape. See https://grafana.com/docs/grafana-cloud/monitor-infrastructure/aws/cloudwatch-metrics/services/ for supported services, metrics, and their statistics.
81+
- `resource_discovery_tag_filter` (Block List) One or more configuration blocks to configure tag filters applied to discovery of resource entities in the associated AWS account. When accessing this as an attribute reference, it is a list of objects. (see [below for nested schema](#nestedblock--scrape_job--service--resource_discovery_tag_filter))
82+
- `scrape_interval_seconds` (Number) The interval in seconds to scrape the service. See https://grafana.com/docs/grafana-cloud/monitor-infrastructure/aws/cloudwatch-metrics/services/ for supported scrape intervals.
83+
- `tags_to_add_to_metrics` (Set of String) A set of tags to add to all metrics exported by this scrape job, for use in PromQL queries.
84+
85+
<a id="nestedblock--scrape_job--service--metric"></a>
86+
### Nested Schema for `scrape_job.service.metric`
87+
88+
Read-Only:
89+
90+
- `name` (String) The name of the metric to scrape.
91+
- `statistics` (Set of String) A set of statistics to scrape.
92+
93+
94+
<a id="nestedblock--scrape_job--service--resource_discovery_tag_filter"></a>
95+
### Nested Schema for `scrape_job.service.resource_discovery_tag_filter`
96+
97+
Read-Only:
98+
99+
- `key` (String) The key of the tag filter.
100+
- `value` (String) The value of the tag filter.

0 commit comments

Comments
 (0)