Skip to content

Commit 07545ac

Browse files
lewinkedrsbonclay7
lewinkedrs
and
bonclay7
authored
AMP Observability Pattern (#55)
* working first run * removing core module dependencies * adding CW datasource * alarms MVP * readmes * Adding Screenshot * Adding billing note * adding billing module * Revert "adding billing module" This reverts commit 40d667e37db1036cd71a471ef2fde83ec02aaa13. reverting * adding billing module * Updating Screenshot * resolving feedback * removing unused modules * fmt * Support for tf 1.3.x * removing unused variables * support alarms for multiple workspaces * Updating Readme * amp to managed prometheus * sub-module * Fix pre-commit Co-authored-by: Rodrigue Koffi <[email protected]>
1 parent 0f772e9 commit 07545ac

File tree

23 files changed

+1256
-6
lines changed

23 files changed

+1256
-6
lines changed

‎examples/managed-prometheus-monitoring/README.md

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
# Existing Managed Prometheus Workspace Observability Pattern
2+
3+
This example demonstrates how to use the AWS Observability Accelerator Terraform
4+
modules with Amazon Managed Prometheus (AMP) workspace monitoring enabled.
5+
6+
The current example deploys a dashboard into an existing Amazon Managed Grafana (AMG) workspace to provide observability over an existing AMP workspace. It also deploys CloudWatch alarms for AMP usage service limits.
7+
8+
## Prerequisites
9+
10+
Ensure that you have the following tools installed locally:
11+
12+
1. [aws cli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
13+
2. [terraform](https://learn.hashicorp.com/tutorials/terraform/install-cli)
14+
15+
It is also required to have existing AMP and Grafana workspaces. These could be created through the [other example modules](../) in this repository.
16+
17+
## Setup
18+
19+
This example uses a local terraform state. If you need states to be saved remotely,
20+
on Amazon S3 for example, visit the [terraform remote states](https://www.terraform.io/language/state/remote) documentation
21+
22+
1. **Clone the repo using the command below**
23+
24+
```sh
25+
git clone https://github.com/aws-observability/terraform-aws-observability-accelerator.git
26+
```
27+
28+
2. **Initialize terraform**
29+
30+
```sh
31+
cd examples/amp-monitoring
32+
terraform init
33+
```
34+
35+
3. **AWS Region**
36+
37+
Specify the AWS Region where the resources will be deployed. Edit the `terraform.tfvars` file and modify `aws_region="..."`. You can also use environement variables `export TF_VAR_aws_region=xxx`.
38+
39+
4. **Amazon Managed Service for Prometheus workspace**
40+
41+
If you have an existing workspace, add `managed_prometheus_workspace_id=ws-xxx`
42+
or use an environment variable `export TF_VAR_managed_prometheus_workspace_id=ws-xxx`.
43+
44+
If you would like to create CloudWatch alarms for multiple workspaces in a region you can pass them in a comma seperated string.
45+
46+
`managed_prometheus_workspace_id = "ws-xxx,ws-xxx"`
47+
48+
You can use the following export command to create alarms for all of the workspaces in a region.
49+
50+
```sh
51+
export TF_VAR_managed_prometheus_workspace_id=$(aws amp list-workspaces --query 'workspaces[].workspaceId' --output text | sed -E 's/\t/,/g')
52+
```
53+
54+
5. **Amazon Managed Grafana workspace**
55+
56+
Use an existing workspace, add `managed_grafana_workspace_id=g-xxx`
57+
or use an environment variable `export TF_VAR_managed_grafana_workspace_id=g-xxx`.
58+
59+
6. <a name="apikey"></a> **Grafana API Key**
60+
61+
Amazon Managed Service for Grafana provides a control plane API for generating Grafana API keys. We will provide to Terraform
62+
a short lived API key to run the `apply` or `destroy` command.
63+
Ensure you have necessary IAM permissions (`CreateWorkspaceApiKey, DeleteWorkspaceApiKey`)
64+
65+
```sh
66+
export TF_VAR_grafana_api_key=`aws grafana create-workspace-api-key --key-name "observability-accelerator-$(date +%s)" --key-role ADMIN --seconds-to-live 1200 --workspace-id $TF_VAR_managed_grafana_workspace_id --query key --output text`
67+
```
68+
69+
## Deploy
70+
71+
```sh
72+
terraform apply -var-file=terraform.tfvars
73+
```
74+
75+
or if you had only setup environment variables, run
76+
77+
```sh
78+
terraform apply
79+
```
80+
81+
## Visualization
82+
83+
1. **Cloudwatch datasource on Grafana**
84+
85+
Open your Grafana workspace and under Configuration -> Data sources, you should see `aws-observability-accelerator-cloudwatch`. Open and click `Save & test`. You should see a notification confirming that the CloudWatch datasource is ready to be used on Grafana.
86+
87+
2. **Grafana dashboards**
88+
89+
Go to the Dashboards panel of your Grafana workspace. You should see a list of dashboards under the `AMP Monitoring Dashboards` folder.
90+
91+
Open the `AMP Accelerator Dashboard` to see a visualization of the AMP workspace.
92+
93+
<img width="1786" alt="Screen Shot 2022-10-11 at 2 16 17 PM" src="https://user-images.githubusercontent.com/97046295/196742772-fba1a5fb-dd38-445c-88a9-607f38994713.png">
94+
95+
3. **Amazon Managed Service for Prometheus CloudWatch Alarms.**
96+
97+
Open the CloudWatch console and click `Alarms` > `All Alarms` to review the service limit alarms.
98+
99+
<img width="1525" alt="image" src="https://user-images.githubusercontent.com/97046295/196742923-876e3b1c-6f2a-419d-ad39-9c057a0f7650.png">
100+
101+
In us-east-1 region an alarm is created for billing. This alarm utilizes anomaly detection to detect anomalies in the Estimated Charges billing metric.
102+
103+
<img width="1346" alt="image" src="https://user-images.githubusercontent.com/97046295/197042518-a98d69df-8f53-4a4a-afb8-f424d91da56f.png">
104+
105+
106+
107+
<!-- BEGIN_TF_DOCS -->
108+
## Requirements
109+
110+
| Name | Version |
111+
|------|---------|
112+
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.1.0, < 1.3.0 |
113+
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.0.0 |
114+
| <a name="requirement_grafana"></a> [grafana](#requirement\_grafana) | >= 1.25.0 |
115+
116+
## Providers
117+
118+
| Name | Version |
119+
|------|---------|
120+
| <a name="provider_aws"></a> [aws](#provider\_aws) | 4.36.1 |
121+
| <a name="provider_grafana"></a> [grafana](#provider\_grafana) | 1.30.0 |
122+
123+
## Modules
124+
125+
| Name | Source | Version |
126+
|------|--------|---------|
127+
| <a name="module_amp_monitor"></a> [amp\_monitor](#module\_amp\_monitor) | ../../modules/workloads/amp-monitoring | n/a |
128+
| <a name="module_billing"></a> [billing](#module\_billing) | ../../modules/Billing | n/a |
129+
130+
## Resources
131+
132+
| Name | Type |
133+
|------|------|
134+
| [grafana_folder.this](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/folder) | resource |
135+
| [aws_grafana_workspace.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/grafana_workspace) | data source |
136+
137+
## Inputs
138+
139+
| Name | Description | Type | Default | Required |
140+
|------|-------------|------|---------|:--------:|
141+
| <a name="input_aws_region"></a> [aws\_region](#input\_aws\_region) | AWS Region | `string` | n/a | yes |
142+
| <a name="input_grafana_api_key"></a> [grafana\_api\_key](#input\_grafana\_api\_key) | API key for authorizing the Grafana provider to make changes to Amazon Managed Grafana | `string` | n/a | yes |
143+
| <a name="input_managed_grafana_workspace_id"></a> [managed\_grafana\_workspace\_id](#input\_managed\_grafana\_workspace\_id) | Amazon Managed Grafana (AMG) workspace ID | `string` | n/a | yes |
144+
| <a name="input_managed_prometheus_workspace_id"></a> [managed\_prometheus\_workspace\_id](#input\_managed\_prometheus\_workspace\_id) | Amazon Managed Service for Prometheus Workspace ID to create Alarms for | `string` | n/a | yes |
145+
146+
## Outputs
147+
148+
| Name | Description |
149+
|------|-------------|
150+
| <a name="output_grafana_dashboards_folder_id"></a> [grafana\_dashboards\_folder\_id](#output\_grafana\_dashboards\_folder\_id) | Grafana folder ID for automatic dashboards. Required by workload modules |
151+
<!-- END_TF_DOCS -->

‎examples/managed-prometheus-monitoring/main.tf

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
provider "aws" {
2+
region = local.region
3+
}
4+
5+
provider "grafana" {
6+
url = local.amg_ws_endpoint
7+
auth = var.grafana_api_key
8+
}
9+
10+
data "aws_grafana_workspace" "this" {
11+
count = var.managed_grafana_workspace_id == "" ? 0 : 1
12+
workspace_id = var.managed_grafana_workspace_id
13+
}
14+
15+
locals {
16+
region = var.aws_region
17+
amg_ws_endpoint = "https://${data.aws_grafana_workspace.this[0].endpoint}"
18+
}
19+
20+
resource "grafana_folder" "this" {
21+
title = "Amazon Managed Prometheus monitoring dashboards"
22+
}
23+
24+
module "managed_prometheus_monitoring" {
25+
source = "../../modules/workloads/managed-prometheus-monitoring"
26+
dashboards_folder_id = resource.grafana_folder.this.id
27+
aws_region = local.region
28+
managed_prometheus_workspace_ids = var.managed_prometheus_workspace_ids
29+
}

‎examples/managed-prometheus-monitoring/outputs.tf

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
output "grafana_dashboard_urls" {
2+
description = "URLs for dashboards created"
3+
value = module.managed_prometheus_monitoring.grafana_dashboard_urls
4+
}

‎examples/managed-prometheus-monitoring/variables.tf

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
variable "grafana_api_key" {
2+
description = "API key for authorizing the Grafana provider to make changes to Amazon Managed Grafana"
3+
type = string
4+
sensitive = true
5+
}
6+
7+
variable "aws_region" {
8+
description = "AWS Region"
9+
type = string
10+
}
11+
12+
variable "managed_prometheus_workspace_ids" {
13+
description = "Amazon Managed Service for Prometheus Workspace IDs to create Alarms for"
14+
type = string
15+
}
16+
17+
variable "managed_grafana_workspace_id" {
18+
description = "Amazon Managed Grafana workspace ID"
19+
type = string
20+
}

‎examples/managed-prometheus-monitoring/versions.tf

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
terraform {
2+
required_version = ">= 1.1.0"
3+
4+
required_providers {
5+
aws = {
6+
source = "hashicorp/aws"
7+
version = ">= 4.0.0"
8+
}
9+
grafana = {
10+
source = "grafana/grafana"
11+
version = ">= 1.25.0"
12+
}
13+
}
14+
}

‎modules/workloads/infra/dashboards/cluster.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2905,4 +2905,4 @@
29052905
"uid": "efa86fd1d0c121a26444b636a3f509a8",
29062906
"version": 3,
29072907
"weekStart": ""
2908-
}
2908+
}

‎modules/workloads/infra/dashboards/kubelet.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2233,4 +2233,4 @@
22332233
"uid": "3138fa155d5915769fbded898ac09fd9",
22342234
"version": 19,
22352235
"weekStart": ""
2236-
}
2236+
}

‎modules/workloads/infra/dashboards/namespace-workloads.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2657,4 +2657,4 @@
26572657
"uid": "a87fb0d919ec0ea5f6543124e16c42a5",
26582658
"version": 2,
26592659
"weekStart": ""
2660-
}
2660+
}

‎modules/workloads/infra/dashboards/nodeexporter-nodes.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1279,4 +1279,4 @@
12791279
"uid": "v8yDYJqnz",
12801280
"version": 18,
12811281
"weekStart": ""
1282-
}
1282+
}

‎modules/workloads/infra/dashboards/nodes.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1480,4 +1480,4 @@
14801480
"uid": "200ac8fdbfbb74b39aff88118e4d1c2c",
14811481
"version": 7,
14821482
"weekStart": ""
1483-
}
1483+
}

0 commit comments

Comments
 (0)