Skip to content

Commit daed34d

Browse files
authored
Compose EKS monitoring modules (#115)
* Move modules around * Update amp billing source * Merge Java monitoring to EKS * Update docs * Merge nginx pattern * Pre-commit * Add save and test URL output * Move EKS dependencies to EKS monitoring module * update docs * Update examples and docs * Add java doc * Add NGINX doc * Update nginx doc * Fix amp monitoring example path * Fix pre-commit * Todo: move to main after merge * Update docs, fix tags
1 parent fe83579 commit daed34d

File tree

99 files changed

+887
-1477
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

99 files changed

+887
-1477
lines changed

README.md

Lines changed: 14 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,15 @@ visit the [Amazon EKS cluster monitoring documentation](https://aws-observabilit
2828
The sections below demonstrate how you can leverage AWS Observability Accelerator
2929
to enable monitoring to an existing EKS cluster.
3030

31+
### v2.x changes
32+
33+
v2+ releases introduces couple of breaking changes compared to previous versions:
34+
35+
- `modules/workloads/infra` module moves to `modules/eks-monitoring`
36+
- All EKS configuration options moves from the base module to the `eks-monitoring` module
37+
- All EKS workload modules `modules/workloads/{java,nginx}` merge into `eks-monitoring` as configuration options (patterns), see [examples](./examples) to provide a more complete visiblity.
38+
- All examples have been updated to reflect these changes
39+
3140
### Base Module
3241

3342
The base module allows you to configure the AWS Observability services for your cluster and
@@ -38,15 +47,15 @@ and ADOT Operator deployed for you and ready to receive your data.
3847
The base module serve as an anchor to the workload modules and cannot run on its own.
3948

4049
```hcl
41-
module "eks_observability_accelerator" {
50+
module "aws_observability_accelerator" {
4251
# use release tags and check for the latest versions
4352
# https://github.com/aws-observability/terraform-aws-observability-accelerator/releases
4453
source = "github.com/aws-observability/terraform-aws-observability-accelerator?ref=v1.6.1"
4554
4655
aws_region = "eu-west-1"
4756
eks_cluster_id = "my-eks-cluster"
4857
49-
# As Grafana shares a different lifecycle, it's best to use an existing workspace.
58+
# As Grafana shares a different lifecycle, we recommend using an existing workspace.
5059
managed_grafana_workspace_id = var.managed_grafana_workspace_id
5160
grafana_api_key = var.grafana_api_key
5261
}
@@ -55,7 +64,7 @@ module "eks_observability_accelerator" {
5564
You can optionally reuse an existing Amazon Managed Servce for Prometheus Workspace:
5665

5766
```hcl
58-
module "eks_observability_accelerator" {
67+
module "aws_observability_accelerator" {
5968
# use release tags and check for the latest versions
6069
# https://github.com/aws-observability/terraform-aws-observability-accelerator/releases
6170
source = "github.com/aws-observability/terraform-aws-observability-accelerator?ref=v1.6.1"
@@ -78,10 +87,9 @@ View all the configuration options in the module documentation below.
7887

7988
### Workload modules
8089

81-
[Workloads modules](./modules/workloads) are provided, which essentially provide curated
90+
[Workloads modules](./modules) are provided, which essentially provide curated
8291
metrics collection, alerting rule and Grafana dashboards.
8392

84-
8593
#### Infrastructure monitoring
8694

8795
```hcl
@@ -143,7 +151,6 @@ If you are interested in contributing, see the [Contribution guide](https://gith
143151
| Name | Source | Version |
144152
|------|--------|---------|
145153
| <a name="module_managed_grafana"></a> [managed\_grafana](#module\_managed\_grafana) | terraform-aws-modules/managed-service-grafana/aws | ~> 1.3 |
146-
| <a name="module_operator"></a> [operator](#module\_operator) | ./modules/add-ons/adot-operator | n/a |
147154

148155
## Resources
149156

@@ -153,26 +160,18 @@ If you are interested in contributing, see the [Contribution guide](https://gith
153160
| [aws_prometheus_workspace.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/prometheus_workspace) | resource |
154161
| [grafana_data_source.amp](https://registry.terraform.io/providers/grafana/grafana/1.25.0/docs/resources/data_source) | resource |
155162
| [grafana_folder.this](https://registry.terraform.io/providers/grafana/grafana/1.25.0/docs/resources/folder) | resource |
156-
| [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source |
157-
| [aws_eks_cluster.eks_cluster](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/eks_cluster) | data source |
158163
| [aws_grafana_workspace.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/grafana_workspace) | data source |
159-
| [aws_partition.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/partition) | data source |
160164
| [aws_region.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source |
161165

162166
## Inputs
163167

164168
| Name | Description | Type | Default | Required |
165169
|------|-------------|------|---------|:--------:|
166170
| <a name="input_aws_region"></a> [aws\_region](#input\_aws\_region) | AWS Region | `string` | n/a | yes |
167-
| <a name="input_eks_cluster_id"></a> [eks\_cluster\_id](#input\_eks\_cluster\_id) | Name of the EKS cluster | `string` | n/a | yes |
168171
| <a name="input_enable_alertmanager"></a> [enable\_alertmanager](#input\_enable\_alertmanager) | Creates Amazon Managed Service for Prometheus AlertManager for all workloads | `bool` | `false` | no |
169-
| <a name="input_enable_amazon_eks_adot"></a> [enable\_amazon\_eks\_adot](#input\_enable\_amazon\_eks\_adot) | Enables the ADOT Operator on the EKS Cluster | `bool` | `true` | no |
170-
| <a name="input_enable_cert_manager"></a> [enable\_cert\_manager](#input\_enable\_cert\_manager) | Allow reusing an existing installation of cert-manager | `bool` | `true` | no |
171172
| <a name="input_enable_managed_grafana"></a> [enable\_managed\_grafana](#input\_enable\_managed\_grafana) | Creates a new Amazon Managed Grafana Workspace | `bool` | `true` | no |
172173
| <a name="input_enable_managed_prometheus"></a> [enable\_managed\_prometheus](#input\_enable\_managed\_prometheus) | Creates a new Amazon Managed Service for Prometheus Workspace | `bool` | `true` | no |
173174
| <a name="input_grafana_api_key"></a> [grafana\_api\_key](#input\_grafana\_api\_key) | Grafana API key for the Amazon Managed Grafana workspace | `string` | n/a | yes |
174-
| <a name="input_irsa_iam_permissions_boundary"></a> [irsa\_iam\_permissions\_boundary](#input\_irsa\_iam\_permissions\_boundary) | IAM permissions boundary for IRSA roles | `string` | `null` | no |
175-
| <a name="input_irsa_iam_role_path"></a> [irsa\_iam\_role\_path](#input\_irsa\_iam\_role\_path) | IAM role path for IRSA roles | `string` | `"/"` | no |
176175
| <a name="input_managed_grafana_workspace_id"></a> [managed\_grafana\_workspace\_id](#input\_managed\_grafana\_workspace\_id) | Amazon Managed Grafana Workspace ID | `string` | `""` | no |
177176
| <a name="input_managed_prometheus_workspace_id"></a> [managed\_prometheus\_workspace\_id](#input\_managed\_prometheus\_workspace\_id) | Amazon Managed Service for Prometheus Workspace ID | `string` | `""` | no |
178177
| <a name="input_managed_prometheus_workspace_region"></a> [managed\_prometheus\_workspace\_region](#input\_managed\_prometheus\_workspace\_region) | Region where Amazon Managed Service for Prometheus is deployed | `string` | `null` | no |
@@ -183,9 +182,8 @@ If you are interested in contributing, see the [Contribution guide](https://gith
183182
| Name | Description |
184183
|------|-------------|
185184
| <a name="output_aws_region"></a> [aws\_region](#output\_aws\_region) | AWS Region |
186-
| <a name="output_eks_cluster_id"></a> [eks\_cluster\_id](#output\_eks\_cluster\_id) | EKS Cluster Id |
187-
| <a name="output_eks_cluster_version"></a> [eks\_cluster\_version](#output\_eks\_cluster\_version) | EKS Cluster version |
188185
| <a name="output_grafana_dashboards_folder_id"></a> [grafana\_dashboards\_folder\_id](#output\_grafana\_dashboards\_folder\_id) | Grafana folder ID for automatic dashboards. Required by workload modules |
186+
| <a name="output_grafana_prometheus_datasource_test"></a> [grafana\_prometheus\_datasource\_test](#output\_grafana\_prometheus\_datasource\_test) | Grafana save & test URL for Amazon Managed Prometheus workspace |
189187
| <a name="output_managed_grafana_workspace_endpoint"></a> [managed\_grafana\_workspace\_endpoint](#output\_managed\_grafana\_workspace\_endpoint) | Amazon Managed Grafana workspace endpoint |
190188
| <a name="output_managed_grafana_workspace_id"></a> [managed\_grafana\_workspace\_id](#output\_managed\_grafana\_workspace\_id) | Amazon Managed Grafana workspace ID |
191189
| <a name="output_managed_prometheus_workspace_endpoint"></a> [managed\_prometheus\_workspace\_endpoint](#output\_managed\_prometheus\_workspace\_endpoint) | Amazon Managed Prometheus workspace endpoint |

docs/concepts.md

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,18 @@ you need to track changes as part of a Git repository or CI/CD pipeline.
3131
!!! warning
3232
When using `tfvars` files, always be careful to not store and commit any secrets (keys, passwords, ...)
3333

34+
35+
## v2.x changes
36+
37+
v2.x [releases](https://github.com/aws-observability/terraform-aws-observability-accelerator/releases) introduce
38+
couple of breaking changes compared to previous versions:
39+
40+
- `modules/workloads/infra` module moves to `modules/eks-monitoring`
41+
- EKS configuration options moves from the base module to the `eks-monitoring` module
42+
- EKS workload modules **java,nginx** merge into `eks-monitoring` as configuration options (patterns),
43+
see [examples](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/examples)
44+
- Examples have been updated to reflect these changes
45+
3446
## Base module
3547

3648
The base module allows you to configure the AWS Observability services for your cluster and
@@ -41,15 +53,15 @@ and ADOT Operator deployed for you and ready to receive your data.
4153
The base module serve as an anchor to the workload modules and cannot run on its own.
4254

4355
```hcl
44-
module "eks_observability_accelerator" {
56+
module "aws_observability_accelerator" {
4557
# use release tags and check for the latest versions
4658
# https://github.com/aws-observability/terraform-aws-observability-accelerator/releases
4759
source = "github.com/aws-observability/terraform-aws-observability-accelerator?ref=v1.6.1"
4860
4961
aws_region = "eu-west-1"
5062
eks_cluster_id = "my-eks-cluster"
5163
52-
# As Grafana shares a different lifecycle, it's best to use an existing workspace.
64+
# As Grafana shares a different lifecycle, we recommend using an existing workspace.
5365
managed_grafana_workspace_id = var.managed_grafana_workspace_id
5466
grafana_api_key = var.grafana_api_key
5567
}
@@ -58,7 +70,7 @@ module "eks_observability_accelerator" {
5870
You can optionally reuse an existing Amazon Managed Service for Prometheus Workspace:
5971

6072
```hcl
61-
module "eks_observability_accelerator" {
73+
module "aws_observability_accelerator" {
6274
# use release tags and check for the latest versions
6375
# https://github.com/aws-observability/terraform-aws-observability-accelerator/releases
6476
source = "github.com/aws-observability/terraform-aws-observability-accelerator?ref=v1.6.1"
@@ -83,7 +95,7 @@ View all the configuration options in the [module's documentation](https://githu
8395

8496
Workloads modules are focused Terraform modules provided in this repository. They essentially provide curated metrics collection, alerts and Grafana dashboards according to the use case. Most of those modules require the base module.
8597

86-
You can check the full workload modules list and their documentation [here](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/modules/workloads).
98+
You can check the full workload modules list and their documentation [here](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/modules/).
8799

88100
All the modules come with end-to-end deployable examples.
89101

docs/eks/destroy.md

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,11 +15,9 @@ terraform destroy
1515
To remove resources from your Terraform state, run
1616

1717
```bash
18-
# grafana workspace
19-
terraform state rm "module.eks_observability_accelerator.module.managed_grafana[0].aws_grafana_workspace.this[0]"
20-
2118
# prometheus workspace
2219
terraform state rm "module.eks_observability_accelerator.aws_prometheus_workspace.this[0]"
2320
```
2421

25-
> **Note:** To view all the features proposed by this module, visit the [module documentation](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/modules/workloads/infra).
22+
!!! note
23+
To view all the features proposed by this module, visit the [module documentation](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/modules/workloads/infra).

docs/eks/index.md

Lines changed: 15 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
This example demonstrates how to monitor your Amazon Elastic Kubernetes Service
44
(Amazon EKS) cluster with the Observability Accelerator's EKS
5-
[infrastructure module](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/modules/workloads/infra).
5+
[infrastructure module](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/feat/modules-composition/modules/eks-monitoring).
66

77
Monitoring Amazon Elastic Kubernetes Service (Amazon EKS) for metrics has two categories:
88
the control plane and the Amazon EKS nodes (with Kubernetes objects).
@@ -72,22 +72,15 @@ aws amp create-workspace --alias observability-accelerator --query '.workspaceId
7272

7373
### 5. Amazon Managed Grafana workspace
7474

75-
To run this example you need an Amazon Managed Grafana workspace. If you have an existing workspace, edit and run:
75+
To run this example you need an Amazon Managed Grafana workspace. If you have an existing workspace, create an environment variable as described below.
76+
To create a new workspace, visit our Amazon Managed Grafana [documentation](https://docs.aws.amazon.com/grafana/latest/userguide/getting-started-with-AMG.html).
77+
Make sure to provide the workspace with Amazon Managed Service for Prometheus read permissions.
7678

77-
```bash
78-
export TF_VAR_managed_grafana_workspace_id=g-xxx
79-
```
80-
81-
To create a new one, within this example's Terraform state (sharing the same lifecycle with all the
82-
other resources created by Terraform):
83-
84-
- Edit main.tf and set `enable_managed_grafana = true`
85-
- Run
79+
!!! note
80+
For the URL `https://g-xyz.grafana-workspace.eu-central-1.amazonaws.com`, the workspace ID would be `g-xyz`
8681

8782
```bash
88-
terraform init
89-
terraform apply -target "module.eks_observability_accelerator.module.managed_grafana[0].aws_grafana_workspace.this[0]"
90-
export TF_VAR_managed_grafana_workspace_id=$(terraform output --raw managed_grafana_workspace_id)
83+
export TF_VAR_managed_grafana_workspace_id=g-xxx
9184
```
9285

9386
### 6. Grafana API Key
@@ -114,7 +107,13 @@ terraform apply
114107

115108
1. Prometheus datasource on Grafana
116109

117-
Open your Grafana workspace and under Configuration -> Data sources, you should see `aws-observability-accelerator`. Open and click `Save & test`. You should see a notification confirming that the Amazon Managed Service for Prometheus workspace is ready to be used on Grafana.
110+
Make sure to open the link in the output. After a successful deployment, this will open
111+
the Prometheus datasource configuration on Grafana.
112+
Click `Save & test` and you should see a notification confirming that the Amazon Managed Service for Prometheus workspace is ready to be used on Grafana.
113+
114+
```bash
115+
terraform output grafana_prometheus_datasource_test
116+
```
118117

119118
2. Grafana dashboards
120119

@@ -126,7 +125,7 @@ Open a specific dashboard and you should be able to view its visualization
126125

127126
<img width="2056" alt="cluster headlines" src="https://user-images.githubusercontent.com/10175027/199110753-9bc7a9b7-1b45-4598-89d3-32980154080e.png">
128127

129-
2. Amazon Managed Service for Prometheus rules and alerts
128+
3. Amazon Managed Service for Prometheus rules and alerts
130129

131130
Open the Amazon Managed Service for Prometheus console and view the details of your workspace. Under the `Rules management` tab, you should find new rules deployed.
132131

0 commit comments

Comments
 (0)