Skip to content

Commit d7daeb8

Browse files
veekalybonclay7
veekaly
and
bonclay7
authored
EKS Cross Account Observability using central AMP (#213)
* Added example for multi-cluster eks-monitoring and made changes to eks-monitoring module to allow cross-cluster IRSA * Updated eks-monitoring module's README to add the variable description * Hard-coded grafana license type in eks-cross-cluster-with-amp/main.tf * Added README page for eks-cross-account-with-central-amp * Extracted out the eks/amg creation and modified it to use existing resources * Update README.md to add multiaccount dashboard png * Updated README.md and multiaccount.md to add eks multiaccount png * Updated eks-cross-cluster-with-amp example to disable dashboard creation for cluster 2 * Pre commit changed committed * Fixed cross-account-observability docs and README.md and added variable for amp_workpace_alias * Removed extra spacing in multiaccount.md and added precommit suggested changes * Modified cross-account-observability example to change cross-account-amp-role to snake_case * Capitalized Terraform string in multiaccount.md and converted iam-role-attach to snake_case --------- Co-authored-by: Rodrigue Koffi <[email protected]>
1 parent a05e82b commit d7daeb8

File tree

16 files changed

+681
-3
lines changed

16 files changed

+681
-3
lines changed

‎docs/eks/multiaccount.md

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
# AWS EKS Cross Account Observability
2+
3+
This example shows how to use the [AWS Observability Accelerator](https://github.com/aws-observability/terraform-aws-observability-accelerator), with two or more EKS clusters in multiple AWS accounts and verify the collected metrics from all the clusters in the dashboards of a common `Amazon Managed Grafana` workspace in a central monitoring account.
4+
5+
## Prerequisites
6+
7+
#### 1. Cross Account IAM access
8+
9+
In order to create/modify resources across multiple AWS accounts, this Terraform example implements the cross-account IAM role assumption. You will need separate IAM roles in all 3 AWS accounts, and each of these IAM roles should have the below specified trust-relationship so that your local AWS user/role will be able to assume them during the terraform execution.
10+
11+
```
12+
{
13+
"Version": "2012-10-17",
14+
"Statement": [
15+
{
16+
"Effect": "Allow",
17+
"Principal": {
18+
"AWS": "<local-aws-user/role-arn>"
19+
},
20+
"Action": "sts:AssumeRole",
21+
"Condition": {}
22+
}
23+
]
24+
}
25+
```
26+
27+
!!! note
28+
The IAM roles in Account 1 and Account 2 (EKS cluster accounts) should have permissions to perform kubernetes API operations against your EKS clusters. For more info, please review documentation for [enabling IAM principal access to your clusters](https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html)
29+
30+
#### 2. EKS clusters in multiple AWS Accounts
31+
32+
Using the example [eks-cluster-with-vpc](https://aws-observability.github.io/terraform-aws-observability-accelerator/helpers/new-eks-cluster/), create two EKS clusters with the below names in two different AWS accounts:
33+
34+
1. `eks-cluster-1` (Account 1)
35+
36+
2. `eks-cluster-2` (Account 2)
37+
38+
Update the cluster names and their corresponding region names in the `variables.tf` file along with the corresponding IAM role ARNs that can be assumed by terraform to perform cross-account API operations.
39+
40+
#### 3. Amazon Managed Grafana (AMG) workspace
41+
42+
To run this example you need an existing Amazon Managed Grafana (AMG) workspace. If not, you can create a new AMG workspace by following the [Getting Started with Amazon Managed Grafana](https://docs.aws.amazon.com/grafana/latest/userguide/getting-started-with-AMG.html) documentation.
43+
44+
Add the Grafana Workspace ID and its corresponding region name in the `variables.tf` file along with the corresponding IAM role ARN that can be assumed by terraform to perform cross-account API operations.
45+
46+
!!! note
47+
You can obtain the AMG Workspace ID based on its URL. For the URL `https://g-xyz.grafana-workspace.eu-central-1.amazonaws.com`, the workspace ID would be `g-xyz`
48+
49+
50+
## Setup
51+
52+
#### 1. Download sources and initialize Terraform
53+
54+
55+
```sh
56+
57+
git clone https://github.com/aws-observability/terraform-aws-observability-accelerator.git
58+
59+
cd terraform-aws-observability-accelerator/examples/eks-cross-account-with-central-amp
60+
61+
terraform init
62+
63+
```
64+
65+
#### 2. Deploy
66+
67+
By looking at the `variables.tf`, you will notice there are two EKS clusters targeted for deployment by the names/ids:
68+
69+
1. `eks-cluster-1`
70+
71+
2. `eks-cluster-2`
72+
73+
While installing the observability settings for the EKS cluster specified in variable `cluster_one.name`, Terraform also sets up:
74+
75+
* Creates an `Amazon Managed Prometheus Workspace`
76+
77+
* Dashboard folder and files in provided `Amazon Managed Grafana Workspace`
78+
79+
80+
!!! warning
81+
To override the defaults, create a `terraform.tfvars` and change the default values of the variables.
82+
83+
84+
85+
Run the following command to deploy
86+
87+
```sh
88+
89+
terraform apply --auto-approve
90+
91+
```
92+
93+
94+
95+
## Verifying Multi Account Observability
96+
97+
98+
99+
One you have successfully run the above setup, you should be able to see dashboards similar to the images shown below in `Amazon Managed Grafana` workspace.
100+
101+
102+
103+
You will notice that you are able to use the `cluster` dropdown to filter the dashboards to metrics collected from a specific EKS cluster.
104+
105+
![eks-cross-account-1](https://github.com/veekaly/terraform-aws-observability-accelerator/assets/119073483/96a68eb1-4fb7-4a6b-bd4a-15f4f6ac7565)
106+
![eks-cross-account-2](https://github.com/veekaly/terraform-aws-observability-accelerator/assets/119073483/1373b834-1082-4a63-98b9-2b90fb32eada)
107+
108+
109+
## Cleanup
110+
111+
To clean up entirely, run the following command:
112+
113+
114+
115+
```sh
116+
117+
terraform destroy --auto-approve
118+
119+
```

‎examples/eks-cross-account-with-central-amp/README.md

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
# AWS EKS Cross Account Observability
2+
3+
4+
5+
This example shows how to use the [AWS Observability Accelerator](https://github.com/aws-observability/terraform-aws-observability-accelerator), with two or more EKS cluster in multiple AWS accounts and verify the collected metrics from all the clusters in the dashboards of a common `Amazon Managed Grafana` workspace in a central monitoring account.
6+
7+
8+
9+
## Prerequisites
10+
11+
#### 1. Cross Account IAM access
12+
13+
In order to create/modify resources across multiple AWS accounts, this Terraform example implements the cross-account IAM role assumption. You will need separate IAM roles in all 3 AWS accounts, and each of these IAM roles should have the below specified trust-relationship so that your local AWS user/role will be able to assume them during the terraform execution.
14+
15+
```
16+
{
17+
"Version": "2012-10-17",
18+
"Statement": [
19+
{
20+
"Effect": "Allow",
21+
"Principal": {
22+
"AWS": "<local-aws-user/role-arn>"
23+
},
24+
"Action": "sts:AssumeRole",
25+
"Condition": {}
26+
}
27+
]
28+
}
29+
```
30+
31+
> [!NOTE]
32+
> The IAM roles in Account 1 and Account 2 (EKS cluster accounts) should have permissions to perform kubernetes API operations against your EKS clusters. For more info, please review documentation for [enabling IAM principal access to your clusters](https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html)
33+
34+
35+
#### 2. EKS clusters in multiple AWS Accounts
36+
37+
Using the example [eks-cluster-with-vpc](../../examples/eks-cluster-with-vpc/), create two EKS clusters with the below names in two different AWS accounts:
38+
39+
1. `eks-cluster-1` (Account 1)
40+
41+
2. `eks-cluster-2` (Account 2)
42+
43+
Update the cluster names and their corresponding region names in the `variables.tf` file along with the corresponding IAM role ARNs that can be assumed by terraform to perform cross-account API operations.
44+
45+
#### 3. Amazon Managed Grafana (AMG) workspace
46+
47+
To run this example you need an existing Amazon Managed Grafana (AMG) workspace. If not, you can create a new AMG workspace by following the [Getting Started with Amazon Managed Grafana](https://docs.aws.amazon.com/grafana/latest/userguide/getting-started-with-AMG.html) documentation.
48+
49+
Add the Grafana Workspace ID and its corresponding region name in the `variables.tf` file along with the corresponding IAM role ARN that can be assumed by terraform to perform cross-account API operations.
50+
51+
!!! note
52+
53+
You can obtain the AMG Workspace ID based on its URL. For the URL `https://g-xyz.grafana-workspace.eu-central-1.amazonaws.com`, the workspace ID would be `g-xyz`
54+
55+
56+
## Setup
57+
58+
#### 1. Download sources and initialize Terraform
59+
60+
61+
```sh
62+
63+
git clone https://github.com/aws-observability/terraform-aws-observability-accelerator.git
64+
65+
cd terraform-aws-observability-accelerator/examples/eks-cross-account-with-central-amp
66+
67+
terraform init
68+
69+
```
70+
71+
#### 2. Deploy
72+
73+
By looking at the `variables.tf`, you will notice there are two EKS clusters targeted for deployment by the names/ids:
74+
75+
1. `eks-cluster-1`
76+
77+
2. `eks-cluster-2`
78+
79+
While installing the observability settings for the EKS cluster specified in variable `cluster_one.name`, Terraform also sets up:
80+
81+
* Creates an `Amazon Managed Prometheus Workspace`
82+
83+
* Dashboard folder and files in provided `Amazon Managed Grafana Workspace`
84+
85+
86+
!!! warning
87+
88+
To override the defaults, create a `terraform.tfvars` and change the default values of the variables.
89+
90+
91+
92+
Run the following command to deploy
93+
94+
```sh
95+
96+
terraform apply --auto-approve
97+
98+
```
99+
100+
101+
102+
## Verifying Multi Account Observability
103+
104+
105+
106+
One you have successfully run the above setup, you should be able to see dashboards similar to the images shown below in `Amazon Managed Grafana` workspace.
107+
108+
109+
110+
You will notice that you are able to use the `cluster` dropdown to filter the dashboards to metrics collected from a specific EKS cluster.
111+
112+
![eks-cross-account-1](https://github.com/veekaly/terraform-aws-observability-accelerator/assets/119073483/96a68eb1-4fb7-4a6b-bd4a-15f4f6ac7565)
113+
![eks-cross-account-2](https://github.com/veekaly/terraform-aws-observability-accelerator/assets/119073483/1373b834-1082-4a63-98b9-2b90fb32eada)
114+
115+
116+
## Cleanup
117+
118+
To clean up entirely, run the following command:
119+
120+
121+
122+
```sh
123+
124+
terraform destroy --auto-approve
125+
126+
```

‎examples/eks-cross-account-with-central-amp/data.tf

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
data "aws_eks_cluster_auth" "eks_one" {
2+
name = var.cluster_one.name
3+
provider = aws.eks_cluster_one
4+
}
5+
6+
data "aws_eks_cluster_auth" "eks_two" {
7+
name = var.cluster_two.name
8+
provider = aws.eks_cluster_two
9+
}
10+
11+
data "aws_eks_cluster" "eks_one" {
12+
name = var.cluster_one.name
13+
provider = aws.eks_cluster_one
14+
}
15+
16+
data "aws_eks_cluster" "eks_two" {
17+
name = var.cluster_two.name
18+
provider = aws.eks_cluster_two
19+
}

‎examples/eks-cross-account-with-central-amp/iam.tf

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
data "aws_caller_identity" "monitoring" {
2+
provider = aws.central_monitoring
3+
}
4+
5+
resource "aws_iam_policy" "irsa_assume_role_policy_one" {
6+
provider = aws.eks_cluster_one
7+
name = "${var.cluster_one.name}-irsa_assume_role_policy"
8+
path = "/"
9+
description = "This role allows the IRSA role to assume the cross-account role for AMP access"
10+
11+
policy = jsonencode({
12+
Version = "2012-10-17"
13+
Statement = [
14+
{
15+
Action = [
16+
"sts:AssumeRole",
17+
]
18+
Effect = "Allow"
19+
Resource = "arn:aws:iam::${data.aws_caller_identity.monitoring.account_id}:role/${local.amp_workspace_alias}-role-for-cross-account"
20+
},
21+
]
22+
})
23+
}
24+
25+
resource "aws_iam_policy" "irsa_assume_role_policy_two" {
26+
provider = aws.eks_cluster_two
27+
name = "${var.cluster_two.name}-irsa_assume_role_policy"
28+
path = "/"
29+
description = "This role allows the IRSA role to assume the cross-account role for AMP access"
30+
31+
policy = jsonencode({
32+
Version = "2012-10-17"
33+
Statement = [
34+
{
35+
Action = [
36+
"sts:AssumeRole",
37+
]
38+
Effect = "Allow"
39+
Resource = "arn:aws:iam::${data.aws_caller_identity.monitoring.account_id}:role/${local.amp_workspace_alias}-role-for-cross-account"
40+
},
41+
]
42+
})
43+
}
44+
45+
resource "aws_iam_role" "cross_account_amp_role" {
46+
provider = aws.central_monitoring
47+
name = "${local.amp_workspace_alias}-role-for-cross-account"
48+
49+
assume_role_policy = <<EOF
50+
{
51+
"Version": "2012-10-17",
52+
"Statement": [
53+
{
54+
"Effect": "Allow",
55+
"Principal": {
56+
"AWS": [
57+
"${module.eks_monitoring_one.adot_irsa_arn}",
58+
"${module.eks_monitoring_two.adot_irsa_arn}"
59+
]
60+
},
61+
"Action": "sts:AssumeRole",
62+
"Condition": {}
63+
}
64+
]
65+
}
66+
EOF
67+
}
68+
69+
resource "aws_iam_role_policy_attachment" "role_attach" {
70+
provider = aws.central_monitoring
71+
role = aws_iam_role.cross_account_amp_role.name
72+
policy_arn = "arn:aws:iam::aws:policy/AmazonPrometheusRemoteWriteAccess"
73+
}

0 commit comments

Comments
 (0)