Skip to content

Commit 6e96ab8

Browse files
csantanaprcsantanapr
authored
Karpenter pattern (#50)
* update argo workflows Signed-off-by: Carlos Santana <[email protected]> * add karpenter pattern Signed-off-by: Carlos Santana <[email protected]> * make eks 1.29 Signed-off-by: Carlos Santana <[email protected]> --------- Signed-off-by: Carlos Santana <[email protected]>
1 parent 54a1627 commit 6e96ab8

File tree

12 files changed

+813
-0
lines changed

12 files changed

+813
-0
lines changed

‎argocd/iac/terraform/examples/eks/karpenter/README.md‎

Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
# ArgoCD on Amazon EKS
2+
3+
This tutorial guides you through deploying an Amazon EKS cluster with addons configured via ArgoCD, employing the [GitOps Bridge Pattern](https://github.com/gitops-bridge-dev).
4+
5+
<img src="static/gitops-bridge.drawio.png" width=100%>
6+
7+
8+
The [GitOps Bridge Pattern](https://github.com/gitops-bridge-dev) enables Kubernetes administrators to utilize Infrastructure as Code (IaC) and GitOps tools for deploying Kubernetes Addons and Workloads. Addons often depend on Cloud resources that are external to the cluster. The configuration metadata for these external resources is required by the Addons' Helm charts. While IaC is used to create these cloud resources, it is not used to install the Helm charts. Instead, the IaC tool stores this metadata either within GitOps resources in the cluster or in a Git repository. The GitOps tool then extracts these metadata values and passes them to the Helm chart during the Addon installation process. This mechanism forms the bridge between IaC and GitOps, hence the term "GitOps Bridge."
9+
10+
Aditonal examples available on the [GitOps Bridge Pattern](https://github.com/gitops-bridge-dev):
11+
- [argocd-ingress](https://github.com/gitops-bridge-dev/gitops-bridge/tree/main/argocd/iac/terraform/examples/eks/argocd-ingress)
12+
- [aws-secrets-manager](https://github.com/gitops-bridge-dev/gitops-bridge/tree/main/argocd/iac/terraform/examples/eks/aws-secrets-manager)
13+
- [crossplane](https://github.com/gitops-bridge-dev/gitops-bridge/tree/main/argocd/iac/terraform/examples/eks/crossplane)
14+
- [external-secrets](https://github.com/gitops-bridge-dev/gitops-bridge/tree/main/argocd/iac/terraform/examples/eks/external-secrets)
15+
- [multi-cluster/distributed](https://github.com/gitops-bridge-dev/gitops-bridge/tree/main/argocd/iac/terraform/examples/eks/multi-cluster/distributed)
16+
- [multi-cluster/hub-spoke](https://github.com/gitops-bridge-dev/gitops-bridge/tree/main/argocd/iac/terraform/examples/eks/multi-cluster/hub-spoke)
17+
- [multi-cluster/hub-spoke-shared](https://github.com/gitops-bridge-dev/gitops-bridge/tree/main/argocd/iac/terraform/examples/eks/multi-cluster/hub-spoke-shared)
18+
- [private-git](https://github.com/gitops-bridge-dev/gitops-bridge/tree/main/argocd/iac/terraform/examples/eks/private-git)
19+
20+
21+
22+
## Prerequisites
23+
Before you begin, make sure you have the following command line tools installed:
24+
- git
25+
- terraform
26+
- kubectl
27+
- argocd
28+
29+
## Fork the Git Repositories
30+
31+
### Fork the Addon GitOps Repo
32+
1. Fork the git repository for addons [here](https://github.com/gitops-bridge-dev/gitops-bridge-argocd-control-plane-template).
33+
2. Update the following environment variables to point to your fork by changing the default values:
34+
```shell
35+
export TF_VAR_gitops_addons_org=https://github.com/gitops-bridge-dev
36+
export TF_VAR_gitops_addons_repo=gitops-bridge-argocd-control-plane-template
37+
```
38+
39+
### Fork the Workloads GitOps Repo
40+
1. Fork the git repository for this pattern [here](https://github.com/gitops-bridge-dev/gitops-bridge)
41+
2. Update the following environment variables to point to your fork by changing the default values:
42+
```shell
43+
export TF_VAR_gitops_workload_org=https://github.com/gitops-bridge-dev
44+
export TF_VAR_gitops_workload_repo=gitops-bridge
45+
```
46+
47+
## Deploy the EKS Cluster
48+
Initialize Terraform and deploy the EKS cluster:
49+
```shell
50+
terraform init
51+
terraform apply -auto-approve
52+
```
53+
Retrieve `kubectl` config, then execute the output command:
54+
```shell
55+
terraform output -raw configure_kubectl
56+
```
57+
58+
Terraform will add GitOps Bridge Metadata to the ArgoCD secret.
59+
The annotations contain metadata for the addons' Helm charts and ArgoCD ApplicationSets.
60+
```shell
61+
kubectl get secret -n argocd -l argocd.argoproj.io/secret-type=cluster -o json | jq '.items[0].metadata.annotations'
62+
```
63+
The output looks like the following:
64+
```json
65+
{
66+
"addons_repo_basepath": "",
67+
"addons_repo_path": "bootstrap/control-plane/addons",
68+
"addons_repo_revision": "main",
69+
"addons_repo_url": "https://github.com/gitops-bridge-dev/gitops-bridge-argocd-control-plane-template",
70+
"aws_account_id": "0123456789",
71+
"aws_cluster_name": "getting-started-gitops",
72+
"aws_load_balancer_controller_iam_role_arn": "arn:aws:iam::0123456789:role/alb-controller",
73+
"aws_load_balancer_controller_namespace": "kube-system",
74+
"aws_load_balancer_controller_service_account": "aws-load-balancer-controller-sa",
75+
"aws_region": "us-west-2",
76+
"aws_vpc_id": "vpc-001d3f00151bbb731",
77+
"cluster_name": "in-cluster",
78+
"environment": "dev",
79+
"workload_repo_basepath": "argocd/iac/terraform/examples/eks/",
80+
"workload_repo_path": "getting-started/k8s",
81+
"workload_repo_revision": "main",
82+
"workload_repo_url": "https://github.com/gitops-bridge-dev/gitops-bridge"
83+
}
84+
```
85+
The labels offer a straightforward way to enable or disable an addon in ArgoCD for the cluster.
86+
```shell
87+
kubectl get secret -n argocd -l argocd.argoproj.io/secret-type=cluster -o json | jq '.items[0].metadata.labels'
88+
```
89+
The output looks like the following:
90+
```json
91+
{
92+
"aws_cluster_name": "getting-started-gitops",
93+
"enable_argocd": "true",
94+
"enable_aws_load_balancer_controller": "true",
95+
"enable_metrics_server": "true",
96+
"kubernetes_version": "1.28",
97+
}
98+
```
99+
100+
## Deploy the Addons
101+
Bootstrap the addons using ArgoCD:
102+
```shell
103+
kubectl apply -f bootstrap/addons.yaml
104+
```
105+
106+
### Monitor GitOps Progress for Addons
107+
Wait until all the ArgoCD applications' `HEALTH STATUS` is `Healthy`. Use Crl+C to exit the `watch` command
108+
```shell
109+
watch kubectl get applications -n argocd
110+
```
111+
112+
### Verify the Addons
113+
Verify that the addons are ready:
114+
```shell
115+
kubectl get deployment -n kube-system \
116+
aws-load-balancer-controller \
117+
metrics-server
118+
```
119+
120+
## Access ArgoCD
121+
Access ArgoCD's UI, run the command from the output:
122+
```shell
123+
terraform output -raw access_argocd
124+
```
125+
126+
127+
## Deploy the Workloads
128+
Deploy a sample application located in [k8s/game-2048.yaml](k8s/game-2048.yaml) using ArgoCD:
129+
```shell
130+
kubectl apply -f bootstrap/workloads.yaml
131+
```
132+
133+
### Monitor GitOps Progress for Workloads
134+
Watch until the Workloads ArgoCD Application is `Healthy`
135+
```shell
136+
watch kubectl get -n argocd applications workloads
137+
```
138+
Wait until the ArgoCD Applications `HEALTH STATUS` is `Healthy`. Crl+C to exit the `watch` command
139+
140+
### Verify the Application
141+
Verify that the application configuration is present and the pod is running:
142+
```shell
143+
kubectl get -n game-2048 deployments,service,ep,ingress
144+
```
145+
Wait until the Ingress/game-2048 `MESSAGE` column value is `Successfully reconciled`. Crl+C to exit the `watch` command
146+
```shell
147+
kubectl events -n game-2048 --for ingress/game-2048 --watch
148+
```
149+
150+
151+
152+
### Access the Application using AWS Load Balancer
153+
Verify the application endpoint health using `curl`:
154+
```shell
155+
curl -I $(kubectl get -n game-2048 ingress game-2048 -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
156+
```
157+
The first line of the output should have `HTTP/1.1 200 OK`.
158+
159+
Retrieve the ingress URL for the application, and access in the browser:
160+
```shell
161+
echo "Application URL: http://$(kubectl get -n game-2048 ingress game-2048 -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')"
162+
```
163+
164+
165+
### Container Metrics
166+
Check the application's CPU and memory metrics:
167+
```shell
168+
kubectl top pods -n game-2048
169+
```
170+
171+
## Destroy the EKS Cluster
172+
To tear down all the resources and the EKS cluster, run the following command:
173+
```shell
174+
./destroy.sh
175+
```

‎argocd/iac/terraform/examples/eks/karpenter/bootstrap/addons.yaml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
apiVersion: argoproj.io/v1alpha1
2+
kind: ApplicationSet
3+
metadata:
4+
name: cluster-addons
5+
namespace: argocd
6+
spec:
7+
syncPolicy:
8+
preserveResourcesOnDeletion: true
9+
generators:
10+
- clusters: {}
11+
template:
12+
metadata:
13+
name: cluster-addons
14+
spec:
15+
project: default
16+
source:
17+
repoURL: '{{metadata.annotations.addons_repo_url}}'
18+
path: '{{metadata.annotations.addons_repo_basepath}}{{metadata.annotations.addons_repo_path}}'
19+
targetRevision: '{{metadata.annotations.addons_repo_revision}}'
20+
directory:
21+
recurse: true
22+
destination:
23+
namespace: argocd
24+
name: '{{name}}'
25+
syncPolicy:
26+
automated: {}

‎argocd/iac/terraform/examples/eks/karpenter/bootstrap/workloads.yaml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
apiVersion: argoproj.io/v1alpha1
2+
kind: ApplicationSet
3+
metadata:
4+
name: workloads
5+
namespace: argocd
6+
spec:
7+
syncPolicy:
8+
preserveResourcesOnDeletion: false
9+
generators:
10+
- clusters: {}
11+
template:
12+
metadata:
13+
name: workloads
14+
finalizers:
15+
# This finalizer is for demo purposes, in production remove apps using argocd CLI "argocd app delete workload --cascade"
16+
# When you invoke argocd app delete with --cascade, the finalizer is added automatically.
17+
- resources-finalizer.argocd.argoproj.io
18+
spec:
19+
project: default
20+
source:
21+
repoURL: '{{metadata.annotations.workload_repo_url}}'
22+
path: '{{metadata.annotations.workload_repo_basepath}}{{metadata.annotations.workload_repo_path}}'
23+
targetRevision: '{{metadata.annotations.workload_repo_revision}}'
24+
destination:
25+
name: '{{name}}'
26+
syncPolicy:
27+
automated:
28+
allowEmpty: true
29+
syncOptions:
30+
- CreateNamespace=true
31+
retry:
32+
backoff:
33+
duration: 1m
34+
limit: 60

‎argocd/iac/terraform/examples/eks/karpenter/destroy.sh‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
#!/bin/bash
2+
3+
set -uo pipefail
4+
5+
SCRIPTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
6+
ROOTDIR="$(cd ${SCRIPTDIR}/../..; pwd )"
7+
[[ -n "${DEBUG:-}" ]] && set -x
8+
9+
# Delete the Ingress/SVC before removing the addons
10+
TMPFILE=$(mktemp)
11+
terraform -chdir=$SCRIPTDIR output -raw configure_kubectl > "$TMPFILE"
12+
# check if TMPFILE contains the string "No outputs found"
13+
if [[ ! $(cat $TMPFILE) == *"No outputs found"* ]]; then
14+
source "$TMPFILE"
15+
kubectl delete -n argocd applicationset workloads
16+
kubectl delete -n argocd applicationset cluster-addons
17+
kubectl delete -n argocd applicationset addons-argocd
18+
kubectl delete -n argocd svc argo-cd-argocd-server
19+
fi
20+
21+
terraform destroy -target="module.gitops_bridge_bootstrap" -auto-approve
22+
terraform destroy -target="module.eks_blueprints_addons" -auto-approve
23+
terraform destroy -target="module.eks" -auto-approve
24+
terraform destroy -target="module.vpc" -auto-approve
25+
terraform destroy -auto-approve

‎argocd/iac/terraform/examples/eks/karpenter/k8s/example.yaml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
---
2+
apiVersion: apps/v1
3+
kind: Deployment
4+
metadata:
5+
name: inflate
6+
namespace: default
7+
spec:
8+
replicas: 0
9+
selector:
10+
matchLabels:
11+
app: inflate
12+
template:
13+
metadata:
14+
labels:
15+
app: inflate
16+
spec:
17+
terminationGracePeriodSeconds: 0
18+
containers:
19+
- name: inflate
20+
image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
21+
resources:
22+
requests:
23+
cpu: 1

‎argocd/iac/terraform/examples/eks/karpenter/k8s/karpenter.yaml‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
---
2+
apiVersion: karpenter.k8s.aws/v1beta1
3+
kind: EC2NodeClass
4+
metadata:
5+
name: default
6+
spec:
7+
amiFamily: AL2
8+
role: karpenter-karpenter
9+
subnetSelectorTerms:
10+
- tags:
11+
karpenter.sh/discovery: karpenter
12+
securityGroupSelectorTerms:
13+
- tags:
14+
karpenter.sh/discovery: karpenter
15+
tags:
16+
karpenter.sh/discovery: karpenter
17+
---
18+
apiVersion: karpenter.sh/v1beta1
19+
kind: NodePool
20+
metadata:
21+
name: default
22+
spec:
23+
template:
24+
spec:
25+
nodeClassRef:
26+
name: default
27+
requirements:
28+
- key: "karpenter.k8s.aws/instance-category"
29+
operator: In
30+
values: ["c", "m", "r"]
31+
- key: "karpenter.k8s.aws/instance-cpu"
32+
operator: In
33+
values: ["4", "8", "16", "32"]
34+
- key: "karpenter.k8s.aws/instance-hypervisor"
35+
operator: In
36+
values: ["nitro"]
37+
- key: "karpenter.k8s.aws/instance-generation"
38+
operator: Gt
39+
values: ["2"]
40+
limits:
41+
cpu: 1000
42+
disruption:
43+
consolidationPolicy: WhenEmpty
44+
consolidateAfter: 30s

0 commit comments

Comments
 (0)