Skip to content

Commit 0fe0bdb

Browse files
fiunchinhoAndiDog
fiunchinho
and
AndiDog
authored
Use helmrelease for the karpenter-bundle (#1422)
* Use helmrelease for the karpenter-bundle * Apply suggestions from code review Co-authored-by: Andreas Sommer <[email protected]> * Make baseDomain required * Bump flux HelmRelease apiversion to v2 * Update changelog --------- Co-authored-by: Andreas Sommer <[email protected]>
1 parent 6d65b78 commit 0fe0bdb

9 files changed

+152
-86
lines changed

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1111

1212
- Chart: Update `cluster` to v4.0.2.
1313
- The container registry passed as value to default apps is set to `gsoci.azurecr.io`, regardless of the cluster region. The mirroring feature of `containerd` will make sure the right registry is used.
14+
- Switch to HelmReleases to install `karpenter` and `karpenter-crossplane-resources` charts.
15+
- Bump flux `HelmReleases` api version to v2.
1416

1517
### Removed
1618

‎helm/cluster-aws/README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -188,6 +188,13 @@ Configuration of apps that are part of the cluster.
188188
| `global.apps.k8sDnsNodeCache.extraConfigs[*].name` | **Name** - Name of the config map or secret. The object must exist in the same namespace as the cluster App.|**Type:** `[string]`<br/>|
189189
| `global.apps.k8sDnsNodeCache.extraConfigs[*].priority` | **Priority**|**Type:** `[integer]`<br/>**Default:** `25`|
190190
| `global.apps.k8sDnsNodeCache.values` | **Config map** - Helm Values to be passed to the app as user config.|**Type:** `[object]`<br/>|
191+
| `global.apps.karpenter` | **App resource** - Configuration of a default app that is part of the cluster and is deployed as an App resource.|**Type:** `[object]`<br/>|
192+
| `global.apps.karpenter.extraConfigs` | **Extra config maps or secrets** - Extra config maps or secrets that will be used to customize to the app. The desired values must be under configmap or secret key 'values'. The values are merged in the order given, with the later values overwriting earlier, and then inline values overwriting those. Resources must be in the same namespace as the cluster.|**Type:** `[array]`<br/>|
193+
| `global.apps.karpenter.extraConfigs[*]` | **Config map or secret**|**Type:** `[object]`<br/>|
194+
| `global.apps.karpenter.extraConfigs[*].kind` | **Kind** - Specifies whether the resource is a config map or a secret.|**Type:** `[string]`<br/>|
195+
| `global.apps.karpenter.extraConfigs[*].name` | **Name** - Name of the config map or secret. The object must exist in the same namespace as the cluster App.|**Type:** `[string]`<br/>|
196+
| `global.apps.karpenter.extraConfigs[*].priority` | **Priority**|**Type:** `[integer]`<br/>**Default:** `25`|
197+
| `global.apps.karpenter.values` | **Config map** - Helm Values to be passed to the app as user config.|**Type:** `[object]`<br/>|
191198
| `global.apps.metricsServer` | **App resource** - Configuration of a default app that is part of the cluster and is deployed as an App resource.|**Type:** `[object]`<br/>|
192199
| `global.apps.metricsServer.extraConfigs` | **Extra config maps or secrets** - Extra config maps or secrets that will be used to customize to the app. The desired values must be under configmap or secret key 'values'. The values are merged in the order given, with the later values overwriting earlier, and then inline values overwriting those. Resources must be in the same namespace as the cluster.|**Type:** `[array]`<br/>|
193200
| `global.apps.metricsServer.extraConfigs[*]` | **Config map or secret**|**Type:** `[object]`<br/>|

‎helm/cluster-aws/templates/aws-ebs-csi-driver-helmrelease.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ global:
1515
podSecurityStandards:
1616
enforced: {{ .Values.global.podSecurityStandards.enforced }}
1717
{{- end }}
18-
apiVersion: helm.toolkit.fluxcd.io/v2beta1
18+
apiVersion: helm.toolkit.fluxcd.io/v2
1919
kind: HelmRelease
2020
metadata:
2121
name: {{ include "resource.default.name" $ }}-aws-ebs-csi-driver

‎helm/cluster-aws/templates/cloud-provider-aws-helmrelease.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ global:
1313
podSecurityStandards:
1414
enforced: {{ .Values.global.podSecurityStandards.enforced }}
1515
{{- end }}
16-
apiVersion: helm.toolkit.fluxcd.io/v2beta1
16+
apiVersion: helm.toolkit.fluxcd.io/v2
1717
kind: HelmRelease
1818
metadata:
1919
name: {{ include "resource.default.name" $ }}-cloud-provider-aws

‎helm/cluster-aws/templates/karpenter-apps.yaml‎

Lines changed: 0 additions & 84 deletions
This file was deleted.

‎helm/cluster-aws/templates/karpenter-crossplane-resources-helmrelease.yaml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
{{- if include "hasKarpenterNodePool" . }}
2+
apiVersion: helm.toolkit.fluxcd.io/v2
3+
kind: HelmRelease
4+
metadata:
5+
name: {{ include "resource.default.name" $ }}-karpenter-crossplane-resources
6+
namespace: {{ $.Release.Namespace }}
7+
annotations:
8+
cluster.giantswarm.io/description: "{{ .Values.global.metadata.description }}"
9+
labels:
10+
cluster-apps-operator.giantswarm.io/watching: ""
11+
{{- include "labels.common" . | nindent 4 }}
12+
spec:
13+
suspend: false
14+
releaseName: {{ include "resource.default.name" $ }}-karpenter-crossplane-resources
15+
targetNamespace: {{ $.Release.Namespace | quote }}
16+
storageNamespace: {{ $.Release.Namespace | quote }}
17+
chart:
18+
spec:
19+
chart: karpenter-crossplane-resources
20+
{{- $_ := set $ "appName" "karpenter-crossplane-resources" }}
21+
version: {{ include "cluster.app.version" $ }}
22+
sourceRef:
23+
kind: HelmRepository
24+
name: {{ include "resource.default.name" $ }}-{{ include "cluster.app.catalog" $ }}
25+
serviceAccountName: automation
26+
interval: 5m
27+
install:
28+
remediation:
29+
retries: -1
30+
upgrade:
31+
remediation:
32+
retries: -1
33+
valuesFrom:
34+
- kind: ConfigMap
35+
name: {{ include "resource.default.name" $ }}-crossplane-config
36+
valuesKey: values
37+
{{- end }}

‎helm/cluster-aws/templates/karpenter-helmrelease.yaml‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
{{/* Default Helm values for the app */}}
2+
{{/* See schema for the appropriate app version here https://github.com/giantswarm/karpenter-app/blob/main/helm/karpenter/values.schema.json */}}
3+
{{- define "defaultkarpenterHelmValues" }}
4+
additionalLabels:
5+
application.giantswarm.io/team: {{ index .Chart.Annotations "application.giantswarm.io/team" | default "phoenix" | quote }}
6+
giantswarm.io/managed-by: {{ .Release.Name | quote }}
7+
giantswarm.io/service-type: managed
8+
settings:
9+
clusterEndpoint: {{ printf "%s.%s.%s" "http://api" (include "resource.default.name" $) (required "global.connectivity.baseDomain is required" .Values.global.connectivity.baseDomain) }}
10+
clusterName: {{ include "resource.default.name" $ }}
11+
interruptionQueue: {{ include "resource.default.name" $ }}-karpenter
12+
controller:
13+
env:
14+
- name: AWS_REGION
15+
value: {{ include "aws-region" $ }}
16+
- name: AWS_ROLE_ARN
17+
value: arn:{{ include "aws-partition" $ }}:iam::{{ include "aws-account-id" $ }}:role/{{ include "resource.default.name" $ }}-karpenter
18+
- name: AWS_WEB_IDENTITY_TOKEN_FILE
19+
value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
20+
extraVolumeMounts:
21+
- name: aws-iam-token
22+
mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount/
23+
readOnly: true
24+
image:
25+
repository: {{ include "awsContainerImageRegistry" . }}/giantswarm/karpenter-controller
26+
dnsPolicy: Default
27+
extraVolumes:
28+
- name: aws-iam-token
29+
projected:
30+
sources:
31+
- serviceAccountToken:
32+
audience: {{ include "awsApiServerApiAudiences" $ | trim }}
33+
expirationSeconds: 86400
34+
path: token
35+
nodeSelector:
36+
node-role.kubernetes.io/control-plane: ""
37+
serviceAccount:
38+
annotations:
39+
eks.amazonaws.com/role-arn: arn:{{ include "aws-partition" $ }}:iam::{{ include "aws-account-id" $ }}:role/{{ include "resource.default.name" $ }}-karpenter
40+
tolerations:
41+
- key: "node-role.kubernetes.io/control-plane"
42+
operator: "Exists"
43+
effect: "NoSchedule"
44+
{{- end }}
45+
{{- if include "hasKarpenterNodePool" . }}
46+
apiVersion: helm.toolkit.fluxcd.io/v2
47+
kind: HelmRelease
48+
metadata:
49+
name: {{ include "resource.default.name" $ }}-karpenter
50+
namespace: {{ $.Release.Namespace }}
51+
annotations:
52+
cluster.giantswarm.io/description: "{{ .Values.global.metadata.description }}"
53+
labels:
54+
cluster-apps-operator.giantswarm.io/watching: ""
55+
{{- include "labels.common" . | nindent 4 }}
56+
spec:
57+
suspend: false
58+
releaseName: karpenter
59+
targetNamespace: karpenter
60+
storageNamespace: karpenter
61+
chart:
62+
spec:
63+
chart: karpenter
64+
{{- $_ := set $ "appName" "karpenter" }}
65+
version: {{ include "cluster.app.version" $ }}
66+
sourceRef:
67+
kind: HelmRepository
68+
name: {{ include "resource.default.name" $ }}-{{ include "cluster.app.catalog" $ }}
69+
kubeConfig:
70+
secretRef:
71+
name: {{ include "resource.default.name" $ }}-kubeconfig
72+
interval: 5m
73+
timeout: 15m # We need a bigger timeout because it could take a while for IRSA (via CloudFront) to become available
74+
install:
75+
createNamespace: true
76+
remediation:
77+
retries: -1
78+
upgrade:
79+
remediation:
80+
retries: -1
81+
{{- $karpenterHelmValues := (include "defaultkarpenterHelmValues" .) | fromYaml -}}
82+
{{- $customkarpenterHelmValues := $.Values.global.apps.karpenter.values -}}
83+
{{- if $customkarpenterHelmValues }}
84+
{{- $karpenterHelmValues = merge (deepCopy $customkarpenterHelmValues) $karpenterHelmValues -}}
85+
{{- end }}
86+
{{- if $karpenterHelmValues }}
87+
values: {{- $karpenterHelmValues | toYaml | nindent 4 }}
88+
{{- end }}
89+
{{- if $.Values.global.apps.karpenter.extraConfigs }}
90+
{{- range $config := $.Values.global.apps.karpenter.extraConfigs }}
91+
- kind: {{ $config.kind }}
92+
name: {{ $config.name }}
93+
valuesKey: values
94+
optional: {{ $config.optional | default false }}
95+
{{- end }}
96+
{{- end }}
97+
{{- end }}

‎helm/cluster-aws/values.schema.json‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1299,6 +1299,12 @@
12991299
"title": "k8s-dns-node-cache",
13001300
"description": "Configuration of k8s-dns-node-cache. For all available values see https://github.com/giantswarm/k8s-dns-node-cache-app."
13011301
},
1302+
"karpenter": {
1303+
"$ref": "#/$defs/app",
1304+
"type": "object",
1305+
"title": "karpenter",
1306+
"description": "Configuration of karpenter. For all available values see https://github.com/giantswarm/karpenter-app."
1307+
},
13021308
"metricsServer": {
13031309
"$ref": "#/$defs/app",
13041310
"type": "object",

‎helm/cluster-aws/values.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -291,6 +291,7 @@ global:
291291
irsaServiceMonitors: {}
292292
k8sAuditMetrics: {}
293293
k8sDnsNodeCache: {}
294+
karpenter: {}
294295
metricsServer: {}
295296
netExporter: {}
296297
networkPolicies: {}

0 commit comments

Comments
 (0)