Skip to content

Commit ed24971

Browse files
VietND96VietND96
committed
Improve keda objects cleanup when helm delete
Signed-off-by: Viet Nguyen Duc <[email protected]>
1 parent 52cd00b commit ed24971

File tree

8 files changed

+121
-24
lines changed

8 files changed

+121
-24
lines changed

‎charts/selenium-grid/CONFIGURATION.md‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -388,15 +388,16 @@ A Helm chart for creating a Selenium Grid Server in Kubernetes
388388
| autoscaling.enabled | bool | `false` | Enable autoscaling. Implies installing KEDA |
389389
| autoscaling.enableWithExistingKEDA | bool | `false` | Enable autoscaling without automatically installing KEDA |
390390
| autoscaling.scalingType | string | `"job"` | Which type of KEDA scaling to use: job or deployment |
391-
| autoscaling.authenticationRef | object | `{"annotations":{"helm.sh/hook":"post-install,post-upgrade,post-rollback","helm.sh/hook-weight":"0"},"name":""}` | Specify an external KEDA TriggerAuthentication resource is used for scaler triggers config. Apply for all browser nodes |
391+
| autoscaling.authenticationRef | object | `{"annotations":{"helm.sh/hook":"post-upgrade","helm.sh/hook-delete-policy":"before-hook-creation","helm.sh/hook-weight":"0"},"name":""}` | Specify an external KEDA TriggerAuthentication resource is used for scaler triggers config. Apply for all browser nodes |
392392
| autoscaling.useCachedMetrics | bool | `false` | Enables caching of metric values during polling interval (as specified in .spec.pollingInterval, the default: false in KEDA). |
393393
| autoscaling.metricType | string | `""` | The type of metric that should be used (The default: AverageValue in KEDA) |
394-
| autoscaling.annotations | object | `{"helm.sh/hook":"post-install,post-upgrade,post-rollback","helm.sh/hook-weight":"1"}` | Annotations for KEDA resources: ScaledObject and ScaledJob |
394+
| autoscaling.annotations | object | `{"helm.sh/hook":"post-install,post-upgrade","helm.sh/hook-delete-policy":"before-hook-creation","helm.sh/hook-weight":"1"}` | Annotations for KEDA resources: ScaledObject and ScaledJob |
395395
| autoscaling.patchObjectFinalizers.nameOverride | string | `nil` | Override the name of the patch job |
396396
| autoscaling.patchObjectFinalizers.enabled | bool | `true` | Enable patching finalizers for KEDA scaled resources. Workaround for Hook post-upgrade selenium-grid/templates/x-node-hpa.yaml failed: object is being deleted: scaledobjects.keda.sh "x" already exists |
397-
| autoscaling.patchObjectFinalizers.activeDeadlineSeconds | int | `120` | Deadline (in seconds) for patch job to complete |
398-
| autoscaling.patchObjectFinalizers.annotations | object | `{"helm.sh/hook":"post-install,post-upgrade,post-rollback,pre-delete","helm.sh/hook-delete-policy":"hook-succeeded,before-hook-creation","helm.sh/hook-weight":"-1"}` | Annotations for patch job |
399-
| autoscaling.patchObjectFinalizers.cleanUpScript | string | `""` | Define your custom script to replace the default script |
397+
| autoscaling.patchObjectFinalizers.activeDeadlineSeconds | int | `300` | Deadline (in seconds) for patch job to complete |
398+
| autoscaling.patchObjectFinalizers.annotations | object | `{"helm.sh/hook":"pre-delete","helm.sh/hook-delete-policy":"hook-succeeded,before-hook-creation"}` | Annotations for patch job |
399+
| autoscaling.patchObjectFinalizers.deleteObjectsScript | string | `""` | Define your custom script to replace the default script |
400+
| autoscaling.patchObjectFinalizers.patchFinalizersScript | string | `""` | Define your custom script to replace the default script |
400401
| autoscaling.patchObjectFinalizers.defaultMode | int | `493` | Default mode for ConfigMap is mounted as file |
401402
| autoscaling.patchObjectFinalizers.serviceAccount | string | `""` | Define an external service account name contains permissions to patch KEDA scaled resources |
402403
| autoscaling.patchObjectFinalizers.imagePullSecret | string | `""` | Custom pull secret for container in patch job |

‎charts/selenium-grid/templates/_nameHelpers.tpl‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -260,11 +260,18 @@ Server ConfigMap fullname
260260
{{- tpl (default (include "seleniumGrid.component.name" (list "selenium-server-config" $)) .Values.serverConfigMap.nameOverride) $ | trunc 63 | trimSuffix "-" -}}
261261
{{- end -}}
262262

263+
{{/*
264+
Delete scaledObjects leafover job fullname
265+
*/}}
266+
{{- define "seleniumGrid.keda.deleteObjectsJob.fullname" -}}
267+
{{- printf "%s-scaledobjects-deletion" (tpl ( default (include "seleniumGrid.component.name" (list "selenium-patch" $)) .Values.autoscaling.patchObjectFinalizers.nameOverride) $) | trunc 63 | trimSuffix "-" -}}
268+
{{- end -}}
269+
263270
{{/*
264271
Patch scaledObjects finalizers job fullname
265272
*/}}
266273
{{- define "seleniumGrid.keda.patchObjectsJob.fullname" -}}
267-
{{- tpl ( default (include "seleniumGrid.component.name" (list "selenium-patch-scaledobjects-finalizers" $)) .Values.autoscaling.patchObjectFinalizers.nameOverride) $ | trunc 63 | trimSuffix "-" -}}
274+
{{- printf "%s-scaledobjects-finalizers" (tpl ( default (include "seleniumGrid.component.name" (list "selenium-patch" $)) .Values.autoscaling.patchObjectFinalizers.nameOverride) $) | trunc 63 | trimSuffix "-" -}}
268275
{{- end -}}
269276

270277
{{/*

‎charts/selenium-grid/templates/patch-keda/delete-keda-objects-job.yaml‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
{{- if and (eq (include "seleniumGrid.useKEDA" $) "true") $.Values.autoscaling.patchObjectFinalizers.enabled }}
2+
apiVersion: batch/v1
3+
kind: Job
4+
metadata:
5+
name: {{ template "seleniumGrid.keda.deleteObjectsJob.fullname" $ }}
6+
namespace: {{ .Release.Namespace }}
7+
annotations: &patch_objects_job_annotations
8+
"helm.sh/hook-weight": "-10"
9+
{{- with $.Values.autoscaling.patchObjectFinalizers.annotations }}
10+
{{- toYaml . | nindent 4 }}
11+
{{- end }}
12+
labels: &patch_objects_job_labels
13+
app: {{ template "seleniumGrid.keda.deleteObjectsJob.fullname" $ }}
14+
app.kubernetes.io/name: {{ template "seleniumGrid.keda.deleteObjectsJob.fullname" $ }}
15+
{{- include "seleniumGrid.commonLabels" $ | nindent 4 }}
16+
{{- with $.Values.customLabels }}
17+
{{- toYaml . | nindent 4 }}
18+
{{- end }}
19+
spec:
20+
template:
21+
metadata:
22+
{{- with $.Values.autoscaling.patchObjectFinalizers.annotations }}
23+
annotations: *patch_objects_job_annotations
24+
{{- end }}
25+
labels: *patch_objects_job_labels
26+
name: {{ template "seleniumGrid.keda.deleteObjectsJob.fullname" $ }}
27+
spec:
28+
activeDeadlineSeconds: {{ $.Values.autoscaling.patchObjectFinalizers.activeDeadlineSeconds }}
29+
serviceAccountName: {{ default (include "seleniumGrid.serviceAccount.fullname" $) $.Values.autoscaling.patchObjectFinalizers.serviceAccount }}
30+
serviceAccount: {{ default (include "seleniumGrid.serviceAccount.fullname" $) $.Values.autoscaling.patchObjectFinalizers.serviceAccount }}
31+
containers:
32+
- name: kubectl
33+
image: {{ $.Values.global.seleniumGrid.kubectlImage }}
34+
command: ["/bin/bash", "-c", "/deleteObjectsScript.sh"]
35+
volumeMounts:
36+
- name: cleanup-script
37+
mountPath: /deleteObjectsScript.sh
38+
subPath: deleteObjectsScript.sh
39+
{{- with $.Values.autoscaling.patchObjectFinalizers.resources }}
40+
resources: {{ toYaml . | nindent 12 }}
41+
{{- end }}
42+
volumes:
43+
- name: cleanup-script
44+
configMap:
45+
name: {{ template "seleniumGrid.keda.patchObjectsJob.fullname" $ }}
46+
defaultMode: {{ $.Values.autoscaling.patchObjectFinalizers.defaultMode }}
47+
{{- if or $.Values.global.seleniumGrid.imagePullSecret $.Values.autoscaling.patchObjectFinalizers.imagePullSecret }}
48+
imagePullSecrets:
49+
- name: {{ default $.Values.global.seleniumGrid.imagePullSecret $.Values.autoscaling.patchObjectFinalizers.imagePullSecret }}
50+
{{- end }}
51+
restartPolicy: Never
52+
{{- with .Values.autoscaling.patchObjectFinalizers.nodeSelector }}
53+
nodeSelector: {{- toYaml . | nindent 8 }}
54+
{{- end }}
55+
{{- end }}

‎charts/selenium-grid/templates/patch-keda/patch-keda-objects-cm.yaml‎

Lines changed: 36 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,9 @@ kind: ConfigMap
44
metadata:
55
name: {{ template "seleniumGrid.keda.patchObjectsJob.fullname" $ }}
66
namespace: {{ .Release.Namespace }}
7-
{{- with $.Values.autoscaling.patchObjectFinalizers.annotations }}
87
annotations:
8+
"helm.sh/hook-weight": "-20"
9+
{{- with $.Values.autoscaling.patchObjectFinalizers.annotations }}
910
{{- toYaml . | nindent 4 }}
1011
{{- end }}
1112
labels:
@@ -16,14 +17,44 @@ metadata:
1617
{{- toYaml . | nindent 4 }}
1718
{{- end }}
1819
data:
19-
{{- if $.Values.autoscaling.patchObjectFinalizers.cleanUpScript }}
20-
cleanUpScript.sh: {{- toYaml $.Values.autoscaling.patchObjectFinalizers.cleanUpScript | nindent 4 }}
20+
{{- if $.Values.autoscaling.patchObjectFinalizers.deleteObjectsScript }}
21+
deleteObjectsScript.sh: {{- toYaml $.Values.autoscaling.patchObjectFinalizers.deleteObjectsScript | nindent 4 }}
22+
{{- else }}
23+
deleteObjectsScript.sh: |
24+
#!/bin/bash
25+
set -e
26+
set -x
27+
echo "Delete ScaledObjects, ScaledJobs and HPAs for {{ .Release.Name }} when upgrading or disabling autoscaling."
28+
for i in $(kubectl get ScaledObjects -n {{ .Release.Namespace }} -l app.kubernetes.io/instance={{ .Release.Name }} \
29+
-o jsonpath='{.items[*].metadata.name}{"\n"}');
30+
do
31+
if [ -n "$i" ]; then
32+
kubectl delete ScaledObjects $i -n {{ .Release.Namespace }}
33+
fi
34+
done
35+
for i in $(kubectl get ScaledJobs -n {{ .Release.Namespace }} -l app.kubernetes.io/instance={{ .Release.Name }} \
36+
-o jsonpath='{.items[*].metadata.name}{"\n"}');
37+
do
38+
if [ -n "$i" ]; then
39+
kubectl delete ScaledJobs $i -n {{ .Release.Namespace }}
40+
fi
41+
done
42+
for i in $(kubectl get TriggerAuthentication -n {{ .Release.Namespace }} -l app.kubernetes.io/instance={{ .Release.Name }} \
43+
-o jsonpath='{.items[*].metadata.name}{"\n"}');
44+
do
45+
if [ -n "$i" ]; then
46+
kubectl delete TriggerAuthentication $i -n {{ .Release.Namespace }}
47+
fi
48+
done
49+
{{- end }}
50+
{{- if $.Values.autoscaling.patchObjectFinalizers.patchFinalizersScript }}
51+
patchFinalizersScript.sh: {{- toYaml $.Values.autoscaling.patchObjectFinalizers.patchFinalizersScript | nindent 4 }}
2152
{{- else }}
22-
cleanUpScript.sh: |
53+
patchFinalizersScript.sh: |
2354
#!/bin/bash
2455
set -e
2556
set -x
26-
echo "Cleaning up ScaledObjects, ScaledJobs and HPAs for {{ .Release.Name }} when upgrading or disabling autoscaling."
57+
echo "Patch finalizers of ScaledObjects, ScaledJobs and HPAs for {{ .Release.Name }} when upgrading or disabling autoscaling."
2758
for i in $(kubectl get ScaledObjects -n {{ .Release.Namespace }} -l app.kubernetes.io/instance={{ .Release.Name }} \
2859
-o jsonpath='{.items[*].metadata.name}{"\n"}');
2960
do

‎charts/selenium-grid/templates/patch-keda/patch-keda-objects-job.yaml‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,9 @@ kind: Job
44
metadata:
55
name: {{ template "seleniumGrid.keda.patchObjectsJob.fullname" $ }}
66
namespace: {{ .Release.Namespace }}
7-
{{- with $.Values.autoscaling.patchObjectFinalizers.annotations }}
87
annotations: &patch_objects_job_annotations
8+
"helm.sh/hook-weight": "-1"
9+
{{- with $.Values.autoscaling.patchObjectFinalizers.annotations }}
910
{{- toYaml . | nindent 4 }}
1011
{{- end }}
1112
labels: &patch_objects_job_labels
@@ -30,11 +31,11 @@ spec:
3031
containers:
3132
- name: kubectl
3233
image: {{ $.Values.global.seleniumGrid.kubectlImage }}
33-
command: ["/bin/bash", "-c", "/cleanUpScript.sh"]
34+
command: ["/bin/bash", "-c", "/patchFinalizersScript.sh"]
3435
volumeMounts:
3536
- name: cleanup-script
36-
mountPath: /cleanUpScript.sh
37-
subPath: cleanUpScript.sh
37+
mountPath: /patchFinalizersScript.sh
38+
subPath: patchFinalizersScript.sh
3839
{{- with $.Values.autoscaling.patchObjectFinalizers.resources }}
3940
resources: {{ toYaml . | nindent 12 }}
4041
{{- end }}

‎charts/selenium-grid/values.yaml‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1013,7 +1013,8 @@ autoscaling:
10131013
authenticationRef:
10141014
name: ""
10151015
annotations:
1016-
"helm.sh/hook": post-install,post-upgrade,post-rollback
1016+
"helm.sh/hook": post-upgrade
1017+
"helm.sh/hook-delete-policy": before-hook-creation
10171018
# TriggerAuthentication is used by ScaledObject/ScaledJob, hence weight should be less than those hooks
10181019
"helm.sh/hook-weight": "0"
10191020
# Configuration for ScaledObject triggers https://keda.sh/docs/latest/reference/scaledobject-spec/#triggers
@@ -1023,7 +1024,8 @@ autoscaling:
10231024
metricType: ""
10241025
# -- Annotations for KEDA resources: ScaledObject and ScaledJob
10251026
annotations:
1026-
"helm.sh/hook": post-install,post-upgrade,post-rollback
1027+
"helm.sh/hook": post-install,post-upgrade
1028+
"helm.sh/hook-delete-policy": before-hook-creation
10271029
# Ensure the weight should be higher than TriggerAuthentication hook
10281030
"helm.sh/hook-weight": "1"
10291031
patchObjectFinalizers:
@@ -1032,15 +1034,15 @@ autoscaling:
10321034
# -- Enable patching finalizers for KEDA scaled resources. Workaround for Hook post-upgrade selenium-grid/templates/x-node-hpa.yaml failed: object is being deleted: scaledobjects.keda.sh "x" already exists
10331035
enabled: true
10341036
# -- Deadline (in seconds) for patch job to complete
1035-
activeDeadlineSeconds: 120
1037+
activeDeadlineSeconds: 300
10361038
# -- Annotations for patch job
10371039
annotations:
1038-
"helm.sh/hook": post-install,post-upgrade,post-rollback,pre-delete
1040+
"helm.sh/hook": pre-delete
10391041
"helm.sh/hook-delete-policy": hook-succeeded,before-hook-creation
1040-
# This should be run before all other hooks (since delete action is called), so use a negative weight
1041-
"helm.sh/hook-weight": "-1"
10421042
# -- Define your custom script to replace the default script
1043-
cleanUpScript: ""
1043+
deleteObjectsScript: ""
1044+
# -- Define your custom script to replace the default script
1045+
patchFinalizersScript: ""
10441046
# -- Default mode for ConfigMap is mounted as file
10451047
defaultMode: 0755
10461048
# -- Define an external service account name contains permissions to patch KEDA scaled resources

‎tests/charts/ci/nameOverride-values.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,4 +28,4 @@ secrets:
2828
nameOverride: "selenium-grid-common-secrets"
2929
autoscaling:
3030
patchObjectFinalizers:
31-
nameOverride: "selenium-grid-job-patch-objects"
31+
nameOverride: "selenium-grid-patch"

‎tests/charts/make/chart_test.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -94,7 +94,7 @@ cleanup() {
9494
if [ "${SKIP_CLEANUP}" = "false" ] || [ "${CI:-false}" != "false" ]; then
9595
helm ls -A
9696
echo "Clean up chart release and namespace"
97-
helm delete ${RELEASE_NAME} --namespace ${SELENIUM_NAMESPACE} --no-hooks || true
97+
helm delete ${RELEASE_NAME} --namespace ${SELENIUM_NAMESPACE} || true
9898
wait_for_terminated
9999
sudo chmod -R 777 ${HOST_PATH}/logs
100100
fi

0 commit comments

Comments
 (0)