Merge pull request #265 from cognifloyd/jobs_extra_hooks

cognifloyd · web-flow · commit 5060b0f7e6e3 · 2022-03-23T22:44:11.000-05:00
Enable extra post-install and post-upgrade helm-hook jobs for environment-specific config
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,6 +7,7 @@
 * Initialize basic unittest infrastructure using `helm-unittest`. Added tests for labels, custom annotations, SecurityContext, pullSecrets, pullPolicy, Resources, nodeSelector, tolerations, affinity, dnsPolicy, dnsConfig, ServiceAccount attach, postStartScript, both sensor-modes, env, envFrom, st2.packs.images, and st2.packs.volumes. (#284, #288, #292)
 * Allow partitioning sensors using the hash_range strategy instead of one sensor per pod. (#218) (by @cognifloyd)
 * New feature to include possibility for external services in st2api, st2stream and st2auth, setting default value for this services as `ClusterIP` and `hostname: ""`. Also, added new entry for custom_annotations_test.yaml and created new unit test services_test.yaml. (by @sandesvitor)
+* Advanced Feature: Add extra Helm hook Jobs. This minimizes the boilerplate required to run stackstorm workflows at various helm hook stages: post-install, pre-upgrade, post-upgrade. (#265) (by @cognifloyd)
 
 ## v0.80.0
 * Switch st2 to `v3.6` as a new default stable version (#274)
diff --git a/README.md b/README.md
@@ -325,6 +325,36 @@ Grab all logs only for stackstorm backend services, excluding st2web and DB/MQ/r
 kubectl logs -l release=<release-name>,tier=backend
 ```
 
+## Running jobs before/after install, upgrade, or rollback
+WARNING: The feature described in this section is an Advanced feature that new users should not need.
+
+It may be convenient to run one or more `Job`(s) in your `stackstorm-ha` cluster to manage your release's life cycle.
+As the [Helm docs]() explain:
+
+> Helm provides a _hook_ mechanism to allow chart developers to intervene at certain points in a release's life cycle.
+
+The `jobs.extra_hooks` feature in this chart simplifies creating `Jobs` that Helm will run in its hooks.
+These jobs will use the same settings as any other job defined by this chart (eg image, annotations, pod placement).
+The `st2.conf` files and packs volumes will be mounted in the Job and the `st2` cli will be configured.
+This feature is primarily useful when you need to run a StackStorm workflow (with `st2 run ...`) after install,
+before/after upgrades, or before/after rollbacks.
+
+NOTE: The `jobs.extra_hooks` feature is very opinionated. If you need to to apply helm hooks to anything other than
+`Jobs`, or if these jobs do not meet your needs, then you will need to do so from a parent chart. For example, parent charts
+are much better suited to jobs that don't need access to the packs, configs, configmaps, and secrets that this chart provides.
+See "Extending this chart" below.
+
+These extra hooks jobs can be used for st2 installation-specific jobs like:
+
+- running a pre-upgrade st2 workflow that notifies on various channels that the upgrade is happening,
+- running post-upgrade smoke tests to ensure st2 can connect to vital services (vault, kubernetes, aws, etc),
+- running a pre-upgrade st2 workflow that pauses long-running workflows,
+- running a post-upgrade st2 workflow that resumes long-running workflows,
+- running one-time post-install configuration (such as generating dynamic secrets in the st2kv datastore),
+
+To configure the `jobs.extra_hooks`, set `jobs.extra_hooks` in your values file.
+Please refer to stackstorm-ha's default values.yaml file for examples.
+
 ## Extending this chart
 If you have any suggestions or ideas about how to extend this chart functionality,
 we welcome you to collaborate in [Issues](https://github.com/stackstorm/stackstorm-ha/issues)
diff --git a/templates/jobs.yaml b/templates/jobs.yaml
@@ -504,3 +504,139 @@ spec:
     {{- end }}
 
 {{- end }}
+{{- range .Values.jobs.extra_hooks -}}
+  {{- $name := print "extra-helm-hook" (include "stackstorm-ha.hyphenPrefix" (required "You must name each entry in jobs.extra_hooks." .name)) }}
+  {{- if not ($.Values.jobs.skip | has $name) }}
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: {{ $.Release.Name }}-job-{{ $name }}
+  labels:
+    app: {{ $name }}
+    tier: backend
+    vendor: stackstorm
+    chart: {{ $.Chart.Name }}-{{ $.Chart.Version }}
+    release: {{ $.Release.Name }}
+    heritage: {{ $.Release.Service }}
+  annotations:
+    helm.sh/hook: {{ required "Each entry in jobs.extra_hooks must include 'hook' (the helm.sh/hook value)" .hook }}
+    helm.sh/hook-delete-policy: before-hook-creation
+    helm.sh/hook-weight: {{ .hook_weight | default 10 | toString | quote }}
+  {{- if $.Values.jobs.annotations }}
+    {{- toYaml $.Values.jobs.annotations | nindent 4 }}
+  {{- end }}
+spec:
+  template:
+    metadata:
+      name: job-{{ $name }}
+      labels:
+        app: {{ $name }}
+        tier: backend
+        vendor: stackstorm
+        chart: {{ $.Chart.Name }}-{{ $.Chart.Version }}
+        release: {{ $.Release.Name }}
+        heritage: {{ $.Release.Service }}
+      annotations:
+        checksum/config: {{ include (print $.Template.BasePath "/configmaps_st2-conf.yaml") $ | sha256sum }}
+        checksum/packs: {{ include (print $.Template.BasePath "/configmaps_packs.yaml") $ | sha256sum }}
+        {{- if $.Values.jobs.annotations }}
+          {{- toYaml $.Values.jobs.annotations | nindent 8 }}
+        {{- end }}
+    spec:
+      imagePullSecrets:
+      {{- if $.Values.image.pullSecret }}
+      - name: {{ $.Values.image.pullSecret }}
+      {{- end }}
+      {{- if $.Values.st2.packs.images -}}
+        {{- include "stackstorm-ha.packs-pullSecrets" $ | nindent 6 }}
+      {{- end }}
+      initContainers:
+      {{- include "stackstorm-ha.init-containers-wait-for-db" $ | nindent 6 }}
+      {{- include "stackstorm-ha.packs-initContainers" $ | nindent 6 }}
+      - name: generate-st2client-config
+        image: '{{ template "stackstorm-ha.imageRepository" $ }}/st2actionrunner:{{ tpl ($.Values.jobs.image.tag | default ($.Values.st2actionrunner.image.tag | default $.Values.image.tag)) $ }}'
+        imagePullPolicy: {{ $.Values.image.pullPolicy }}
+        {{- with $.Values.securityContext }}
+        securityContext: {{- toYaml . | nindent 10 }}
+        {{- end }}
+        envFrom:
+        - configMapRef:
+            name: {{ $.Release.Name }}-st2-urls
+        - secretRef:
+            name: {{ $.Release.Name }}-st2-auth
+        {{- range $.Values.jobs.envFromSecrets }}
+        - secretRef:
+            name: {{ . }}
+        {{- end }}
+        volumeMounts:
+        - name: st2client-config-vol
+          mountPath: /root/.st2/
+        # `st2 login` doesn't exit on failure correctly, use old methods instead. See bug: https://github.com/StackStorm/st2/issues/4338
+        command:
+          - 'sh'
+          - '-ec'
+          - |
+            cat <<EOT > /root/.st2/config
+            [credentials]
+            {{- tpl $.Values.jobs.st2clientConfig $ | nindent 12 }}
+            EOT
+      containers:
+      - name: {{ $name }}
+        image: '{{ template "stackstorm-ha.imageRepository" $ }}/st2actionrunner:{{ tpl ($.Values.jobs.image.tag | default ($.Values.st2actionrunner.image.tag | default $.Values.image.tag)) $ }}'
+        imagePullPolicy: {{ $.Values.image.pullPolicy }}
+        {{- with $.Values.securityContext }}
+        securityContext: {{- toYaml . | nindent 10 }}
+        {{- end }}
+        {{- if $.Values.jobs.env }}
+        env: {{- include "stackstorm-ha.customEnv" $.Values.jobs | nindent 8 }}
+        {{- end }}
+        envFrom:
+        - configMapRef:
+            name: {{ $.Release.Name }}-st2-urls
+        {{- range $.Values.jobs.envFromSecrets }}
+        - secretRef:
+            name: {{ . }}
+        {{- end }}
+        command:
+        {{- range (required "Each entry in jobs.extra_hooks must include the 'command' to run." .command) }}
+          {{- tpl . $ | list | toYaml | nindent 10 }}
+        {{- end }}
+        volumeMounts:
+        - name: st2client-config-vol
+          mountPath: /root/.st2/
+        {{- include "stackstorm-ha.st2-config-volume-mounts" $ | nindent 8 }}
+        {{- include "stackstorm-ha.packs-volume-mounts-for-register-job" $ | nindent 8 }}
+        {{- include "stackstorm-ha.pack-configs-volume-mount" $ | nindent 8 }}
+        {{- if .resources }}
+        resources: {{- toYaml .resources | nindent 10 }}
+        {{- end }}
+      volumes:
+        - name: st2client-config-vol
+          emptyDir:
+            medium: Memory
+        {{- include "stackstorm-ha.st2-config-volume" $ | nindent 8 }}
+        {{- include "stackstorm-ha.packs-volumes" $ | nindent 8 }}
+        {{- include "stackstorm-ha.pack-configs-volume" $ | nindent 8 }}
+      restartPolicy: OnFailure
+    {{- if $.Values.dnsPolicy }}
+      dnsPolicy: {{ $.Values.dnsPolicy }}
+    {{- end }}
+    {{- with $.Values.dnsConfig }}
+      dnsConfig: {{- toYaml . | nindent 8 }}
+    {{- end }}
+    {{- with $.Values.podSecurityContext }}
+      securityContext: {{- toYaml . | nindent 8 }}
+    {{- end }}
+    {{- with $.Values.jobs.nodeSelector }}
+      nodeSelector: {{- toYaml . | nindent 8 }}
+    {{- end }}
+    {{- with $.Values.jobs.affinity }}
+      affinity: {{- toYaml . | nindent 8 }}
+    {{- end }}
+    {{- with $.Values.jobs.tolerations }}
+      tolerations: {{- toYaml . | nindent 8 }}
+    {{- end }}
+
+  {{- end }}
+{{- end }}
diff --git a/tests/unit/custom_annotations_test.yaml b/tests/unit/custom_annotations_test.yaml
@@ -153,13 +153,19 @@ tests:
         annotations:
           foo: bar
           answer: "42"
+        extra_hooks:
+          - name: upgrade-warning
+            hook: pre-upgrade, pre-rollback
+            hook_weight: -5
+            command: ["st2", "run", "--tail", "custom_pack.warn_about_upgrade"]
     asserts:
       - hasDocuments:
-          count: 4
+          count: 5
           # job-st2-apply-rbac-defintions
           # job-st2-apikey-load
           # job-st2-key-load
           # job-st2-register-content
+          # extra_hooks job
 
       # job annotations
       - isNotNull:
diff --git a/tests/unit/dns_test.yaml b/tests/unit/dns_test.yaml
@@ -26,6 +26,12 @@ tests:
       st2: 
         packs: { sensors: [] } # ensure only 1 sensor
         rbac: { enabled: true } # enable rbac job
+      jobs:
+        extra_hooks: &jobs_extra_hooks
+          - name: upgrade-warning
+            hook: pre-upgrade, pre-rollback
+            hook_weight: -5
+            command: ["st2", "run", "--tail", "custom_pack.warn_about_upgrade"]
     asserts:
       - isNull:
           path: spec.template.spec.dnsPolicy
@@ -51,6 +57,8 @@ tests:
       st2: 
         packs: { sensors: [] } # ensure only 1 sensor
         rbac: { enabled: true } # enable rbac job
+      jobs:
+        extra_hooks: *jobs_extra_hooks
     asserts:
       - equal:
           path: spec.template.spec.dnsPolicy
diff --git a/tests/unit/env_test.yaml b/tests/unit/env_test.yaml
@@ -66,9 +66,15 @@ tests:
     set:
       st2:
         rbac: { enabled: true } # enable rbac job
+      jobs:
+        extra_hooks: &extra_hooks_jobs
+          - name: upgrade-warning
+            hook: pre-upgrade, pre-rollback
+            hook_weight: -5
+            command: ["st2", "run", "--tail", "custom_pack.warn_about_upgrade"]
     asserts:
       - hasDocuments:
-          count: 4
+          count: 5
       - isNull: *is_null_env
 
   - it: Deployments accept custom env
@@ -121,9 +127,10 @@ tests:
         rbac: { enabled: true } # enable rbac job
       jobs:
         env: *env
+        extra_hooks: *extra_hooks_jobs
     asserts:
       - hasDocuments:
-          count: 4
+          count: 5
 
       - contains: *contains_env
 
@@ -156,11 +163,13 @@ tests:
     set:
       st2:
         rbac: { enabled: true } # enable rbac job
+      jobs:
+        extra_hooks: *extra_hooks_jobs
     release:
       name: st2ha
     asserts:
       - hasDocuments:
-          count: 4
+          count: 5
 
       - contains: *contains_st2_urls
         documentIndex: 1
@@ -215,6 +224,7 @@ tests:
         rbac: { enabled: true } # enable rbac job
       jobs:
         envFromSecrets: *envFromSecrets
+        extra_hooks: *extra_hooks_jobs
     asserts:
       - contains: *contains_external_secret1
       - contains: *contains_external_secret2
diff --git a/tests/unit/image_pull_test.yaml b/tests/unit/image_pull_test.yaml
@@ -44,6 +44,7 @@ tests:
         # job-st2-apikey-load
         # job-st2-key-load
         # job-st2-register-content
+        # extra_hooks job
     set:
       # image.pullPolicy defaults to IfNotPresent
       # image.pullSecret defaults to None
@@ -56,6 +57,12 @@ tests:
         packs: { sensors: [] } # ensure only 1 sensor
       st2chatops:
         enabled: true
+      jobs:
+        extra_hooks: &jobs_extra_hooks
+          - name: upgrade-warning
+            hook: pre-upgrade, pre-rollback
+            hook_weight: -5
+            command: ["st2", "run", "--tail", "custom_pack.warn_about_upgrade"]
     asserts:
       - isNull:
           path: spec.template.spec.imagePullSecrets
@@ -84,6 +91,8 @@ tests:
         packs: { sensors: [] } # ensure only 1 sensor
       st2chatops:
         enabled: true
+      jobs:
+        extra_hooks: *jobs_extra_hooks
     asserts:
       - equal:
           path: spec.template.spec.imagePullSecrets[0].name
diff --git a/tests/unit/labels_test.yaml b/tests/unit/labels_test.yaml
@@ -101,13 +101,20 @@ tests:
       st2:
         rbac:
           enabled: true # enable rbac job
+      jobs:
+        extra_hooks:
+          - name: upgrade-warning
+            hook: pre-upgrade, pre-rollback
+            hook_weight: -5
+            command: ["st2", "run", "--tail", "custom_pack.warn_about_upgrade"]
     asserts:
       - hasDocuments:
-          count: 4
+          count: 5
           # job-st2-apply-rbac-defintions
           # job-st2-apikey-load
           # job-st2-key-load
           # job-st2-register-content
+          # extra_hooks job
 
       # unlike deployments, jobs should not have selector.matchLabels
 
diff --git a/tests/unit/packs_volumes_test.yaml b/tests/unit/packs_volumes_test.yaml
@@ -140,11 +140,17 @@ tests:
           volumes:
             enabled: false
           configs: {} # has one core.yaml config file by default (dicts get merged)
+      jobs:
+        extra_hooks: &extra_hooks_jobs
+          - name: upgrade-warning
+            hook: pre-upgrade, pre-rollback
+            hook_weight: -5
+            command: ["st2", "run", "--tail", "custom_pack.warn_about_upgrade"]
     release:
       name: st2ha
     asserts:
       - hasDocuments:
-          count: 4
+          count: 5
 
       - notContains: *packs_vol
       - notContains: *venvs_vol
@@ -393,11 +399,13 @@ tests:
           volumes:
             enabled: false
           configs: {} # has one core.yaml config file by default (dicts get merged)
+      jobs:
+        extra_hooks: *extra_hooks_jobs
     release:
       name: st2ha
     asserts:
       - hasDocuments:
-          count: 4
+          count: 5
 
       - contains: *packs_vol
         documentIndex: 3 # register_content
@@ -687,11 +695,13 @@ tests:
             virtualenvs: *nfs_venvs_volume
             configs: *nfs_pack_configs_volume
           configs: {} # has one core.yaml config file by default (dicts get merged)
+      jobs:
+        extra_hooks: *extra_hooks_jobs
     release:
       name: st2ha
     asserts:
       - hasDocuments:
-          count: 4
+          count: 5
 
       - contains: *nfs_packs_vol
         documentIndex: 3 # register_content
@@ -960,11 +970,13 @@ tests:
             virtualenvs: *nfs_venvs_volume
             configs: *nfs_pack_configs_volume
           configs: {} # has one core.yaml config file by default (dicts get merged)
+      jobs:
+        extra_hooks: *extra_hooks_jobs
     release:
       name: st2ha
     asserts:
       - hasDocuments:
-          count: 4
+          count: 5
 
       - contains: *nfs_packs_vol
         documentIndex: 3 # register_content
diff --git a/tests/unit/placement_test.yaml b/tests/unit/placement_test.yaml
diff --git a/tests/unit/security_context_test.yaml b/tests/unit/security_context_test.yaml
diff --git a/values.yaml b/values.yaml
