fix: use Recreate strategy for snyk-monitor when using PVC as storage

When using a PersistentVolumeClaim, it has a Volume attached to it that can be shared by Pods only on the same worker node.

On upgrades to the Deployment, Kubernetes assigns the new Pods to random Kubernetes worker nodes. If the new Pod is assigned to a different worker node, the PVC cannot be detached from the old Pod and assigned to the new Pod.

This is why when we use a PVC we now change the snyk-monitor Deployment strategy to Recreate. Kubernetes will first delete the old Pod, freeing up the PVC, which can then be used in the new Pod.

Author: ivanstanev

.circleci/config.yml

@@ -535,12 +535,24 @@ jobs:
                       sleep $SLEEP_SECONDS_BETWEEN_ATTEMPTS
                     done
 
+                    SNYK_MONITOR_POD=$(kubectl get pods -n snyk-monitor --no-headers | \
+                      grep "snyk-monitor" | \
+                      awk 'END { if (NR==0) exit 1; else print $1 }')
+
                     # If we polled for 5 minutes and the snyk-monitor still hasn't upgraded, fail the current job.
                     if [[ "${VERSION}" != "${LATEST_TAG}" ]]; then
                       &>2 echo "versions (${VERSION}) does not match expected (${LATEST_TAG})!"
+
+                      kubectl describe pod ${SNYK_MONITOR_POD} -n snyk-monitor
+                      kubectl describe catalogsource snyk-operator -n openshift-marketplace
+                      kubectl get snykmonitors.charts.helm.k8s.io -n snyk-monitor -o yaml
+
                       exit 1
                     fi
 
+                    # We need to wait for the Pod to become Ready
+                    kubectl wait pod/${SNYK_MONITOR_POD} -n snyk-monitor --timeout 120s --for condition=Ready
+
                     echo "Update complete!"
                 name: Upgrade Operator and check that snyk-monitor also upgraded
             - run:

.circleci/config/jobs/operator_upgrade_tests.yml

@@ -190,12 +190,24 @@ steps:
           sleep $SLEEP_SECONDS_BETWEEN_ATTEMPTS
         done
 
+        SNYK_MONITOR_POD=$(kubectl get pods -n snyk-monitor --no-headers | \
+          grep "snyk-monitor" | \
+          awk 'END { if (NR==0) exit 1; else print $1 }')
+
         # If we polled for 5 minutes and the snyk-monitor still hasn't upgraded, fail the current job.
         if [[ "${VERSION}" != "${LATEST_TAG}" ]]; then
           &>2 echo "versions (${VERSION}) does not match expected (${LATEST_TAG})!"
+
+          kubectl describe pod ${SNYK_MONITOR_POD} -n snyk-monitor
+          kubectl describe catalogsource snyk-operator -n openshift-marketplace
+          kubectl get snykmonitors.charts.helm.k8s.io -n snyk-monitor -o yaml
+
           exit 1
         fi
 
+        # We need to wait for the Pod to become Ready
+        kubectl wait pod/${SNYK_MONITOR_POD} -n snyk-monitor --timeout 120s --for condition=Ready
+
         echo "Update complete!"
 
   - run:

snyk-monitor/templates/deployment.yaml

@@ -8,6 +8,10 @@ metadata:
     app.kubernetes.io/instance: {{ .Release.Name }}
     app.kubernetes.io/managed-by: {{ .Release.Service }}
 spec:
+  {{- if .Values.pvc.enabled }}
+  strategy:
+    type: Recreate
+  {{- end }}
   selector:
     matchLabels:
       app.kubernetes.io/name: {{ include "snyk-monitor.name" . }}