[update] Improve update of new operator version

Add validation to ensure the subscription's installedCSV matches
currentCSV after the installplan is applied. This confirms the operator
upgrade has completed successfully before proceeding with the rest
of the update workflow. Uses label selector to find the subscription
rather than hardcoding the name.

Add missing wait for OpenStackControlPlane to reach ready state after
operator updates in OLM mode. Include a 60-second delay before the
wait to allow the control plane's ready condition to transition from
its initial state, preventing premature completion while the control
plane is still reconciling the operator changes.
This step is important to validate that the current deployed control
plane can successfully reach ready state with the new set of operators,
but using the old service containers. Otherwise we start alreay updating
to the new service images.

Jira: OSPRH-23688

Co-Authored-By: Claude <[email protected]>
Signed-off-by: Martin Schuppert <[email protected]>

Author: 2 people

roles/update/README.md

@@ -21,5 +21,6 @@ Role to run update
 * `cifmw_update_ansible_ssh_private_key_file`: (String) Define the path to the private key file used for the compute nodes.
 * `cifmw_update_wait_retries_reboot`: (Integer) Number of retries to wait for a compute node reboot. One retry is done every five seconds. Default to 60, so five minutes.
 * `cifmw_update_resources_monitoring_interval`: (Integer) Interval, in seconds, between two resources monitor during update. Default to 10 seconds.
+* `cifmw_update_wait_controplane_status_change_sec`: (Integer) Time, in seconds, to wait before checking openstack control plane deployment status. Used when need to wait to allow the control plane's ready condition to transition from its initial state, preventing premature completion while the control plane is still reconciling the operator changes. Defaults to `60`.
 
 ## Examples

roles/update/defaults/main.yml

@@ -67,3 +67,8 @@ cifmw_update_ctl_plane_max_tries: 84
 
 # Resource Monitoring during update
 cifmw_update_resources_monitoring_interval: 10 # in seconds.
+
+# How long should we wait (in seconds) before checking control plane status. Useful
+# when we install the new operators and have to wait until they start to reconcile
+# the current deployed control plane.
+cifmw_update_wait_controplane_status_change_sec: 60

roles/update/tasks/main.yml

@@ -50,6 +50,34 @@
     - cifmw_ci_gen_kustomize_values_installplan_approval is defined
     - cifmw_ci_gen_kustomize_values_installplan_approval | lower == 'manual'
 
+- name: Validate subscription CSV version after installplan
+  when:
+    - cifmw_ci_gen_kustomize_values_installplan_approval is defined
+    - cifmw_ci_gen_kustomize_values_installplan_approval | lower == 'manual'
+  block:
+    - name: Wait for subscription installedCSV to match currentCSV
+      kubernetes.core.k8s_info:
+        kubeconfig: "{{ cifmw_openshift_kubeconfig }}"
+        api_key: "{{ cifmw_openshift_token | default(omit) }}"
+        context: "{{ cifmw_openshift_context | default(omit) }}"
+        api_version: operators.coreos.com/v1alpha1
+        kind: Subscription
+        namespace: openstack-operators
+        label_selectors:
+          - operators.coreos.com/openstack-operator.openstack-operators
+      register: _cifmw_update_subscription_info
+      until: >
+        _cifmw_update_subscription_info.resources | length > 0
+        and
+        _cifmw_update_subscription_info.resources[0].status.installedCSV is defined
+        and
+        _cifmw_update_subscription_info.resources[0].status.currentCSV is defined
+        and
+        _cifmw_update_subscription_info.resources[0].status.installedCSV ==
+        _cifmw_update_subscription_info.resources[0].status.currentCSV
+      retries: 30
+      delay: 10
+
 - name: Handle OpenStack operator initialization
   when:
     - cifmw_ci_gen_kustomize_values_deployment_version is defined
@@ -145,6 +173,29 @@
           Got new version {{ cifmw_update_next_available_version }}
           ({{ openstackversion_info.resources[0].status.deployedVersion }})
 
+- name: Wait for OpenStackControlPlane to be ready after operator update
+  when:
+    - not (cifmw_update_run_dryrun | bool)
+    - cifmw_ci_gen_kustomize_values_deployment_version is defined
+  block:
+    - name: Delay to allow OpenStackControlPlane ready condition to transition
+      # If we install the new operators, the control plane is in 'ready' status, so
+      # we need to wait a few seconds so it changes to a 'non-ready' status first.
+      ansible.builtin.pause:
+        seconds: "{{ cifmw_update_wait_controplane_status_change_sec }}"
+
+    - name: Wait for OpenStackControlPlane ready condition
+      environment:
+        KUBECONFIG: "{{ cifmw_openshift_kubeconfig }}"
+        PATH: "{{ cifmw_path }}"
+      ansible.builtin.command:
+        cmd: >-
+          oc wait OpenStackControlPlane
+          --all
+          --namespace={{ cifmw_update_namespace }}
+          --for=condition=ready
+          --timeout={{ cifmw_update_openstack_update_run_timeout }}
+
 - name: Set openstack_update_run Makefile environment variables
   tags:
     - always