Batch operator test script fix for Openshift 4.20

[edcdavid]

The changes in this PR fix the following issues with the Openshift 4.20 test run:

• Dependent operators are no longer installed by olm so the openshift storage operators are failing. Added the option to not delete an operator after it is installed by adding a "+" suffix to its package manifest name in the list of operators to run.

• ocs-operator, odf-csi-addons-operator, and mcg-operator are dependent on lvms-operator and odf-operator . So lvms-operator and odf-operator are not uninstalled at the end of their test.

• topology-aware-lifecycle-manager is dependent on advanced-cluster-management and multicluster-engine operators. Also it requires the v1.ManifestWork CR dependency. The v1.ManifestWork CR is created when a MultiClusterEngine is created (added this to the script) :

 apiVersion: multicluster.openshift.io/v1
kind: MultiClusterEngine
metadata:
  name: multiclusterengine # The name is typically 'multiclusterengine'
spec: {}

• Missing operators:

  • power monitoring
  • crunchy-postgres-operator
  • stackable-zookeeper-operator

• odf-csi-addons-operator and mcg-operator-> no suggested namespaces so trying to re-create openshift storage CRs in the "test-namesapce" namespace which is not allowed. Using openshift-storage namespace.

Detailed changes:

• Fix CNF_TYPE_DIR path construction (missing slash)
• Add comprehensive cleanup of test labels from all resources
• Add support for package suffixes (+ for skip cleanup, - for forced test namespace)
• Skip deletion of openshift-storage namespace
• Add special handling for odf-csi-addons-operator and mcg-operator
• Add MultiClusterEngine CR creation for multicluster-engine operator
• Improve CSV labeling with better error handling and label propagation wait
• Increase CSV wait timeout and retry parameters
• Remove unused variable declarations

Update command line for 4.20 operators:

time ./script/run-basic-batch-operators-test.sh registry.redhat.io/redhat/redhat-operator-index:v4.20 "lvms-operator+ odf-operator+ ocs-operator+ advanced-cluster-management+ multicluster-engine+ topology-aware-lifecycle-manager sriov-network-operator local-storage-operator cluster-logging compliance-operator odf-csi-addons-operator cincinnati-operator nfd ptp-operator rhsso-operator file-integrity-operator mcg-operator openshift-cert-manager-operator openshift-gitops-operator quay-operator servicemeshoperator3 metallb-operator kubevirt-hyperconverged gatekeeper-operator-product ansible-automation-platform-operator mtc-operator redhat-oadp-operator openshift-pipelines-operator-rh jaeger kiali-ossm kubernetes-nmstate-operator rhacs-operator  kernel-module-management-hub kernel-module-management mta-operator loki-operator amq-broker-rhel8 amq-streams amq7-interconnect-operator lifecycle-agent numaresources-operator volsync-product rhbk-operator cluster-observability-operator openshift-custom-metrics-autoscaler-operator node-healthcheck-operator self-node-remediation tempo-product"

time ./script/run-basic-batch-operators-test.sh registry.redhat.io/redhat/certified-operator-index:v4.20 "sriov-fec cloud-native-postgresql mongodb-enterprise vault-secrets-operator"

[acornett21]

• Dependent operators are no longer installed by olm

Can you explain this more? In OCP 4.20 I have a dependent operator and it's installed successfully by olm.

[dcibot]

from change #3312:

• FAILURE https://www.distributed-ci.io/jobs/6446b4cd-9b27-4d86-a845-ae4dc5dcb463/jobStates

[dcibot]

from change #3312:

• SUCCESS https://www.distributed-ci.io/jobs/b51cacab-be6d-4f83-b6b7-6bfba0c7ae7d/jobStates

[edcdavid]

• Dependent operators are no longer installed by olm

Can you explain this more? In OCP 4.20 I have a dependent operator and it's installed successfully by olm.

After some in depth investigation, the issue seems to be due to the removal of dependencies.yaml file from the odf-operator bundle in 4.20. So it is not an olm issue. Another operator bundle named odf-dependencies seems to install the dependencies separately.

[acornett21]

After some in depth investigation, the issue seems to be due to the removal of dependencies.yaml file from the odf-operator bundle in 4.20. So it is not an olm issue. Another operator bundle named odf-dependencies seems to install the dependencies separately.

In my testing on a CRC for OCP 4.19 (which is the same bundle as OCP 4.20) cluster, creating a Subscription for odf-operator the instillation is successful, and that operator installs the odf-dependencies operator, which installs all relevant/required operators.

[edcdavid]

After some in depth investigation, the issue seems to be due to the removal of dependencies.yaml file from the odf-operator bundle in 4.20. So it is not an olm issue. Another operator bundle named odf-dependencies seems to install the dependencies separately.

In my testing on a CRC for OCP 4.19 (which is the same bundle as OCP 4.20) cluster, creating a Subscription for odf-operator the instillation is successful, and that operator installs the odf-dependencies operator, which installs all relevant/required operators.

The problem happened when trying to install ocs-operator without installing odf-operator first. ocs-operator is a dependency of odf-operator so it is installed if odf-operator is installed. But if ocs-operator is installed by itself it fails because it expects a CR created by odf-operator.
odf-operator also failed in the test because the script uninstalled the lvms-operator (openshift-storage) after testing it which left pvcs pending.
So part of the fix is to allow operators to remain installed if needed so we can reuse the script to test still test them individually, using the + suffix.