Backup and setup automation

[bluzarraga]

What this PR does / why we need it: Automates the installation and setup of OADP on both the backup and restore cluster (if specified), preps the backup cluster for backup by deploying necessary BR resources and labeling CPFS resources, then executes and validates the backup.

Which issue(s) this PR fixes:
Fixes # https://github.ibm.com/IBMPrivateCloud/roadmap/issues/67462 & https://github.ibm.com/IBMPrivateCloud/roadmap/issues/67461

Special notes for your reviewer:

1. How the test is done?

How to backport this PR to other branch:

1. Add label to this PR with the target branch name backport <branch-name>
2. The PR will be automatically created in the target branch after merging this PR
3. If this PR is already merged, you can still add the label with the target branch name backport <branch-name> and leave a comment /backport to trigger the backport action

[ibm-ci-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bluzarraga

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [bluzarraga]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[bluzarraga]

Successful E2E test on two fresh clusters so this PR is ready for review.

Methodology:

• provision and prep two clusters for fresh install of CPFS
• fresh install CPFS on one cluster (but not the other)
• fill in the appropriate parameter values based on desired outcome. For E2E, make sure to enable Backup, Restore, Backup Setup, and Restore setup
• After CPFS instance is ready, from the same cluster, run the auto br script. Make sure to specify the parameter file otherwise it will look for environment variables to be set
• Verify the following milestones are met
  • OADP operator installed on both clusters
  • dataprotectionapplication configured on both clusters
  • Backup labeling completes, velero backup completes with all labeled resources
  • Restore completes on the restore cluster

Especially if zen is installed, it can take time for the entire script to complete so be sure to give it time.

[qpdpQ]

Tested in the cluster, backup succeed

% ./auto-br-oadp.sh --env-file env-oadp.properties
All arguments passed into the auto-br-oadp.sh: --env-file env-oadp.properties


[✔] oc command available
[✔] yq command available
[✔] oc command logged in as kube:admin
[✔] Backup and Restore cluster login credentials verified.
[INFO] checking cluster for existing OADP install...
[INFO] OADP already installed on cluster, skipping oeprator setup.
[INFO] DataProtectionApplication matching parameter DPA_NAME () found in namespace velero. Skipping creation...
# Prepping CPFS instance with Operator NS cs-operator for backup.
[INFO] Labeling cert manager resources in namespaces: cs-operator cs-data 
[INFO] All arguments passed into the label-cert-manager.sh: --namespaces


[✔] oc command logged in as kube:admin
[INFO] NAMESPACES: cs-operator cs-data 
[INFO] Labeling resources in namespace cs-operator
[INFO] No zenservices found in namespace cs-operator, skipping labeling zen custom route secrets...
[INFO] Labeling secret zen-ca-cert-secret in namespace cs-operator...
secret/zen-ca-cert-secret not labeled
---
[INFO] Configmap cs-onprem-tenant-config not found in namespace cs-operator, skipping copying custom secrets...
[INFO] Secret platform-auth-idp-credentials not present in namespace cs-operator. Skipping...
[INFO] Secret user-mgmt-bootstrap not present in namespace cs-operator. Skipping...
[INFO] Secret platform-auth-scim-credentials not present in namespace cs-operator. Skipping...
[INFO] Secret platform-auth-ldaps-ca-cert not present in namespace cs-operator. Skipping...
[INFO] Secret icp-serviceid-apikey-secret not present in namespace cs-operator. Skipping...
[INFO] Secret zen-serviceid-apikey-secret not present in namespace cs-operator. Skipping...
No resources found in cs-operator namespace.
No resources found in cs-operator namespace.
[INFO] Labeling resources in namespace cs-data
[INFO] Labeling Issuers cs-ca-issuer in namespace cs-data...
issuer.cert-manager.io/cs-ca-issuer not labeled
---
[INFO] Labeling Issuers cs-ss-issuer in namespace cs-data...
issuer.cert-manager.io/cs-ss-issuer not labeled
---
[INFO] Labeling Issuers zen-tls-issuer in namespace cs-data...
issuer.cert-manager.io/zen-tls-issuer not labeled
---
[INFO] Labeling issuers.cert-manager.io cs-ca-issuer in namespace cs-data...
issuer.cert-manager.io/cs-ca-issuer not labeled
---
[INFO] Labeling issuers.cert-manager.io cs-ss-issuer in namespace cs-data...
issuer.cert-manager.io/cs-ss-issuer not labeled
---
[INFO] Labeling issuers.cert-manager.io zen-tls-issuer in namespace cs-data...
issuer.cert-manager.io/zen-tls-issuer not labeled
---
[INFO] Labeling certificates cs-ca-certificate in namespace cs-data...
certificate.cert-manager.io/cs-ca-certificate not labeled
---
[INFO] Labeling certificates.cert-manager.io cs-ca-certificate in namespace cs-data...
certificate.cert-manager.io/cs-ca-certificate not labeled
---
[INFO] Labeling secret cs-ca-certificate-secret in namespace cs-data...
secret/cs-ca-certificate-secret not labeled
---
[INFO] No custom zen secret in namespace cs-data, skipping...
[INFO] Labeling secret zen-ca-cert-secret in namespace cs-data...
secret/zen-ca-cert-secret not labeled
---
[INFO] Configmap cs-onprem-tenant-config not found in namespace cs-data, skipping copying custom secrets...
[INFO] Labeling secret platform-auth-idp-credentials in namespace cs-data...
secret/platform-auth-idp-credentials not labeled
---
[INFO] Secret user-mgmt-bootstrap not present in namespace cs-data. Skipping...
[INFO] Labeling secret platform-auth-scim-credentials in namespace cs-data...
secret/platform-auth-scim-credentials not labeled
---
[INFO] Labeling secret platform-auth-ldaps-ca-cert in namespace cs-data...
secret/platform-auth-ldaps-ca-cert not labeled
---
[INFO] Secret icp-serviceid-apikey-secret not present in namespace cs-data. Skipping...
[INFO] Secret zen-serviceid-apikey-secret not present in namespace cs-data. Skipping...
[INFO] removing label from zen-metastore-edb-secret and certificate.
label "foundationservices.cloudpak.ibm.com" not found.
secret/ibm-zen-metastore-edb-secret not labeled
label "foundationservices.cloudpak.ibm.com" not found.
certificate.cert-manager.io/ibm-zen-metastore-edb-certificate not labeled
[✔] Certificates and secrets successfully labeled in namespaces cs-operator cs-data .
[INFO] Labeling CPFS instance and Singleton resources...
[INFO] Labeling script parameters: --operator-ns cs-operator --services-ns cs-data --cert-manager-ns ibm-cert-manager --additional-catalog-sources ibm-operator-catalog-latest
All arguments passed into the label-common-service.sh: --operator-ns cs-operator --services-ns cs-data --cert-manager-ns ibm-cert-manager --additional-catalog-sources ibm-operator-catalog-latest


# Start to validate the parameters passed into script... 
[✔] oc command logged in as kube:admin
# Start to label the catalog sources... 

# Start to label the Subscriptions... 
subscription.operators.coreos.com/ibm-common-service-operator not labeled
subscription.operators.coreos.com/ibm-cert-manager-operator not labeled

[INFO] No UMS CRD found on cluster, skipping...
# Start to label the namespaces, operatorgroups and secrets... 
namespace/cs-operator not labeled
operatorgroup.operators.coreos.com/common-service not labeled
operandrequest.operator.ibm.com/zen-ca-operand-request labeled
customresourcedefinition.apiextensions.k8s.io/zenservices.zen.cpd.ibm.com not labeled
customresourcedefinition.apiextensions.k8s.io/zenextensions.zen.cpd.ibm.com not labeled

namespace/cs-data not labeled
operandrequest.operator.ibm.com/request not labeled
customresourcedefinition.apiextensions.k8s.io/zenservices.zen.cpd.ibm.com not labeled
customresourcedefinition.apiextensions.k8s.io/zenextensions.zen.cpd.ibm.com not labeled
zenservice.zen.cpd.ibm.com/lite-zen labeled

namespace/ibm-cert-manager not labeled
operatorgroup.operators.coreos.com/ibm-cert-manager-5qnrf not labeled
No resources found in ibm-cert-manager namespace.
customresourcedefinition.apiextensions.k8s.io/zenservices.zen.cpd.ibm.com not labeled
customresourcedefinition.apiextensions.k8s.io/zenextensions.zen.cpd.ibm.com not labeled

secret/pull-secret not labeled

# Start to label the ConfigMaps... 
configmap/common-service-maps not labeled
configmap/common-web-ui-config not labeled
configmap/platform-auth-idp not labeled

# Start to label the Subscriptions... 
subscription.operators.coreos.com/ibm-common-service-operator not labeled
subscription.operators.coreos.com/ibm-cert-manager-operator not labeled

# Start to label the Cert Manager resources... 
customresourcedefinition.apiextensions.k8s.io/certmanagerconfigs.operator.ibm.com not labeled
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io not labeled
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io not labeled
[INFO] Start to label the Cert Manager Configs
certmanagerconfig.operator.ibm.com/default not labeled
# Start to label the CommonService CR... 
customresourcedefinition.apiextensions.k8s.io/commonservices.operator.ibm.com not labeled
commonservice.operator.ibm.com/common-service not labeled

# Label Namespacescope resources
subscription.operators.coreos.com/ibm-namespace-scope-operator not labeled
customresourcedefinition.apiextensions.k8s.io/namespacescopes.operator.ibm.com not labeled
namespacescope.operator.ibm.com/common-service not labeled
serviceaccount/ibm-namespace-scope-operator not labeled
role.rbac.authorization.k8s.io/nss-managed-role-from-cs-operator not labeled
role.rbac.authorization.k8s.io/nss-managed-role-from-cs-operator not labeled
rolebinding.rbac.authorization.k8s.io/nss-managed-role-from-cs-operator not labeled
rolebinding.rbac.authorization.k8s.io/nss-managed-role-from-cs-operator not labeled

# Start to label mcsp resources

[✔] Successfully labeled all the resources
[INFO] Deploying necessary backup resources for tenant cs-operator...
[INFO] Backup resource deployment script parameters: --services-ns cs-data --im --zen --storage-class rook-cephfs
[INFO] Using specified storage class rook-cephfs.
[INFO] Creating IM Backup/Restore resources in namespace cs-data.
deployment.apps/cs-db-backup created
persistentvolumeclaim/cs-db-backup-pvc created
configmap/cs-db-br-configmap created
role.rbac.authorization.k8s.io/cs-db-backup-role created
rolebinding.rbac.authorization.k8s.io/cs-db-backup-rolebinding created
serviceaccount/cs-db-backup-sa created
[✔] Resources to backup IM deployed in namespace cs-data.
[INFO] Creating Zen Backup/Restore resources in namespace cs-data.
deployment.apps/zen5-backup created
persistentvolumeclaim/zen5-backup-pvc created
configmap/zen5-br-configmap created
role.rbac.authorization.k8s.io/zen5-backup-role created
rolebinding.rbac.authorization.k8s.io/zen5-backup-rolebinding created
serviceaccount/zen5-backup-sa created
[✔] Resources to backup Zen deployed in namespace cs-data.
[✔] Backup/Restore resources created in namespace cs-data.
[INFO] Waiting for Deployment cs-db-backup to be ready
[✔] Deployment cs-db-backup is running

[INFO] Waiting for Deployment zen5-backup to be ready
[✔] Deployment zen5-backup is running

[✔] CPFS instance with operator namespace cs-operator labeled for backup.
# Starting backup...
[INFO] Copying backup template...
backup.velero.io/test-br created
[INFO] Backup resource created, backup in progress
# Waiting for backup to complete...
[INFO] Wait for backup test-br to complete. Try again in 20 seconds.
[INFO] Wait for backup test-br to complete. Try again in 20 seconds.
[INFO] Wait for backup test-br to complete. Try again in 20 seconds.
[✔] Backup test-br completed successfully. For more details, run "velero backup describe --details test-br".

[qpdpQ]

Hi @bluzarraga, I hit an issue when testing restore

#  Smoke test for Cert Manager existence...
[DEBUG] Creating Issuer test-issuer in namespace cs-data .
../cp3pt0-deployment/common/utils.sh: line 884: /issuer.yaml: Read-only file system
[INFO] Creating following issuer:

cat: /issuer.yaml: No such file or directory

cat: /issuer.yaml: No such file or directory
error: no objects passed to apply
[✘] Failed to create Issuer test-issuer in cs-data 

I think we need to declare two more variables DEBUG and PREVIEW_DIR
It is used in check_cert_manager function:
https://github.com/IBM/ibm-common-service-operator/blob/scripts-adopter/cp3pt0-deployment/common/utils.sh#L884 and https://github.com/IBM/ibm-common-service-operator/blob/scripts-adopter/cp3pt0-deployment/common/utils.sh#L1657

[qpdpQ]

/lgtm tested and verified in the cluster, thank you @bluzarraga for this pr