increas timeout limit and cleanup resource in rerun

[qpdpQ]

What this PR does / why we need it:
increas timeout limit and cleanup resource in rerun
Which issue(s) this PR fixes:
Fixes # https://github.ibm.com/IBMPrivateCloud/roadmap/issues/65997

[qpdpQ]

full test log

./preload_data.sh --rerun --original-cs-ns ibm-common-services --services-ns ltsr-data
All arguments passed into the script: --rerun --original-cs-ns ibm-common-services --services-ns ltsr-data
[✔] oc command available
[✔] yq command available
[✔] oc command logged in as kube:admin
[INFO] Checking cert manager readiness.
issuer.cert-manager.io/test-issuer created
certificate.cert-manager.io/test-certificate created
certificate.cert-manager.io "test-certificate" deleted
issuer.cert-manager.io "test-issuer" deleted
[✔] Cert manager is ready, preload can proceed.
[INFO] Rerun specified...
# Cleaning up any previous copy operations...      <------------First cleanup
-----------------------------------------------------------------------
[✔] Previous run cleaned up.
# Deleting the stand up mongodb statefulset in ltsr-data
-----------------------------------------------------------------------
configmap "ibm-cpp-config" deleted
[✗] Volume for pvc cs-mongodump not found in ltsr-data. It may have already been deleted.
[✔] MongoDB removed from services namespace ltsr-data
[INFO] Copying mongodb from namespace ibm-common-services to namespace ltsr-data
[INFO] Adding full DNS name into icp-mongodb-client-cert certificate in ibm-common-services
[INFO] Adding full DNS name into leaf certificate icp-mongodb runtime in ibm-common-services
deployment.apps/ibm-mongodb-operator scaled
statefulset.apps/icp-mongodb scaled
Warning: resource configmaps/icp-mongodb-init is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically.
configmap/icp-mongodb-init configured
statefulset.apps/icp-mongodb scaled
[INFO] Waiting for MongoDB statefulSet to initialize
NAME          READY   AGE
icp-mongodb   2/3     51d
[INFO] Waiting for MongoDB statefulSet to initialize
NAME          READY   AGE
icp-mongodb   2/3     51d
[INFO] Waiting for MongoDB statefulSet to initialize
NAME          READY   AGE
icp-mongodb   2/3     51d
[INFO] Waiting for MongoDB statefulSet to initialize
NAME          READY   AGE
icp-mongodb   2/3     51d
[INFO] Waiting for MongoDB statefulSet to initialize
NAME          READY   AGE
icp-mongodb   2/3     51d
[INFO] Waiting for MongoDB statefulSet to initialize
NAME          READY   AGE
icp-mongodb   3/3     51d
[✔] DNS name in namespace: ibm-common-services updated
# Cleaning up any previous copy operations...  <------------Second cleanup
-----------------------------------------------------------------------
[✔] Previous run cleaned up.
# Deploying a temporary mongodb in ltsr-data
-----------------------------------------------------------------------
configmap/icp-mongodb-install created
configmap/icp-mongodb-init created
issuer.cert-manager.io/god-issuer created
configmap/ibm-cpp-config created
secret/icp-mongodb-admin configured
certificate.cert-manager.io/icp-mongodb-client-cert created
configmap/icp-mongodb created
secret/icp-mongodb-keyfile created
secret/icp-mongodb-metrics created
serviceaccount/ibm-mongodb-operand created
service/mongodb created
service/icp-mongodb created
certificate.cert-manager.io/mongodb-root-ca-cert created
issuer.cert-manager.io/mongodb-root-ca-issuer created
configmap/namespace-scope created
statefulset.apps/icp-mongodb created
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0   0/1   Init:1/2   0     10s
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0   0/1   Init:1/2   0     21s
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0   0/1   Init:1/2   0     32s
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0   0/1   Init:1/2   0     43s
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0   0/1   Init:1/2   0     53s
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0   0/1   Init:1/2   0     64s
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0   0/1   Init:1/2   0     75s
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0   0/1   Running   0     86s
[✔] Temporary Mongo copy deployed to namespace ltsr-data
# Creating a PVC for the MongoDB dump
-----------------------------------------------------------------------
Now using project "ibm-common-services" on server "https://api.allen.cp.fyre.ibm.com:6443".
persistentvolumeclaim/cs-mongodump created
[✔] MongoDB PVC ready
# Backing up MongoDB in namespace ibm-common-services
-----------------------------------------------------------------------
[INFO] Running Backup
job.batch/mongodb-backup created
mongodb-backup-5mkrl                                   0/1     ContainerCreating            0               0s
[INFO] Waiting for job pod mongodb-backup to complete
[✔] Job mongodb-backup completed in namespace ibm-common-services

[INFO] Saving mongodb-backup logs in _mongodb-backup.log
[INFO] Deleting job mongodb-backup
job.batch "mongodb-backup" deleted
[✔] Backup Complete
# Moving restored mongodb volume to ltsr-data
-----------------------------------------------------------------------
persistentvolume/pvc-f133239b-b99e-4132-b106-108800abfb5e patched
persistentvolumeclaim "cs-mongodump" deleted
persistentvolume/pvc-f133239b-b99e-4132-b106-108800abfb5e patched
persistentvolumeclaim/cs-mongodump created
[✔] Restored MongoDB volume moved to namespace ltsr-data
# Restoring MongoDB to copy in namespace ltsr-data
-----------------------------------------------------------------------
[INFO] Running Restore
job.batch/mongodb-restore created
[INFO] Waiting for job pod mongodb-restore to complete
[✔] Job mongodb-restore completed in namespace ltsr-data

[INFO] Saving mongodb-restore logs in _mongodb-restore.log
[INFO] Deleting job mongodb-restore
job.batch "mongodb-restore" deleted
[✔] Restore Complete
# Deleting the stand up mongodb statefulset in ltsr-data
-----------------------------------------------------------------------
statefulset.apps "icp-mongodb" deleted
service "icp-mongodb" deleted
issuer.cert-manager.io "god-issuer" deleted
configmap "ibm-cpp-config" deleted
certificate.cert-manager.io "icp-mongodb-client-cert" deleted
configmap "icp-mongodb" deleted
configmap "icp-mongodb-init" deleted
configmap "icp-mongodb-install" deleted
secret "icp-mongodb-keyfile" deleted
secret "icp-mongodb-metrics" deleted
serviceaccount "ibm-mongodb-operand" deleted
service "mongodb" deleted
certificate.cert-manager.io "mongodb-root-ca-cert" deleted
issuer.cert-manager.io "mongodb-root-ca-issuer" deleted
configmap "namespace-scope" deleted
persistentvolume/pvc-f133239b-b99e-4132-b106-108800abfb5e patched
persistentvolumeclaim "cs-mongodump" deleted
[✔] MongoDB removed from services namespace ltsr-data
configmap/mongodb-preload-endpoint unchanged
#  Copying secret platform-auth-idp-credentials from ibm-common-services to ltsr-data 
secret/platform-auth-idp-credentials unchanged
[✔] secret platform-auth-idp-credentials copied over to ltsr-data.
#  Copying secret platform-auth-ldaps-ca-cert from ibm-common-services to ltsr-data 
secret/platform-auth-ldaps-ca-cert unchanged
[✔] secret platform-auth-ldaps-ca-cert copied over to ltsr-data.
#  Copying secret platform-oidc-credentials from ibm-common-services to ltsr-data 
secret/platform-oidc-credentials unchanged
[✔] secret platform-oidc-credentials copied over to ltsr-data.
#  Copying secret oauth-client-secret from ibm-common-services to ltsr-data 
secret/oauth-client-secret unchanged
[✔] secret oauth-client-secret copied over to ltsr-data.
#  Copying configmap ibm-cpp-config from ibm-common-services to ltsr-data 
configmap/ibm-cpp-config created
[✔] configmap ibm-cpp-config copied over to ltsr-data.
#  Copying configmap common-web-ui-config from ibm-common-services to ltsr-data 
configmap/common-web-ui-config unchanged
[✔] configmap common-web-ui-config copied over to ltsr-data.
#  Copying configmap platform-auth-idp from ibm-common-services to ltsr-data 
configmap/platform-auth-idp unchanged
[✔] configmap platform-auth-idp copied over to ltsr-data.
#  Copying commonservice common-service from ibm-common-services to ltsr-data 
Deleting storageClass field from commonservice CR
commonservice.operator.ibm.com/preload-common-service-from-ibm-common-services configured
[✔] commonservice preload-common-service-from-ibm-common-services copied over to ltsr-data.
#  Copying secret icp-mongodb-client-cert from ibm-common-services to ltsr-data 
secret/icp-mongodb-client-cert unchanged
[✔] secret icp-mongodb-client-cert copied over to ltsr-data.
#  Copying secret mongodb-root-ca-cert from ibm-common-services to ltsr-data 
secret/mongodb-root-ca-cert unchanged
[✔] secret mongodb-root-ca-cert copied over to ltsr-data.
#  Copying secret icp-mongodb-admin from ibm-common-services to ltsr-data 
secret/icp-mongodb-admin configured
[✔] secret icp-mongodb-admin copied over to ltsr-data.

[Daniel-Fan]

/lgtm

[ibm-ci-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Daniel-Fan, qpdpQ

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [Daniel-Fan,qpdpQ]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment