NooBaa BackingStore in Rejected phase due to "Scaling down the number of nodes is not currently supported"

[naveenpaul1]

Environment info

• NooBaa Operator Version: 4.16.9
• Platform: Kubernetes 1.14.1 | minikube 1.1.1 | OpenShift 4.1 | other: specify

Actual behavior

1. NooBaa's default BackingStore in Rejected phase due to error "Scaling down the number of nodes is not currently supported"

Possible solution is DB clean up, Delete stale node from nodes table. Needs to find why the pod got deleted.

Expected behavior

1. Default BackingStore should be in OPTIMAL state

Steps to reproduce

1. Increase noobaa-default-backing-store backingstore pvPool count to 4, I could see the new pods and PVC are coming
2. Once all the backingstore pods are up and running, I decreased/increased the backingstore pvPool count and same time deleted one noobaa-default-backing-store pod manually.
3. Verified the db, I could see the entry for the deleted backing store inside nodes table and PVC(in the customer cluster, PVC was not there). And noobaa-default-backing-store backingstore in InvalidBackingStore state.

More information - Screenshots / Logs / Other output

Customer bug: https://issues.redhat.com/browse/DFBUGS-3756

[naveenpaul1]

@dannyzaken @liranmauda @jackyalbo @nimrod-becker
In this issue DFBUGS-4800, Observed that the customer manually deleted the default NooBaa BackingStore pod during an upscale. Implementing a validating admission webhook would prevent such manual deletions and help avoid similar issues in the future.
So the plan is to add a webhook that prevents default backingstore pod deletion when backingstore is in a scaling state, which could be upscale or downscale. Getting a BackingStore Object inside the webhook validator is a challeng.
Other option we have is add a finalizer and remove it only when the upscale completes.

Could you please share your thoughts on this?