K8SPSMDB-1219: PBM multi storage support (#1843)

* K8SPSMDB-1219: PBM multi storage support

Operator always supported multiple storages but didn't have native
support for having multiple backup storages until v2.6.0. In operator we
were reconfiguring PBM every time user selects a storage for their
backups/restores different than the previous storage. This was causing
long wait periods esp. in storages with lots of backups due to resync
operation.

Another limitation was forcing users to have only 1 backup storage if
they want to enable point-in-time-recovery. In case of multiple
storages, PBM would upload oplog chunks to whatever storage is last used
by a backup/restore and this would make consistent recovery impossible.

PBM v2.6.0 added native support for multiple storages and these changes
introduce it to our operator:

* User can have one main storage in PBM configuration. Any other
  storages can be added as profiles.

Main storage can be found in:
```
kubectl exec cluster1-rs0-0 -c backup-agent -- pbm config
```

This commit introduces a new field `main` in storage spec:
```
storages:
  s3-us-west:
    main: true
    type: s3
    s3:
      bucket: operator-testing
      credentialsSecret: cluster1-s3-secrets
      region: us-west-2
````

If user only has 1 storage configured in `cr.yaml`, operator will
automatically use it as main storage. If more than 1 storage is
configured, one of them must have `main: true`. User can't have more
than 1 storage with `main: true`.

If user changes main storage in cr.yaml, operator will configure PBM
with the new storage and start resync.

Any other storage in `cr.yaml` will be added to PBM as a profile.

User can see profiles using cli:
```
kubectl exec cluster1-rs0-0 -c backup-agent -- pbm profile list
```

When user adds a new profile to `cr.yaml`, operator will add it to PBM
but won't start resync.

**`pbm config --force-resync` only start resync for the main storage.**

To manually resync a profile:
```
kubectl exec cluster1-rs0-0 -c backup-agent -- pbm profile sync <storage-name>
```

If user starts a restore using a backup in a storage configured as
profile, operator will start resync operation for profile and block
restore until resync finishes.

Note: Profiles are also called external storages in PBM documentation.

If user has multiple storages in `cr.yaml` and changes main storage
between them, operator:
1. configures PBM with the new main storage.
2. adds the old main as a profile.
2. deletes profile for the new main storage.

If user configures `backupSource` in backups/restores:
* if `cr.yaml` has no storages configured, operator configures PBM with
  storage data in `backupSource` field. This storage will effectively be
  the main storage until user adds a storage to `cr.yaml`. After a
  storage is configured PBM configuration will be overwritten and
  `backupSource` storage will be gone.
* if `cr.yaml` has a storage configured, operator adds `backupSource`
  storage as a profile.

* Oplog chunks will be only be uploaded to main storage.

User can use any backup as base backup for point-in-time-recovery.

* Incremental backup chains all need to be stored in the same storage.

TBD after #1836 merged.

---

Other significant changes in operator behavior:

* Operator now configures automatically configures PBM on a fresh
  cluster.

  Before this changes, PBM was not configured until user starts a
  backup/restore after deploying a fresh cluster. Now, PBM will directly
  configured with main storage in `cr.yaml` and resync will be started
  in the background.

  There's a new field in `PerconaServerMongoDB` status:
  `backupConfigHash`. Operator will maintain hash of the current PBM
  configuration and reconfigures PBM if hash is changed. Fields in
  `spec.backup.pitr` are excluded from hash calculation, they're handled
  separately.
* If `PerconaServerMongoDB` is annotated with `percona.com/resync-pbm=true`,
  operator will start resync operation both for main storage and
  profiles. Resync for profiles are started with the equivalent of `pbm
  profile sync --all`. These resync operations will be run in the
  background and will not block the reconciliation.
* If a backup that has `percona.com/delete-backup` finalizer is deleted,
  operator will only delete oplogs chunks if it's in main storage.

* fix users

* fix repeatedly sending backup commands

* fix users

* address review comments

* assert err in main storage unit test

* don't use break labels

---------

Co-authored-by: Viacheslav Sarzhan <slava.sarzhan@percona.com>

Author: egegunes

build/Dockerfile

@@ -7,6 +7,7 @@ RUN go mod download
 
 ARG GIT_COMMIT
 ARG GIT_BRANCH
+ARG BUILD_TIME
 ARG GO_LDFLAGS
 ARG GOOS=linux
 ARG TARGETARCH
@@ -16,7 +17,7 @@ COPY . .
 
 RUN mkdir -p build/_output/bin \
     && GOOS=$GOOS GOARCH=${TARGETARCH} CGO_ENABLED=$CGO_ENABLED GO_LDFLAGS=$GO_LDFLAGS \
-       go build -ldflags "-w -s -X main.GitCommit=$GIT_COMMIT -X main.GitBranch=$GIT_BRANCH" \
+       go build -ldflags "-w -s -X main.GitCommit=$GIT_COMMIT -X main.GitBranch=$GIT_BRANCH -X main.BuildTime=$BUILD_TIME" \
            -o build/_output/bin/percona-server-mongodb-operator \
                cmd/manager/main.go \
     && cp -r build/_output/bin/percona-server-mongodb-operator /usr/local/bin/percona-server-mongodb-operator

cmd/manager/main.go

@@ -35,6 +35,7 @@ import (
 var (
 	GitCommit string
 	GitBranch string
+	BuildTime string
 	scheme    = k8sruntime.NewScheme()
 	setupLog  = ctrl.Log.WithName("setup")
 )
@@ -69,7 +70,7 @@ func main() {
 	ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts)))
 
 	setupLog.Info("Manager starting up", "gitCommit", GitCommit, "gitBranch", GitBranch,
-		"goVersion", runtime.Version(), "os", runtime.GOOS, "arch", runtime.GOARCH)
+		"buildTime", BuildTime, "goVersion", runtime.Version(), "os", runtime.GOOS, "arch", runtime.GOARCH)
 
 	namespace, err := k8s.GetWatchNamespace()
 	if err != nil {

config/crd/bases/psmdb.percona.com_perconaservermongodbs.yaml

@@ -321,6 +321,8 @@ spec:
                           required:
                           - path
                           type: object
+                        main:
+                          type: boolean
                         s3:
                           properties:
                             bucket:
@@ -18790,6 +18792,8 @@ spec:
             properties:
               backup:
                 type: string
+              backupConfigHash:
+                type: string
               backupVersion:
                 type: string
               conditions:

deploy/bundle.yaml

@@ -1017,6 +1017,8 @@ spec:
                           required:
                           - path
                           type: object
+                        main:
+                          type: boolean
                         s3:
                           properties:
                             bucket:
@@ -19486,6 +19488,8 @@ spec:
             properties:
               backup:
                 type: string
+              backupConfigHash:
+                type: string
               backupVersion:
                 type: string
               conditions:

deploy/crd.yaml

@@ -1017,6 +1017,8 @@ spec:
                           required:
                           - path
                           type: object
+                        main:
+                          type: boolean
                         s3:
                           properties:
                             bucket:
@@ -19486,6 +19488,8 @@ spec:
             properties:
               backup:
                 type: string
+              backupConfigHash:
+                type: string
               backupVersion:
                 type: string
               conditions:

deploy/cw-bundle.yaml

@@ -1017,6 +1017,8 @@ spec:
                           required:
                           - path
                           type: object
+                        main:
+                          type: boolean
                         s3:
                           properties:
                             bucket:
@@ -19486,6 +19488,8 @@ spec:
             properties:
               backup:
                 type: string
+              backupConfigHash:
+                type: string
               backupVersion:
                 type: string
               conditions:

e2e-tests/custom-replset-name/conf/some-name.yml

@@ -12,6 +12,7 @@ spec:
     serviceAccountName: percona-server-mongodb-operator
     storages:
       aws-s3:
+        main: true
         type: s3
         s3:
           credentialsSecret: aws-s3-secret

e2e-tests/custom-users-roles-sharded/conf/some-name-rs0.yml

@@ -63,6 +63,7 @@ spec:
     image: perconalab/percona-server-mongodb-operator:1.1.0-backup
     storages:
       aws-s3:
+        main: true
         type: s3
         s3:
           credentialsSecret: aws-s3-secret

e2e-tests/custom-users-roles/conf/some-name-rs0.yml

@@ -63,6 +63,7 @@ spec:
     image: perconalab/percona-server-mongodb-operator:1.1.0-backup
     storages:
       aws-s3:
+        main: true
         type: s3
         s3:
           credentialsSecret: aws-s3-secret

e2e-tests/data-at-rest-encryption/conf/some-name-unencrypted.yml

@@ -12,6 +12,7 @@ spec:
     image: perconalab/percona-server-mongodb-operator:main-backup
     storages:
       aws-s3:
+        main: true
         type: s3
         s3:
           credentialsSecret: aws-s3-secret
@@ -82,4 +83,4 @@ spec:
         resources:
           requests:
             storage: 1Gi
-    size: 3
+    size: 3