K8SPSMDB 1472 fixes on sts updates and df execution flows when pvc resizing

[gkech]

Due to the high volume of requests, we're unable to provide free service for this account. To continue using the service, please upgarde to a paid plan.

CHANGE DESCRIPTION

Problem:

During the PVC resize process, either through the manual or the automated fashion, errors regarding the following

Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden"

This happens because when the operator is checking for the resize annotation existence, it is using an outdated sts object. This means that we should fetch the latest instead of relying on the local one.

In this PR we are also handling errors regarding the failure of execution of df due to the pod not being fully available. We do that by wrapping the logic with retries.

Also adding some more validation rules.

Cause:
Short explanation of the root cause of the issue if applicable.

Solution:
Short explanation of the solution we are providing with this PR.

CHECKLIST

Jira

• Is the Jira ticket created and referenced properly?
• Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
• Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

• Is an E2E test/test case added for the new feature/change?
• Are unit tests added where appropriate?
• Are OpenShift compare files changed for E2E tests (compare/*-oc.yml)?

Config/Logging/Testability

• Are all needed new/changed options added to default YAML files?
• Are all needed new/changed options added to the Helm Chart?
• Did we add proper logging messages for operator actions?
• Did we ensure compatibility with the previous version or cluster upgrade process?
• Does the change support oldest and newest supported MongoDB version?
• Does the change support oldest and newest supported Kubernetes version?

[JNKPercona]

Test Name	Result	Time
arbiter	passed	00:11:34
balancer	passed	00:18:19
cross-site-sharded	passed	00:18:53
custom-replset-name	passed	00:10:23
custom-tls	passed	00:14:15
custom-users-roles	passed	00:10:38
custom-users-roles-sharded	passed	00:11:55
data-at-rest-encryption	passed	00:13:03
data-sharded	passed	00:23:22
demand-backup	passed	00:16:05
demand-backup-eks-credentials-irsa	passed	00:00:08
demand-backup-fs	passed	00:24:27
demand-backup-if-unhealthy	passed	00:10:29
demand-backup-incremental-aws	passed	00:12:02
demand-backup-incremental-azure	passed	00:12:00
demand-backup-incremental-gcp-native	passed	00:11:46
demand-backup-incremental-gcp-s3	passed	00:10:59
demand-backup-incremental-minio	passed	00:25:43
demand-backup-incremental-sharded-aws	passed	00:18:38
demand-backup-incremental-sharded-azure	passed	00:17:43
demand-backup-incremental-sharded-gcp-native	passed	00:17:22
demand-backup-incremental-sharded-gcp-s3	passed	00:17:46
demand-backup-incremental-sharded-minio	passed	00:26:32
demand-backup-physical-parallel	passed	00:08:29
demand-backup-physical-aws	passed	00:12:31
demand-backup-physical-azure	passed	00:12:15
demand-backup-physical-gcp-s3	passed	00:11:54
demand-backup-physical-gcp-native	passed	00:11:54
demand-backup-physical-minio	passed	00:20:16
demand-backup-physical-minio-native	passed	00:25:50
demand-backup-physical-minio-native-tls	passed	00:20:00
demand-backup-physical-sharded-parallel	passed	00:11:32
demand-backup-physical-sharded-aws	passed	00:20:02
demand-backup-physical-sharded-azure	passed	00:19:08
demand-backup-physical-sharded-gcp-native	passed	00:18:26
demand-backup-physical-sharded-minio	passed	00:17:49
demand-backup-physical-sharded-minio-native	passed	00:17:36
demand-backup-sharded	passed	00:25:51
disabled-auth	passed	00:16:24
expose-sharded	passed	00:33:51
finalizer	passed	00:10:06
ignore-labels-annotations	passed	00:07:25
init-deploy	passed	00:13:18
ldap	passed	00:08:34
ldap-tls	passed	00:12:38
limits	passed	00:06:19
liveness	passed	00:08:58
mongod-major-upgrade	passed	00:13:20
mongod-major-upgrade-sharded	passed	00:21:14
monitoring-2-0	passed	00:24:48
monitoring-pmm3	passed	00:28:10
multi-cluster-service	passed	00:13:24
multi-storage	passed	00:19:59
non-voting-and-hidden	passed	00:17:19
one-pod	passed	00:08:11
operator-self-healing-chaos	passed	00:12:43
pitr	passed	00:31:59
pitr-physical	passed	01:01:23
pitr-sharded	passed	00:22:32
pitr-to-new-cluster	passed	00:25:31
pitr-physical-backup-source	passed	00:54:35
preinit-updates	passed	00:05:17
pvc-auto-resize	passed	00:14:46
pvc-resize	passed	00:17:07
recover-no-primary	passed	00:28:49
replset-overrides	passed	00:17:50
replset-remapping	passed	00:16:20
replset-remapping-sharded	passed	00:17:44
rs-shard-migration	passed	00:14:18
scaling	passed	00:11:25
scheduled-backup	passed	00:17:31
security-context	passed	00:06:57
self-healing-chaos	passed	00:15:13
service-per-pod	passed	00:19:10
serviceless-external-nodes	passed	00:08:00
smart-update	passed	00:08:07
split-horizon	passed	00:13:54
stable-resource-version	passed	00:04:44
storage	passed	00:07:35
tls-issue-cert-manager	passed	00:29:49
unsafe-psa	passed	00:07:45
upgrade	passed	00:10:06
upgrade-consistency	passed	00:07:54
upgrade-consistency-sharded-tls	passed	00:57:20
upgrade-sharded	passed	00:19:38
upgrade-partial-backup	passed	00:16:30
users	passed	00:17:32
users-vault	passed	00:13:41
version-service	passed	00:25:37

Summary	Value
Tests Run	89/89
Job Duration	02:53:43
Total Test Time	25:31:33

commit: 8a1ac7f
image: perconalab/percona-server-mongodb-operator:PR-2220-8a1ac7ff