K8SPSMDB-1524: add default s3 region by nmarukovich · Pull Request #2191 · percona/percona-server-mongodb-operator

nmarukovich · 2026-01-13T10:20:46Z

CHANGE DESCRIPTION

Problem:
Short explanation of the problem.
PBM defaults S3 storage region to us-east-1 when not specified. The operator compares the CR spec (region: "") with PBM config (region: "us-east-1"), detects a mismatch, and triggers continuous reconciliation loops attempting to remove the region from PBM config.

2025-11-20T14:40:23.838Z        INFO    PBM     Setting config  {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"some-name2","namespace":"demand-backup-physical-minio-21454"}, "namespace": "demand-backup-physical-minio-21454", "name": "some-name2", "reconcileID": "97fe6758-d089-4983-8593-9068915fdf6f", "cluster": "some-name2", "mainStorage": "minio"}
2025-11-20T14:40:23.847Z        INFO    PBM     main storage changed. starting resync   {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"some-name2","namespace":"demand-backup-physical-minio-21454"}, "namespace": "demand-backup-physical-minio-21454", "name": "some-name2", "reconcileID": "97fe6758-d089-4983-8593-9068915fdf6f", "old": {"type":"s3","s3":{"region":"us-east-1","endpointUrl":"http://minio-service:9000/","forcePathStyle":true,"bucket":"operator-testing","prefix":"mongodb/sample","maxUploadParts":10000,"storageClass":"STANDARD","insecureSkipTLSVerify":true,"retryer":{"numMaxRetries":3,"minRetryDelay":100000000000,"maxRetryDelay":300000000000}}}, "new": {"type":"s3","s3":{"region":"","endpointUrl":"http://minio-service:9000/","bucket":"operator-testing","prefix":"mongodb/sample","insecureSkipTLSVerify":true,"retryer":{"numMaxRetries":3,"minRetryDelay":100000000000,"maxRetryDelay":300000000000}}}}
2025-11-20T14:40:31.052Z        INFO    PBM     configuration changed or resync is needed       {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"some-name2","namespace":"demand-backup-physical-minio-21454"}, "namespace": "demand-backup-physical-minio-21454", "name": "some-name2", "reconcileID": "91e84779-55c6-41bd-af26-81e75b34a988", "oldHash": "0734ccf6981f329883e7274f9bd82a7d83de9b6f4b84f0dd79450d21db82d49f", "newHash": "0734ccf6981f329883e7274f9bd82a7d83de9b6f4b84f0dd79450d21db82d49f"}
2025-11-20T14:40:31.071Z        INFO    PBM     Setting config  {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"some-name2","namespace":"demand-backup-physical-minio-21454"}, "namespace": "demand-backup-physical-minio-21454", "name": "some-name2", "reconcileID": "91e84779-55c6-41bd-af26-81e75b34a988", "cluster": "some-name2", "mainStorage": "minio"}
2025-11-20T14:40:31.081Z        INFO    PBM     main storage changed. starting resync   {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"some-name2","namespace":"demand-backup-physical-minio-21454"}, "namespace": "demand-backup-physical-minio-21454", "name": "some-name2", "reconcileID": "91e84779-55c6-41bd-af26-81e75b34a988", "old": {"type":"s3","s3":{"region":"us-east-1","endpointUrl":"http://minio-service:9000/","forcePathStyle":true,"bucket":"operator-testing","prefix":"mongodb/sample","maxUploadParts":10000,"storageClass":"STANDARD","insecureSkipTLSVerify":true,"retryer":{"numMaxRetries":3,"minRetryDelay":100000000000,"maxRetryDelay":300000000000}}}, "new": {"type":"s3","s3":{"region":"","endpointUrl":"http://minio-service:9000/","bucket":"operator-testing","prefix":"mongodb/sample","insecureSkipTLSVerify":true,"retryer":{"numMaxRetries":3,"minRetryDelay":100000000000,"maxRetryDelay":300000000000}}}}

Cause:
Short explanation of the root cause of the issue if applicable.

Solution:
Short explanation of the solution we are providing with this PR.

CHECKLIST

Jira

Is the Jira ticket created and referenced properly?
Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

Is an E2E test/test case added for the new feature/change?
Are unit tests added where appropriate?
Are OpenShift compare files changed for E2E tests (compare/*-oc.yml)?

Config/Logging/Testability

Are all needed new/changed options added to default YAML files?
Are all needed new/changed options added to the Helm Chart?
Did we add proper logging messages for operator actions?
Did we ensure compatibility with the previous version or cluster upgrade process?
Does the change support oldest and newest supported MongoDB version?
Does the change support oldest and newest supported Kubernetes version?

mayankshah1607 · 2026-01-14T05:57:46Z

@nmarukovich please check this PR - #2189

I think it will fix the reconcile flip-flops because call to storage.Cast() will apply the storage defaults used by PBM

egegunes · 2026-01-14T06:41:18Z

e2e-tests/demand-backup-physical-parallel/run

 check_backup_in_storage ${backup_name_minio} minio rs0

+reconciliation_count=$(kubectl_bin logs ${OPERATOR_NS:+-n $OPERATOR_NS} $(get_operator_pod) \
+  | grep "main storage changed. starting resync" \


this log will be removed, see #2193

please check for configuration changed or resync is needed

@egegunes it looks like @mayankshah1607 returned this code.
Could you check one more time

I'll remove this log entry @nmarukovich

Found! I will update test.

nmarukovich · 2026-01-14T08:27:40Z

@nmarukovich please check this PR - #2189

I think it will fix the reconcile flip-flops because call to storage.Cast() will apply the storage defaults used by PBM

I rechecked. It looks like your PR fixed it. I close this one.

e2e-tests/demand-backup-physical-parallel/run

egegunes · 2026-01-16T05:39:42Z

e2e-tests/demand-backup-physical-parallel/run

+#	wait_backup "${backup_name_gcp}"
+#	check_backup_in_storage ${backup_name_gcp} gcs rs0


please revert these

gkech · 2026-01-16T10:35:50Z

deploy/cr.yaml

 #            minRetryDelay: 30ms
 #            maxRetryDelay: 5m
-#          region: us-west-2
+#          region: us-east-1


Since we are changing this, I believe we should adjust the values of the storages names.

note that we configure them using the word west

didn't catch the comment. I change the region here because pbm use us-east-1 as default and adds it to pbm config if region is missed.

so should we change the following?

storages: # s3-us-west: ....

e2e-tests/demand-backup-physical-parallel/run

gkech · 2026-01-16T10:40:25Z

e2e-tests/demand-backup-physical-parallel/run

 wait_backup "${backup_name_minio}"
 check_backup_in_storage ${backup_name_minio} minio rs0

+reconciliation_count=$(kubectl_bin logs ${OPERATOR_NS:+-n $OPERATOR_NS} $(get_operator_pod) \


why this check is needed?

we need to test somehow that the problem fixed. we discussed with @eleo007 and it is the only idea that we have.

gkech · 2026-01-16T10:59:11Z

pkg/apis/psmdb/v1/psmdb_defaults.go

 			}
+
+			if cr.CompareVersion("1.22.0") >= 0 && stg.S3.Region == "" {
+				stg.S3.Region = "us-east-1"


i would move this to a default defaultS3Region = "us-east-1"

…godb-operator into K8SPSMDB-1524

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

JNKPercona · 2026-01-19T11:30:04Z

Test Name	Result	Time
arbiter	passed	00:00:00
balancer	passed	00:00:00
cross-site-sharded	passed	00:00:00
custom-replset-name	passed	00:00:00
custom-tls	passed	00:00:00
custom-users-roles	passed	00:00:00
custom-users-roles-sharded	passed	00:00:00
data-at-rest-encryption	passed	00:00:00
data-sharded	passed	00:00:00
demand-backup	passed	00:00:00
demand-backup-eks-credentials-irsa	passed	00:00:00
demand-backup-fs	passed	00:00:00
demand-backup-if-unhealthy	passed	00:00:00
demand-backup-incremental	passed	00:00:00
demand-backup-incremental-sharded	passed	00:00:00
demand-backup-physical-parallel	passed	00:00:00
demand-backup-physical-aws	passed	00:00:00
demand-backup-physical-azure	passed	00:00:00
demand-backup-physical-gcp-s3	passed	00:00:00
demand-backup-physical-gcp-native	passed	00:00:00
demand-backup-physical-minio	passed	00:00:00
demand-backup-physical-minio-native	passed	00:00:00
demand-backup-physical-sharded-parallel	passed	00:00:00
demand-backup-physical-sharded-aws	passed	00:00:00
demand-backup-physical-sharded-azure	passed	00:00:00
demand-backup-physical-sharded-gcp-native	passed	00:00:00
demand-backup-physical-sharded-minio	passed	00:00:00
demand-backup-physical-sharded-minio-native	passed	00:00:00
demand-backup-sharded	passed	00:00:00
disabled-auth	passed	00:00:00
expose-sharded	passed	00:00:00
finalizer	passed	00:00:00
ignore-labels-annotations	passed	00:00:00
init-deploy	passed	00:00:00
ldap	passed	00:00:00
ldap-tls	passed	00:00:00
limits	passed	00:00:00
liveness	passed	00:00:00
mongod-major-upgrade	passed	00:00:00
mongod-major-upgrade-sharded	passed	00:00:00
monitoring-2-0	passed	00:00:00
monitoring-pmm3	passed	00:00:00
multi-cluster-service	passed	00:12:23
multi-storage	passed	00:00:00
non-voting-and-hidden	passed	00:00:00
one-pod	passed	00:00:00
operator-self-healing-chaos	passed	00:00:00
pitr	passed	00:00:00
pitr-physical	passed	00:00:00
pitr-sharded	passed	00:00:00
pitr-to-new-cluster	passed	00:00:00
pitr-physical-backup-source	passed	00:00:00
preinit-updates	passed	00:00:00
pvc-resize	passed	00:00:00
recover-no-primary	passed	00:00:00
replset-overrides	passed	00:00:00
replset-remapping	passed	00:00:00
replset-remapping-sharded	passed	00:00:00
rs-shard-migration	passed	00:00:00
scaling	passed	00:00:00
scheduled-backup	passed	00:00:00
security-context	passed	00:00:00
self-healing-chaos	passed	00:00:00
service-per-pod	passed	00:00:00
serviceless-external-nodes	passed	00:00:00
smart-update	passed	00:00:00
split-horizon	passed	00:00:00
stable-resource-version	passed	00:00:00
storage	passed	00:00:00
tls-issue-cert-manager	passed	00:00:00
unsafe-psa	passed	00:00:00
upgrade	passed	00:00:00
upgrade-consistency	passed	00:00:00
upgrade-consistency-sharded-tls	passed	00:00:00
upgrade-sharded	passed	00:00:00
upgrade-partial-backup	passed	00:00:00
users	passed	00:00:00
users-vault	passed	00:00:00
version-service	passed	00:00:00

Summary	Value
Tests Run	79/79
Job Duration	00:40:32
Total Test Time	00:12:23

commit: d5bed1c
image: perconalab/percona-server-mongodb-operator:PR-2191-d5bed1c7

K8SPSMDB-1524 add default

f506511

pull-request-size bot added the size/XS 0-9 lines label Jan 13, 2026

nmarukovich added 2 commits January 13, 2026 11:22

update CR

4a0bfaf

update test

ed3a34a

pull-request-size bot added size/S 10-29 lines and removed size/XS 0-9 lines labels Jan 13, 2026

Merge branch 'main' into K8SPSMDB-1524

041afda

nmarukovich marked this pull request as ready for review January 13, 2026 16:40

nmarukovich requested review from bogdanjeler-ev, egegunes, eleo007, gkech, hors, jvpasinatto, mayankshah1607, oksana-grishchenko, pooknull and valmiranogueira as code owners January 13, 2026 16:40

github-actions bot added the tests label Jan 13, 2026

egegunes requested changes Jan 14, 2026

View reviewed changes

egegunes changed the title ~~K8SPSMDB-1524 add default~~ K8SPSMDB-1524: add default s3 region Jan 14, 2026

nmarukovich closed this Jan 14, 2026

nmarukovich reopened this Jan 14, 2026

nmarukovich and others added 3 commits January 14, 2026 16:55

Merge branch 'main' into K8SPSMDB-1524

6ab010a

Merge branch 'main' into K8SPSMDB-1524

8fc5f52

fix PR comments

36f9621

github-actions bot reviewed Jan 15, 2026

View reviewed changes

e2e-tests/demand-backup-physical-parallel/run Outdated Show resolved Hide resolved

egegunes requested changes Jan 16, 2026

View reviewed changes

revert backup

43c95a9

gkech reviewed Jan 16, 2026

View reviewed changes

Merge branch 'main' into K8SPSMDB-1524

05a4899

nmarukovich requested a review from egegunes January 16, 2026 16:02

nmarukovich and others added 4 commits January 16, 2026 17:06

fix PR comments

4edef00

Merge branch 'K8SPSMDB-1524' of github.com:percona/percona-server-mon…

f69931f

…godb-operator into K8SPSMDB-1524

Apply suggestions from code review

caad469

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Merge branch 'main' into K8SPSMDB-1524

d5bed1c

egegunes approved these changes Jan 19, 2026

View reviewed changes

nmarukovich requested a review from gkech January 19, 2026 09:12

mayankshah1607 approved these changes Jan 19, 2026

View reviewed changes

Merge branch 'main' into K8SPSMDB-1524

f9d8c6a

hors merged commit c9718ad into main Jan 19, 2026
13 checks passed

hors deleted the K8SPSMDB-1524 branch January 19, 2026 11:35

		# wait_backup "${backup_name_gcp}"
		# check_backup_in_storage ${backup_name_gcp} gcs rs0

Conversation

nmarukovich commented Jan 13, 2026 • edited by atlassian bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CHANGE DESCRIPTION

CHECKLIST

Uh oh!

mayankshah1607 commented Jan 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nmarukovich commented Jan 14, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JNKPercona commented Jan 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

nmarukovich commented Jan 13, 2026 •

edited by atlassian bot

Loading