Fix multi-arch smoke setup and some flaky test #387

nammn · 2025-08-29T12:08:02Z

Summary

This pull request introduces several improvements to CI/CD configuration, test reliability, and scripting. The main changes are the expansion of allowed requesters for certain build variants, enhancements to the authentication test fixtures for better isolation and reliability, and a simplification of the AWS CLI installation script.

Test Improvements:

Changed test fixtures in replica_set_scram_sha_256_connectivity.py from module scope to function scope, ensuring a fresh environment for each test and reducing side effects between tests.
Refactored secret creation and resource loading in authentication tests to use try_load and explicit secret management, improving test reliability and clarity.
Removed unnecessary attempts parameter from the authentication check in sharded cluster test, simplifying the test logic.
Added "commit" as an allowed requester in several buildvariants sections of .evergreen.yml, enabling these builds to be triggered on direct commits in addition to patches and GitHub tags. -> i hope that fixes smoke tests to run on master merges
Updated install_aws_cli_pip() in scripts/evergreen/setup_aws.sh to check if AWS CLI is already installed and working before attempting installation, preventing unnecessary reinstalls.
Simplified the installation process to always use pip3 and improved error handling and messaging for installation failures.

Proof

green ci
openshift test passed as well
ibm worked manually triggered the build: PATCH
in this patch no smoke and ibm stuff is running
after merge ibm stuff is triggered

Checklist

Have you linked a jira ticket and/or is the ticket in the title?
Have you checked whether your jira ticket required DOCSP changes?
Have you added changelog file?
- use skip-changelog label if not needed
- refer to Changelog files and Release Notes section in CONTRIBUTING.md for more details

github-actions · 2025-08-29T12:08:58Z

⚠️ (this preview might not be accurate if the PR is not rebased on current master branch)

MCK 1.3.0 Release Notes

New Features

Multi-Architecture Support

We've added comprehensive multi-architecture support for the kubernetes operator. This enhancement enables deployment on IBM Power (ppc64le) and IBM Z (s390x) architectures alongside
existing x86_64 support. Core images (operator, agent, init containers, database, readiness probe) now support multiple architectures. We do not add support IBM and ARM support for Ops-Manager and the init-ops-manager image.

MongoDB Agent images have been migrated to new container repository: quay.io/mongodb/mongodb-agent.
- the agents in the new repository will support the x86-64, ARM64, s390x, and ppc64le architectures. More can be read in the public docs.
- operator running >=MCK1.3.0 and static cannot use the agent images from the old container repository quay.io/mongodb/mongodb-agent-ubi.
quay.io/mongodb/mongodb-agent-ubi should not be used anymore, it's only there for backwards compatibility.

Bug Fixes

This change fixes the current complex and difficult-to-maintain architecture for stateful set containers, which relies on an "agent matrix" to map operator and agent versions which led to a sheer amount of images.
We solve this by shifting to a 3-container setup. This new design eliminates the need for the operator-version/agent-version matrix by adding one additional container containing all required binaries. This architecture maps to what we already do with the mongodb-database container.
Fixed an issue where the readiness probe reported the node as ready even when its authentication mechanism was not in sync with the other nodes, potentially causing premature restarts.

Other Changes

Optional permissions for PersistentVolumeClaim moved to a separate role. When managing the operator with Helm it is possible to disable permissions for PersistentVolumeClaim resources by setting operator.enablePVCResize value to false (true by default). When enabled, previously these permissions were part of the primary operator role. With this change, permissions have a separate role.
subresourceEnabled Helm value was removed. This setting used to be true by default and made it possible to exclude subresource permissions from the operator role by specifying false as the value. We are removing this configuration option, making the operator roles always have subresource permissions. This setting was introduced as a temporary solution for this OpenShift issue. The issue has since been resolved and the setting is no longer needed.
We have deliberately not published the container images for OpsManager versions 7.0.16, 8.0.8, 8.0.9 and 8.0.10 due to a bug in the OpsManager which prevents MCK customers to upgrade their OpsManager deployments to those versions.

… test

nammn · 2025-08-29T12:42:25Z

docker/mongodb-kubernetes-tests/tests/authentication/replica_set_scram_sha_256_connectivity.py

@@ -40,7 +40,7 @@ def replica_set(namespace: str, custom_mdb_version) -> MongoDB:
    return resource.update()


-@fixture(scope="module")
+@fixture(scope="function")


we are editing the user and secret later - module scope means we wouldn't return the "updated" user/secret...

blocking: Doesn't this also mean that we will revert any change made to the user?

you are right, that whole setup is bad. Let me rework that whole structure. We should have function for reads and generally split up read and write...

adressed here: 06effa5

ugh i remember that we had the idea of reworking all tests to follow a clear pattern such that we are not running into that again..

nammn · 2025-08-29T12:42:40Z

.../mongodb-kubernetes-tests/tests/authentication/sharded_cluster_scram_sha_256_connectivity.py

@@ -117,7 +117,6 @@ def test_user_can_authenticate_with_new_password(self):
            password="my-new-password",
            username="mms-user-1",
            auth_mechanism="SCRAM-SHA-256",
-            attempts=20,


missed that from prior pr where we changed the default to 50...

1m is way too close to the actual time required to pass the test. When it passed it needed 55 seconds

nammn · 2025-08-29T12:43:12Z

.evergreen.yml

@@ -1475,7 +1475,7 @@ buildvariants:
    run_on:
      - rhel9-power-small
      - rhel9-power-large
-    allowed_requesters: [ "patch", "github_tag" ]
+    allowed_requesters: [ "patch", "github_tag" , "commit"]


rag bot says this is correct; also stated here: https://docs.devprod.prod.corp.mongodb.com/evergreen/Project-Configuration/Project-Configuration-Files#controlling-when-tasks-and-variants-run

by this is correct - I mean merging to master will trigger this task

viveksinghggits

Left some nits, looks good overall.

viveksinghggits · 2025-08-29T14:01:37Z

scripts/evergreen/setup_aws.sh

+    if command -v aws &> /dev/null && aws --version &> /dev/null 2>&1; then
+        echo "AWS CLI is already installed and working:"
+        aws --version


We already ran the aws --version command on line 52, so we don't really have to run it on 54 right? if we are just making sure that aws is working properly.

viveksinghggits · 2025-08-29T14:02:21Z

scripts/evergreen/setup_aws.sh

+    if command -v aws &> /dev/null && aws --version &> /dev/null 2>&1; then
        echo "AWS CLI v1 installed successfully:"
        aws --version


Same comment here.
https://github.com/mongodb/mongodb-kubernetes/pull/387/files#r2310251249

viveksinghggits · 2025-08-29T14:04:28Z

.evergreen.yml

@@ -1520,7 +1520,7 @@ buildvariants:
    tags: [ "e2e_test_suite", "e2e_smoke_release_test_suite" ]
    run_on:
      - ubuntu2204-arm64-large
-    allowed_requesters: [ "patch", "github_tag" ]
+    allowed_requesters: [ "patch", "github_tag", "commit"]


enabling these builds to be triggered on direct commits in addition to patches and GitHub tags

Can you please explain this a bit, just for my understanding. Do you mean, now this test will also be run when a commit is pushed to a PR branch? If yes, was not does if we just had patch?

no - only direct commits to master - which means merges to mainline (mainline is master in our case) in evergreen terminology (more on the linked docs).

"Yes, setting allowed_requesters: ["commit"] is correct if you want your Evergreen task to run only on mainline (master) commits—i.e., after a branch is merged into master. This will prevent the task from running on PRs, patches, or other request types."

lucian-tosa · 2025-08-29T14:32:52Z

docker/mongodb-kubernetes-tests/tests/authentication/replica_set_scram_sha_256_connectivity.py

@@ -40,7 +40,7 @@ def replica_set(namespace: str, custom_mdb_version) -> MongoDB:
    return resource.update()


-@fixture(scope="module")
+@fixture(scope="function")


blocking: Doesn't this also mean that we will revert any change made to the user?

docker/mongodb-kubernetes-tests/tests/authentication/replica_set_scram_sha_256_connectivity.py

nammn added 2 commits August 29, 2025 13:59

not install aws but use instead if already installed

abf07d3

add commit to smoke tests to ensure they run on master merges

1d47cbf

nammn changed the title ~~Fix aws setup~~ Fix multi-arch master merge setup Aug 29, 2025

nammn changed the title ~~Fix multi-arch master merge setup~~ Fix multi-arch smoke setup Aug 29, 2025

nammn added the skip-changelog Use this label in Pull Request to not require new changelog entry file label Aug 29, 2025

increase missed timeout in test and use function scope as we edit the…

8dd0fb0

… test

nammn changed the title ~~Fix multi-arch smoke setup~~ Fix multi-arch smoke setup and some flaky test Aug 29, 2025

nammn commented Aug 29, 2025

View reviewed changes

nammn marked this pull request as ready for review August 29, 2025 13:58

nammn requested a review from a team as a code owner August 29, 2025 13:58

nammn requested review from fealebenpae and viveksinghggits August 29, 2025 13:58

viveksinghggits reviewed Aug 29, 2025

View reviewed changes

nammn added 2 commits August 29, 2025 16:05

address feedback

500eef9

address feedback

71f26d5

nammn requested review from viveksinghggits and lucian-tosa August 29, 2025 14:08

lucian-tosa requested changes Aug 29, 2025

View reviewed changes

address feedback

06effa5

nammn requested a review from lucian-tosa August 29, 2025 15:10

lucian-tosa reviewed Aug 29, 2025

View reviewed changes

docker/mongodb-kubernetes-tests/tests/authentication/replica_set_scram_sha_256_connectivity.py Outdated Show resolved Hide resolved

docker/mongodb-kubernetes-tests/tests/authentication/replica_set_scram_sha_256_connectivity.py Outdated Show resolved Hide resolved

nammn added 2 commits August 29, 2025 18:05

address feedback

40124cf

checkf or ac change instead

bdefeda

lucian-tosa approved these changes Aug 29, 2025

View reviewed changes

checkf or ac change instead

aee5180

viveksinghggits approved these changes Sep 1, 2025

View reviewed changes

nammn merged commit 0da8924 into master Sep 1, 2025
35 of 37 checks passed

nammn deleted the fix-aws-setup branch September 1, 2025 08:38

Fix multi-arch smoke setup and some flaky test #387

Fix multi-arch smoke setup and some flaky test #387

Conversation

nammn commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Proof

Checklist

Uh oh!

github-actions bot commented Aug 29, 2025

MCK 1.3.0 Release Notes

New Features

Multi-Architecture Support

Bug Fixes

Other Changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nammn Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nammn Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viveksinghggits left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nammn Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nammn commented Aug 29, 2025 •

edited

Loading

nammn Aug 29, 2025 •

edited

Loading

nammn Aug 29, 2025 •

edited

Loading

nammn Aug 29, 2025 •

edited

Loading