Skip to content

Fix multi-arch smoke setup and some flaky test #387

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Sep 1, 2025
Merged

Fix multi-arch smoke setup and some flaky test #387

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
abf07d3
not install aws but use instead if already installed
nammn Aug 29, 2025
1d47cbf
add commit to smoke tests to ensure they run on master merges
nammn Aug 29, 2025
8dd0fb0
increase missed timeout in test and use function scope as we edit the…
nammn Aug 29, 2025
500eef9
address feedback
nammn Aug 29, 2025
71f26d5
address feedback
nammn Aug 29, 2025
06effa5
address feedback
nammn Aug 29, 2025
40124cf
address feedback
nammn Aug 29, 2025
bdefeda
checkf or ac change instead
nammn Aug 29, 2025
aee5180
checkf or ac change instead
nammn Aug 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions .evergreen.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1475,7 +1475,7 @@ buildvariants:
run_on:
- rhel9-power-small
- rhel9-power-large
allowed_requesters: [ "patch", "github_tag" ]
allowed_requesters: [ "patch", "github_tag" , "commit"]
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rag bot says this is correct; also stated here: https://docs.devprod.prod.corp.mongodb.com/evergreen/Project-Configuration/Project-Configuration-Files#controlling-when-tasks-and-variants-run

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

by this is correct - I mean merging to master will trigger this task

depends_on:
- name: build_operator_ubi
variant: init_test_run
Expand All @@ -1498,7 +1498,7 @@ buildvariants:
run_on:
- rhel9-zseries-small
- rhel9-zseries-large
allowed_requesters: [ "patch", "github_tag" ]
allowed_requesters: [ "patch", "github_tag", "commit"]
depends_on:
- name: build_operator_ubi
variant: init_test_run
Expand All @@ -1520,7 +1520,7 @@ buildvariants:
tags: [ "e2e_test_suite", "e2e_smoke_release_test_suite" ]
run_on:
- ubuntu2204-arm64-large
allowed_requesters: [ "patch", "github_tag" ]
allowed_requesters: [ "patch", "github_tag", "commit"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

enabling these builds to be triggered on direct commits in addition to patches and GitHub tags

Can you please explain this a bit, just for my understanding. Do you mean, now this test will also be run when a commit is pushed to a PR branch? If yes, was not does if we just had patch?

Copy link
Collaborator Author

@nammn nammn Aug 29, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no - only direct commits to master - which means merges to mainline (mainline is master in our case) in evergreen terminology (more on the linked docs).

"Yes, setting allowed_requesters: ["commit"] is correct if you want your Evergreen task to run only on mainline (master) commits—i.e., after a branch is merged into master. This will prevent the task from running on PRs, patches, or other request types."

<<: *base_no_om_image_dependency
tasks:
- name: e2e_smoke_arm_task_group
Expand All @@ -1530,7 +1530,7 @@ buildvariants:
tags: [ "e2e_test_suite", "e2e_smoke_release_test_suite", "static" ]
run_on:
- ubuntu2204-arm64-large
allowed_requesters: [ "patch", "github_tag" ]
allowed_requesters: [ "patch", "github_tag", "commit"]
<<: *base_no_om_image_dependency
tasks:
- name: e2e_smoke_arm_task_group
Expand All @@ -1541,7 +1541,7 @@ buildvariants:
run_on:
- rhel9-zseries-small
- rhel9-zseries-large
allowed_requesters: [ "patch", "github_tag" ]
allowed_requesters: [ "patch", "github_tag", "commit"]
depends_on:
- name: build_operator_ubi
variant: init_test_run
Expand All @@ -1564,7 +1564,7 @@ buildvariants:
run_on:
- rhel9-power-small
- rhel9-power-large
allowed_requesters: [ "patch", "github_tag" ]
allowed_requesters: [ "patch", "github_tag", "commit"]
depends_on:
- name: build_operator_ubi
variant: init_test_run
Expand Down
46 changes: 30 additions & 16 deletions ...r/mongodb-kubernetes-tests/tests/authentication/replica_set_scram_sha_256_connectivity.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
create_secret,
find_fixture,
read_secret,
try_load,
update_secret,
wait_until,
)
Expand All @@ -22,7 +23,16 @@
USER_DATABASE = "admin"


@fixture(scope="module")
def create_password_secret(namespace: str) -> str:
create_or_update_secret(
namespace,
PASSWORD_SECRET_NAME,
{"password": USER_PASSWORD},
)
return PASSWORD_SECRET_NAME


@fixture(scope="function")
def replica_set(namespace: str, custom_mdb_version) -> MongoDB:
resource = MongoDB.from_yaml(
find_fixture("replica-set-scram-sha-256.yaml"),
Expand All @@ -36,37 +46,33 @@ def replica_set(namespace: str, custom_mdb_version) -> MongoDB:
"enabled": True,
"modes": ["SCRAM"],
}

return resource.update()
try_load(resource)
return resource


@fixture(scope="module")
@fixture(scope="function")
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we are editing the user and secret later - module scope means we wouldn't return the "updated" user/secret...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocking: Doesn't this also mean that we will revert any change made to the user?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you are right, that whole setup is bad. Let me rework that whole structure. We should have function for reads and generally split up read and write...

Copy link
Collaborator Author

@nammn nammn Aug 29, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

adressed here: 06effa5

ugh i remember that we had the idea of reworking all tests to follow a clear pattern such that we are not running into that again..

def scram_user(namespace: str) -> MongoDBUser:
resource = MongoDBUser.from_yaml(find_fixture("scram-sha-user.yaml"), namespace=namespace)

create_or_update_secret(
KubernetesTester.get_namespace(),
resource.get_secret_name(),
{"password": USER_PASSWORD},
)

return resource.update()
try_load(resource)
return resource


@fixture(scope="module")
@fixture(scope="function")
def standard_secret(replica_set: MongoDB):
secret_name = "{}-{}-{}".format(replica_set.name, USER_NAME, USER_DATABASE)
return read_secret(replica_set.namespace, secret_name)


@fixture(scope="module")
@fixture(scope="function")
def connection_string_secret(replica_set: MongoDB):
return read_secret(replica_set.namespace, CONNECTION_STRING_SECRET_NAME)


@mark.e2e_replica_set_scram_sha_256_user_connectivity
class TestReplicaSetCreation(KubernetesTester):
def test_replica_set_created(self, replica_set: MongoDB):
replica_set.update()
replica_set.assert_reaches_phase(Phase.Running, timeout=400)

def test_replica_set_connectivity(self, replica_set: MongoDB):
Expand All @@ -82,7 +88,9 @@ def test_ops_manager_state_correctly_updated(self, replica_set: MongoDB):


@mark.e2e_replica_set_scram_sha_256_user_connectivity
def test_create_user(scram_user: MongoDBUser):
def test_create_user(scram_user: MongoDBUser, namespace: str):
create_password_secret(namespace)
scram_user.update()
scram_user.assert_reaches_phase(Phase.Updated)


Expand Down Expand Up @@ -125,9 +133,15 @@ def test_user_cannot_authenticate_with_incorrect_password(self, replica_set: Mon
@mark.e2e_replica_set_scram_sha_256_user_connectivity
class TestCanChangePassword(KubernetesTester):
def test_user_can_authenticate_with_new_password(self, namespace: str, replica_set: MongoDB):
update_secret(namespace, PASSWORD_SECRET_NAME, {"password": "my-new-password7"})
ac_version = replica_set.get_automation_config_tester().automation_config["version"]

new_password = "my-new-password7"
update_secret(namespace, PASSWORD_SECRET_NAME, {"password": new_password})

wait_until(lambda: replica_set.get_automation_config_tester().reached_version(ac_version + 1), timeout=800)

replica_set.tester().assert_scram_sha_authentication(
password="my-new-password7",
password=new_password,
username="mms-user-1",
auth_mechanism="SCRAM-SHA-256",
)
Expand Down
1 change: 0 additions & 1 deletion ...ngodb-kubernetes-tests/tests/authentication/sharded_cluster_scram_sha_256_connectivity.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,6 @@ def test_user_can_authenticate_with_new_password(self):
password="my-new-password",
username="mms-user-1",
auth_mechanism="SCRAM-SHA-256",
attempts=20,
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missed that from prior pr where we changed the default to 50...

Copy link
Collaborator Author

@nammn nammn Aug 29, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1m is way too close to the actual time required to pass the test. When it passed it needed 55 seconds

)

def test_user_cannot_authenticate_with_old_password(self):
Expand Down
17 changes: 8 additions & 9 deletions scripts/evergreen/setup_aws.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,14 +48,14 @@ install_aws_cli_pip() {
return 1
fi

# Use pip3 if available, otherwise pip
local pip_cmd="pip3"
if ! command -v pip3 &> /dev/null; then
pip_cmd="pip"
# Check if AWS CLI exists and works before installing
if command -v aws &> /dev/null && aws --version &> /dev/null 2>&1; then
echo "AWS CLI is already installed and working"
return 0
fi

echo "Installing AWS CLI using ${pip_cmd}..."
${pip_cmd} install --user awscli
echo "Installing AWS CLI using pip3..."
pip3 install --user awscli

# Add ~/.local/bin to PATH if not already there (where pip --user installs)
if [[ ":${PATH}:" != *":${HOME}/.local/bin:"* ]]; then
Expand All @@ -65,10 +65,9 @@ install_aws_cli_pip() {

# Verify installation
if command -v aws &> /dev/null; then
echo "AWS CLI v1 installed successfully:"
aws --version
echo "AWS CLI v1 installed successfully"
else
echo "Error: AWS CLI v1 installation failed or not found in PATH" >&2
echo "Error: AWS CLI v1 installation failed" >&2
return 1
fi
}
Expand Down