Skip to content

Commit b136009

Browse files
pawelpaszkipawelpaszki
authored
RHOAIENG-33408: add rayjob tests and bump codeflare-sdk version (#2693)
## WHAT Addition of rayjob tests (with Bryan's latest changes (matching timestamp of this PR) fro m [this PR](project-codeflare/codeflare-sdk#922) and bumping codeflare-sdk tag (to match the upcoming release version) ## VERIFICATION Verified locally with a [test branch](https://github.com/red-hat-data-services/ods-ci/compare/master...pawelpaszki:ods-ci:[RHOAIENG-33408](https://issues.redhat.com//browse/RHOAIENG-33408)-test) with temp tags to execute only relevant tests ``` bash-5.2# git checkout RHOAIENG-33408-test branch 'RHOAIENG-33408-test' set up to track 'pawel/RHOAIENG-33408-test'. Switched to a new branch 'RHOAIENG-33408-test' bash-5.2# export WORKSPACE="/workspace/ods-ci/ods_ci" bash-5.2# cd ods_ci/ bash-5.2# ./run_robot_test.sh --include ttt --skip-oclogin true test-variables.yml ./run_robot_test.sh: line 211: distro: command not found INFO: we found a yq executable skipping OC login as per parameter --skip-oclogin Git revision refname='RHOAIENG-33408-test', venvdir='RHOAIENG-33408-test'. Checking whether '/root/.local/ods-ci/RHOAIENG-33408-test/.venv' exists. Checking whether '/root/.local/ods-ci/master/.venv' exists. Pre-created virtual environment has not been found in '/root/.local/ods-ci/master/.venv'. All dependencies will be installed from scratch. Python '' is not of the correct version Configuring poetry to use Python /root/.pyenv/shims/python3.11 Creating virtualenv ods-ci-VVJNOhYl-py3.11 in /root/.cache/pypoetry/virtualenvs Using virtualenv: /root/.cache/pypoetry/virtualenvs/ods-ci-VVJNOhYl-py3.11 Installing dependencies from lock file Package operations: 239 installs, 0 updates, 0 removals - Installing attrs (24.2.0) - Installing pyasn1 (0.6.1) ... - Installing robotframework-openshift (1.0.0 1297347) Installing the current project: ods-ci (0.1.0) ============================================================================== Tests ============================================================================== Tests.Distributed Workloads ============================================================================== Tests.Distributed Workloads.Workloads Orchestration ============================================================================== 2025-10-14 13:59:32,344 - RPA.core.certificates - INFO - Truststore not in use, HTTPS traffic validated against `certifi` package. (requires Python 3.10.12 and 'pip' 23.2.1 at minimum) Tests.Distributed Workloads.Workloads Orchestration.Test-Run-Codeflare-Sdk-... ============================================================================== Cloning into 'codeflare-sdk'... Note: switching to 'c5ef9a6c5384e167a24c5c1ac261d3a1f6e3d432'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example: git switch -c <new-branch-name> Or undo this operation with: git switch - Turn off this advice by setting config variable advice.detachedHead to false Product:RHODS Version:2.24.0 [ WARN ] No Prometheus found Run TestRayJobRayVersionValidationOauth test with Python 3.11 :: R... "Running codeflare-sdk test: ray_version_validation_oauth_test.py" HEAD is now at c5ef9a6 RHOAIENG-33283: Change ConfigMaps to Secrets * (no branch) Creating virtualenv codeflare-sdk-_B-kuLxP-py3.11 in /root/.cache/pypoetry/virtualenvs Using virtualenv: /root/.cache/pypoetry/virtualenvs/codeflare-sdk-_B-kuLxP-py3.11 Installing dependencies from lock file Package operations: 175 installs, 0 updates, 0 removals - Installing attrs (25.3.0) - Installing rpds-py (0.26.0) .. - Installing python-client (0.0.0-dev b2fd91b) Warning: The file chosen for install of virtualenv 20.35.1 (virtualenv-20.35.1-py3-none-any.whl) is yanked. Reason for being yanked: Backwards incompatible changes Installing the current project: codeflare-sdk (0.31.1) ============================= test session starts ============================== platform linux -- Python 3.11.5, pytest-7.4.0, pluggy-1.6.0 -- /root/.cache/pypoetry/virtualenvs/codeflare-sdk-_B-kuLxP-py3.11/bin/python cachedir: .pytest_cache rootdir: /workspace/ods-ci/ods_ci/codeflare-sdk configfile: pyproject.toml plugins: anyio-4.9.0, mock-3.11.1, timeout-2.3.1 timeout: 900.0s timeout method: signal timeout func_only: False collecting ... collected 2 items tests/e2e/rayjob/ray_version_validation_oauth_test.py::TestRayJobRayVersionValidationOauth::test_rayjob_lifecycled_cluster_incompatible_ray_version_oauth creating Kueue resources ... 'test-resource-flavor-zdzxw' created! 'test-cluster-queue-6p6d6' created 'test-local-queue-97jxx' created in namespace 'test-ns-11cqv' Creating RayJob with incompatible Ray image in cluster config: quay.io/modh/ray:2.46.1-py311-cu121 Attempting to submit RayJob 'incompatible-lifecycle-rayjob' with incompatible Ray version... ✅ Ray version validation correctly prevented RayJob submission with incompatible cluster config! PASSED 'test-cluster-queue-6p6d6' cluster-queue deleted 'test-resource-flavor-zdzxw' resource-flavor deleted tests/e2e/rayjob/ray_version_validation_oauth_test.py::TestRayJobRayVersionValidationOauth::test_rayjob_lifecycled_cluster_unknown_ray_version_oauth creating Kueue resources ... 'test-resource-flavor-apb9z' created! 'test-cluster-queue-lfkds' created 'test-local-queue-6tos0' created in namespace 'test-ns-nlaen' Creating RayJob with image where Ray version cannot be determined: quay.io/modh/ray@sha256:6d076aeb38ab3c34a6a2ef0f58dc667089aa15826fa08a73273c629333e12f1e Attempting to submit RayJob 'unknown-version-rayjob' with unknown Ray version... ✅ RayJob submission succeeded with warning for unknown Ray version! Note: RayJob 'unknown-version-rayjob' was submitted successfully but may need manual cleanup. PASSED 'test-cluster-queue-lfkds' cluster-queue deleted 'test-resource-flavor-apb9z' resource-flavor deleted =============================== warnings summary =============================== ../../../../root/.cache/pypoetry/virtualenvs/codeflare-sdk-_B-kuLxP-py3.11/lib/python3.11/site-packages/_pytest/config/__init__.py:1373 /root/.cache/pypoetry/virtualenvs/codeflare-sdk-_B-kuLxP-py3.11/lib/python3.11/site-packages/_pytest/config/__init__.py:1373: PytestConfigWarning: Unknown config option: collect_ignore self._warn_or_fail_if_strict(f"Unknown config option: {key}\n") tests/e2e/rayjob/ray_version_validation_oauth_test.py::TestRayJobRayVersionValidationOauth::test_rayjob_lifecycled_cluster_incompatible_ray_version_oauth tests/e2e/rayjob/ray_version_validation_oauth_test.py::TestRayJobRayVersionValidationOauth::test_rayjob_lifecycled_cluster_incompatible_ray_version_oauth tests/e2e/rayjob/ray_version_validation_oauth_test.py::TestRayJobRayVersionValidationOauth::test_rayjob_lifecycled_cluster_incompatible_ray_version_oauth tests/e2e/rayjob/ray_version_validation_oauth_test.py::TestRayJobRayVersionValidationOauth::test_rayjob_lifecycled_cluster_unknown_ray_version_oauth tests/e2e/rayjob/ray_version_validation_oauth_test.py::TestRayJobRayVersionValidationOauth::test_rayjob_lifecycled_cluster_unknown_ray_version_oauth tests/e2e/rayjob/ray_version_validation_oauth_test.py::TestRayJobRayVersionValidationOauth::test_rayjob_lifecycled_cluster_unknown_ray_version_oauth /root/.cache/pypoetry/virtualenvs/codeflare-sdk-_B-kuLxP-py3.11/lib/python3.11/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.6.0. Instead access HTTPResponse.headers directly. return self.urllib3_response.getheaders() -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ======================== 2 passed, 7 warnings in 8.21s ========================= Run TestRayJobRayVersionValidationOauth test with Python 3.11 :: R... | PASS | ------------------------------------------------------------------------------ Run TestRayJobExistingCluster test with Python 3.11 :: Run Python ... "Running codeflare-sdk test: rayjob_existing_cluster_test.py" HEAD is now at c5ef9a6 RHOAIENG-33283: Change ConfigMaps to Secrets * (no branch) Using virtualenv: /root/.cache/pypoetry/virtualenvs/codeflare-sdk-_B-kuLxP-py3.11 Installing dependencies from lock file No dependencies to install or update Installing the current project: codeflare-sdk (0.31.1) ============================= test session starts ============================== platform linux -- Python 3.11.5, pytest-7.4.0, pluggy-1.6.0 -- /root/.cache/pypoetry/virtualenvs/codeflare-sdk-_B-kuLxP-py3.11/bin/python cachedir: .pytest_cache rootdir: /workspace/ods-ci/ods_ci/codeflare-sdk configfile: pyproject.toml plugins: anyio-4.9.0, mock-3.11.1, timeout-2.3.1 timeout: 900.0s timeout method: signal timeout func_only: False collecting ... collected 1 item tests/e2e/rayjob/rayjob_existing_cluster_test.py::TestRayJobExistingCluster::test_existing_kueue_cluster creating Kueue resources ... 'test-resource-flavor-ly938' created! 'test-cluster-queue-q79cp' created 'test-local-queue-tn4bl' created in namespace 'test-ns-kh658' Insecure request warnings have been disabled Warning: TLS verification has been disabled - Endpoint checks will be bypassed Written to: /root/.codeflare/resources/kueue-cluster.yaml Written to: /root/.codeflare/resources/kueue-cluster.yaml Ray Cluster: 'kueue-cluster' has successfully been applied. For optimal resource management, you should delete this Ray Cluster when no longer in use. Waiting for cluster 'kueue-cluster' to be ready... Waiting for requested resources to be set up... Requested cluster is up and running! Dashboard is ready! ✓ Cluster 'kueue-cluster' is ready Ray Cluster: 'kueue-cluster' has successfully been deleted PASSED 'test-cluster-queue-q79cp' cluster-queue deleted 'test-resource-flavor-ly938' resource-flavor deleted =============================== warnings summary =============================== ../../../../root/.cache/pypoetry/virtualenvs/codeflare-sdk-_B-kuLxP-py3.11/lib/python3.11/site-packages/_pytest/config/__init__.py:1373 /root/.cache/pypoetry/virtualenvs/codeflare-sdk-_B-kuLxP-py3.11/lib/python3.11/site-packages/_pytest/config/__init__.py:1373: PytestConfigWarning: Unknown config option: collect_ignore self._warn_or_fail_if_strict(f"Unknown config option: {key}\n") tests/e2e/rayjob/rayjob_existing_cluster_test.py::TestRayJobExistingCluster::test_existing_kueue_cluster tests/e2e/rayjob/rayjob_existing_cluster_test.py::TestRayJobExistingCluster::test_existing_kueue_cluster tests/e2e/rayjob/rayjob_existing_cluster_test.py::TestRayJobExistingCluster::test_existing_kueue_cluster /root/.cache/pypoetry/virtualenvs/codeflare-sdk-_B-kuLxP-py3.11/lib/python3.11/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.6.0. Instead access HTTPResponse.headers directly. return self.urllib3_response.getheaders() -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html =================== 1 passed, 4 warnings in 76.72s (0:01:16) =================== Run TestRayJobExistingCluster test with Python 3.11 :: Run Python ... | PASS | ------------------------------------------------------------------------------ Run TestRayJobLifecycledCluster test with Python 3.11 :: Run Pytho... "Running codeflare-sdk test: rayjob_lifecycled_cluster_test.py" HEAD is now at c5ef9a6 RHOAIENG-33283: Change ConfigMaps to Secrets * (no branch) Using virtualenv: /root/.cache/pypoetry/virtualenvs/codeflare-sdk-_B-kuLxP-py3.11 Installing dependencies from lock file No dependencies to install or update Installing the current project: codeflare-sdk (0.31.1) ============================= test session starts ============================== platform linux -- Python 3.11.5, pytest-7.4.0, pluggy-1.6.0 -- /root/.cache/pypoetry/virtualenvs/codeflare-sdk-_B-kuLxP-py3.11/bin/python cachedir: .pytest_cache rootdir: /workspace/ods-ci/ods_ci/codeflare-sdk configfile: pyproject.toml plugins: anyio-4.9.0, mock-3.11.1, timeout-2.3.1 timeout: 900.0s timeout method: signal timeout func_only: False collecting ... collected 2 items tests/e2e/rayjob/rayjob_lifecycled_cluster_test.py::TestRayJobLifecycledCluster::test_lifecycled_kueue_managed creating Kueue resources ... 'test-resource-flavor-4eshz' created! 'test-cluster-queue-6kxf0' created 'test-local-queue-6l8bq' created in namespace 'test-ns-u48bn' ✓ Secret kueue-lifecycled-files verified with proper owner reference PASSED 'test-cluster-queue-6kxf0' cluster-queue deleted 'test-resource-flavor-4eshz' resource-flavor deleted tests/e2e/rayjob/rayjob_lifecycled_cluster_test.py::TestRayJobLifecycledCluster::test_lifecycled_kueue_resource_queueing Creating limited Kueue resources for preemption testing... 'limited-flavor-lvt8y' created! ✓ Created limited ClusterQueue: limited-cq-slkj3 'limited-lq-8k4tm' created in namespace 'test-ns-95l04' ✓ Limited Kueue resources created successfully Waiting for Kueue admission of job 'waiter'... ✓ Job 'waiter' admitted by Kueue (no longer suspended) PASSED 'limited-cq-slkj3' cluster-queue deleted 'limited-flavor-lvt8y' resource-flavor deleted =============================== warnings summary =============================== ../../../../root/.cache/pypoetry/virtualenvs/codeflare-sdk-_B-kuLxP-py3.11/lib/python3.11/site-packages/_pytest/config/__init__.py:1373 /root/.cache/pypoetry/virtualenvs/codeflare-sdk-_B-kuLxP-py3.11/lib/python3.11/site-packages/_pytest/config/__init__.py:1373: PytestConfigWarning: Unknown config option: collect_ignore self._warn_or_fail_if_strict(f"Unknown config option: {key}\n") tests/e2e/rayjob/rayjob_lifecycled_cluster_test.py::TestRayJobLifecycledCluster::test_lifecycled_kueue_managed tests/e2e/rayjob/rayjob_lifecycled_cluster_test.py::TestRayJobLifecycledCluster::test_lifecycled_kueue_managed tests/e2e/rayjob/rayjob_lifecycled_cluster_test.py::TestRayJobLifecycledCluster::test_lifecycled_kueue_managed tests/e2e/rayjob/rayjob_lifecycled_cluster_test.py::TestRayJobLifecycledCluster::test_lifecycled_kueue_resource_queueing tests/e2e/rayjob/rayjob_lifecycled_cluster_test.py::TestRayJobLifecycledCluster::test_lifecycled_kueue_resource_queueing /root/.cache/pypoetry/virtualenvs/codeflare-sdk-_B-kuLxP-py3.11/lib/python3.11/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.6.0. Instead access HTTPResponse.headers directly. return self.urllib3_response.getheaders() -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html =================== 2 passed, 6 warnings in 98.01s (0:01:38) =================== Run TestRayJobLifecycledCluster test with Python 3.11 :: Run Pytho... | PASS | ------------------------------------------------------------------------------ "Removing directory codeflare-sdk" "Log back as cluster admin" "Logging in as cluster admin to cleanup RBAC permissions" "Removing kueue-batch-user-rolebinding ClusterRoleBinding" Error from server (NotFound): error when deleting "tests/Resources/Files/kueue-batch-user-rolebinding.yaml": clusterrolebindings.rbac.authorization.k8s.io "kueue-batch-user-rolebinding" not found "Warning: Unable to delete kueue-batch-user-rolebinding ClusterRoleBinding (may not exist)" "Removing kueue-batch-user-specific-rolebinding ClusterRoleBinding" Error from server (NotFound): error when deleting "tests/Resources/Files/kueue-batch-user-specific-rolebinding.yaml": clusterrolebindings.rbac.authorization.k8s.io "kueue-batch-user-specific-rolebinding" not found "Warning: Unable to delete kueue-batch-user-specific-rolebinding ClusterRoleBinding (may not exist)" "Removing kueue-batch-user-role ClusterRole" Error from server (NotFound): error when deleting "tests/Resources/Files/kueue-batch-user-role.yaml": clusterroles.rbac.authorization.k8s.io "kueue-batch-user-role" not found "Warning: Unable to delete kueue-batch-user-role ClusterRole (may not exist)" Tests.Distributed Workloads.Workloads Orchestration.Test-Run-Codef... | PASS | 3 tests, 3 passed, 0 failed ============================================================================== Tests.Distributed Workloads.Workloads Orchestration | PASS | 3 tests, 3 passed, 0 failed ============================================================================== Tests.Distributed Workloads | PASS | 3 tests, 3 passed, 0 failed ============================================================================== Tests | PASS | 3 tests, 3 passed, 0 failed ============================================================================== Output: /workspace/ods-ci/ods_ci/test-output/ods-ci-2025-10-14-13-59-qDi05w2uXs/output.xml XUnit: /workspace/ods-ci/ods_ci/test-output/ods-ci-2025-10-14-13-59-qDi05w2uXs/xunit_test_result.xml Log: /workspace/ods-ci/ods_ci/test-output/ods-ci-2025-10-14-13-59-qDi05w2uXs/log.html Report: /workspace/ods-ci/ods_ci/test-output/ods-ci-2025-10-14-13-59-qDi05w2uXs/test_report.html 0 ```
2 parents 908c3a5 + bc1f933 commit b136009

File tree

2 files changed

+28
-1
lines changed

2 files changed

+28
-1
lines changed

‎ods_ci/tests/Resources/Page/DistributedWorkloads/DistributedWorkloads.resource‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ Resource ../../../../tasks/Resources/RHODS_OLM/install/oc_install.robot
66

77

88
*** Variables ***
9-
${CODEFLARE-SDK-RELEASE-TAG} v0.31.1
9+
${CODEFLARE-SDK-RELEASE-TAG} v0.32.0
1010
${CODEFLARE-SDK_DIR} codeflare-sdk
1111
${CODEFLARE-SDK_REPO_URL} %{CODEFLARE-SDK_REPO_URL=https://github.com/project-codeflare/codeflare-sdk.git}
1212
${DISTRIBUTED_WORKLOADS_RELEASE_ASSETS} https://github.com/opendatahub-io/distributed-workloads/releases/latest/download

‎ods_ci/tests/Tests/0600__distributed_workloads/0601__workloads_orchestration/test-run-codeflare-sdk-e2e-tests.robot‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,33 @@ Resource ../../../../tests/Resources/Page/DistributedWorkloads/Distribu
1010

1111

1212
*** Test Cases ***
13+
Run TestRayJobRayVersionValidationOauth test with Python 3.11
14+
[Documentation] Run Python E2E test: TestRayJobRayVersionValidationOauth
15+
[Tags]
16+
... Smoke
17+
... DistributedWorkloads
18+
... WorkloadsOrchestration
19+
... Codeflare-sdk
20+
Run Codeflare-SDK Test e2e/rayjob ray_version_validation_oauth_test.py 3.11 ${RAY_CUDA_IMAGE_3.11} ${CODEFLARE-SDK-RELEASE-TAG}
21+
22+
Run TestRayJobExistingCluster test with Python 3.11
23+
[Documentation] Run Python E2E test: TestRayJobExistingCluster
24+
[Tags]
25+
... Tier1
26+
... DistributedWorkloads
27+
... WorkloadsOrchestration
28+
... Codeflare-sdk
29+
Run Codeflare-SDK Test e2e/rayjob rayjob_existing_cluster_test.py 3.11 ${RAY_CUDA_IMAGE_3.11} ${CODEFLARE-SDK-RELEASE-TAG}
30+
31+
Run TestRayJobLifecycledCluster test with Python 3.11
32+
[Documentation] Run Python E2E test: TestRayJobLifecycledCluster
33+
[Tags]
34+
... Tier1
35+
... DistributedWorkloads
36+
... WorkloadsOrchestration
37+
... Codeflare-sdk
38+
Run Codeflare-SDK Test e2e/rayjob rayjob_lifecycled_cluster_test.py 3.11 ${RAY_CUDA_IMAGE_3.11} ${CODEFLARE-SDK-RELEASE-TAG}
39+
1340
Run TestRayClusterSDKOauth test with Python 3.11
1441
[Documentation] Run Python E2E test: TestRayClusterSDKOauth
1542
[Tags]

0 commit comments

Comments
 (0)