Skip to content

Commit 2961f7f

Browse files
pawelpaszkipawelpaszki
authored
fix (master): pre/post upgrade tests distributed workloads (ray) (#2671)
## WHAT Dw upgrade tests fix (2.25)
2 parents a8f0bce + 6c776c6 commit 2961f7f

File tree

4 files changed

+96
-3
lines changed

4 files changed

+96
-3
lines changed

‎ods_ci/tests/Resources/Files/kueue-batch-user-role.yaml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
apiVersion: rbac.authorization.k8s.io/v1
2+
kind: ClusterRole
3+
metadata:
4+
name: kueue-batch-user-role
5+
rules:
6+
- apiGroups: [""]
7+
resources: ["namespaces"]
8+
verbs: ["create", "get", "list", "watch"]
9+
- apiGroups: ["kueue.x-k8s.io"]
10+
resources: ["clusterqueues", "resourceflavors", "localqueues"]
11+
verbs: ["create", "get", "list", "watch", "update", "patch", "delete"]
12+
- apiGroups: ["ray.io"]
13+
resources: ["rayclusters", "rayjobs"]
14+
verbs: ["create", "get", "list", "watch", "update", "patch", "delete"]
15+
- apiGroups: ["apps"]
16+
resources: ["deployments"]
17+
verbs: ["create", "get", "list", "watch", "update", "patch", "delete"]
18+
- apiGroups: [""]
19+
resources: ["pods", "services", "configmaps", "secrets"]
20+
verbs: ["create", "get", "list", "watch", "update", "patch", "delete"]

‎ods_ci/tests/Resources/Files/kueue-batch-user-specific-rolebinding.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
kind: ClusterRoleBinding
2+
apiVersion: rbac.authorization.k8s.io/v1
3+
metadata:
4+
name: kueue-batch-user-specific-rolebinding
5+
subjects:
6+
- kind: User
7+
apiGroup: rbac.authorization.k8s.io
8+
name: 'ldap-admin1'
9+
roleRef:
10+
apiGroup: rbac.authorization.k8s.io
11+
kind: ClusterRole
12+
name: kueue-batch-user-role

‎ods_ci/tests/Resources/Page/DistributedWorkloads/DistributedWorkloads.resource‎

Lines changed: 63 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,19 @@ Prepare Codeflare-SDK Test Setup
6565
# Perform oc login with Test User
6666
Login To OCP Using API ${TEST_USER.USERNAME} ${TEST_USER.PASSWORD}
6767

68+
Prepare Codeflare-SDK Upgrade Test Setup
69+
[Documentation] Prepare codeflare-sdk upgrade tests with RBAC setup and namespace creation
70+
Clone Git Repository ${CODEFLARE-SDK_REPO_URL} ${CODEFLARE-SDK-RELEASE-TAG} ${CODEFLARE-SDK_DIR}
71+
72+
# Setup Kueue RBAC permissions as admin before switching to test user
73+
Setup Kueue Batch User RoleBinding
74+
75+
# Create the test namespace as admin
76+
Create Test Namespace For Codeflare Test
77+
78+
# Perform oc login with Test User
79+
Login To OCP Using API ${TEST_USER.USERNAME} ${TEST_USER.PASSWORD}
80+
6881
Run Codeflare-SDK Test
6982
[Documentation] Run codeflare-sdk Test
7083
[Arguments] ${TEST_TYPE} ${TEST_NAME} ${PYTHON_VERSION} ${RAY_IMAGE} ${RELEASE_BRANCH}
@@ -106,6 +119,18 @@ Cleanup Codeflare-SDK Setup
106119
Log To Console "Log back as cluster admin"
107120
Login To OCP Using API ${OCP_ADMIN_USER.USERNAME} ${OCP_ADMIN_USER.PASSWORD}
108121

122+
# Cleanup Kueue RBAC resources
123+
Teardown Kueue Batch User RoleBinding
124+
125+
Create Test Namespace For Codeflare Test
126+
[Documentation] Create the test namespace as admin for codeflare tests
127+
Log To Console "Creating test namespace test-ns-rayupgrade as admin"
128+
${rc}= Run And Return Rc oc create namespace test-ns-rayupgrade
129+
130+
IF ${rc} != 0
131+
Log To Console "Warning: Unable to create namespace test-ns-rayupgrade (may already exist)"
132+
END
133+
109134
Prepare Training Operator KFTO E2E Test Suite
110135
[Documentation] Prepare Training Operator KFTO E2E Test Suite
111136
Prepare Training Operator E2E Test Suite ${KFTO_BINARY_NAME}
@@ -321,7 +346,17 @@ Disable appwrapper in Codeflare operator
321346
Log To Console ${result.stdout}
322347

323348
Setup Kueue Batch User RoleBinding
324-
[Documentation] Apply the kueue-batch-user-rolebinding ClusterRoleBinding as admin
349+
[Documentation] Apply the kueue-batch-user-role ClusterRole and kueue-batch-user-rolebinding ClusterRoleBinding as admin
350+
Log To Console "Logging in as cluster admin to setup RBAC permissions"
351+
Login To OCP Using API ${OCP_ADMIN_USER.USERNAME} ${OCP_ADMIN_USER.PASSWORD}
352+
Log To Console "Applying kueue-batch-user-role ClusterRole"
353+
${result} = Run Process oc apply -f tests/Resources/Files/kueue-batch-user-role.yaml
354+
... shell=true
355+
... stderr=STDOUT
356+
Log To Console ${result.stdout}
357+
IF ${result.rc} != 0
358+
FAIL Unable to apply kueue-batch-user-role ClusterRole
359+
END
325360
Log To Console "Applying kueue-batch-user-rolebinding ClusterRoleBinding"
326361
${result} = Run Process oc apply -f tests/Resources/Files/kueue-batch-user-rolebinding.yaml
327362
... shell=true
@@ -330,9 +365,19 @@ Setup Kueue Batch User RoleBinding
330365
IF ${result.rc} != 0
331366
FAIL Unable to apply kueue-batch-user-rolebinding ClusterRoleBinding
332367
END
368+
Log To Console "Applying kueue-batch-user-specific-rolebinding ClusterRoleBinding"
369+
${result} = Run Process oc apply -f tests/Resources/Files/kueue-batch-user-specific-rolebinding.yaml
370+
... shell=true
371+
... stderr=STDOUT
372+
Log To Console ${result.stdout}
373+
IF ${result.rc} != 0
374+
FAIL Unable to apply kueue-batch-user-specific-rolebinding ClusterRoleBinding
375+
END
333376

334377
Teardown Kueue Batch User RoleBinding
335-
[Documentation] Remove the kueue-batch-user-rolebinding ClusterRoleBinding as admin
378+
[Documentation] Remove the kueue-batch-user-role ClusterRole and kueue-batch-user-rolebinding ClusterRoleBinding as admin
379+
Log To Console "Logging in as cluster admin to cleanup RBAC permissions"
380+
Login To OCP Using API ${OCP_ADMIN_USER.USERNAME} ${OCP_ADMIN_USER.PASSWORD}
336381
Log To Console "Removing kueue-batch-user-rolebinding ClusterRoleBinding"
337382
${result} = Run Process oc delete -f tests/Resources/Files/kueue-batch-user-rolebinding.yaml
338383
... shell=true
@@ -341,3 +386,19 @@ Teardown Kueue Batch User RoleBinding
341386
IF ${result.rc} != 0
342387
Log To Console "Warning: Unable to delete kueue-batch-user-rolebinding ClusterRoleBinding (may not exist)"
343388
END
389+
Log To Console "Removing kueue-batch-user-specific-rolebinding ClusterRoleBinding"
390+
${result} = Run Process oc delete -f tests/Resources/Files/kueue-batch-user-specific-rolebinding.yaml
391+
... shell=true
392+
... stderr=STDOUT
393+
Log To Console ${result.stdout}
394+
IF ${result.rc} != 0
395+
Log To Console "Warning: Unable to delete kueue-batch-user-specific-rolebinding ClusterRoleBinding (may not exist)"
396+
END
397+
Log To Console "Removing kueue-batch-user-role ClusterRole"
398+
${result} = Run Process oc delete -f tests/Resources/Files/kueue-batch-user-role.yaml
399+
... shell=true
400+
... stderr=STDOUT
401+
Log To Console ${result.stdout}
402+
IF ${result.rc} != 0
403+
Log To Console "Warning: Unable to delete kueue-batch-user-role ClusterRole (may not exist)"
404+
END

‎ods_ci/tests/Tests/0200__rhoai_upgrade/0201__pre_upgrade.robot‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -94,7 +94,7 @@ Verify Distributed Workload Metrics Resources By Creating Ray Cluster Workload
9494
# robocop: off=too-many-calls-in-test-case
9595
[Documentation] Creates the Ray Cluster and verify resource usage
9696
[Tags] Upgrade WorkloadOrchestration
97-
[Setup] Prepare Codeflare-SDK Test Setup
97+
[Setup] Prepare Codeflare-SDK Upgrade Test Setup
9898
${PRJ_UPGRADE}= Set Variable test-ns-rayupgrade
9999
${JOB_NAME}= Set Variable mnist
100100
Run Codeflare-SDK Test

0 commit comments

Comments
 (0)