RHOAIENG-32532: Update kueue integration

[kryanbeane]

Issue link

RHOAIENG-32532

What changes have been made

• Added local queue label support to RayJobs
• Updated E2E tests
• Made RayJob E2Es support both Kind and Openshift

Verification steps

Be oc logged into an openshift cluster with Kuberay enabled. Kueue should be disabled in the DSC, and you should install the latest version of RHBoK (RedHat Build of Kueue) installed. Search for Kueue in Operator Hub to find this. Run the below:

pip install -e .
poetry run pytest tests/e2e/rayjob/ -v -s -x

You can also test this manually. Try to create a RayJob with the usual Kueue resources created (follow Pat's KS to set up the resources). You should see the local queue admit the job and the cluster get created.

Check the Workloads CR, there should only be one for the job, and no workloads for the RayCluster.

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • Testing is not required for this change

[openshift-ci-robot]

@kryanbeane: This pull request references RHOAIENG-32532 which is a valid jira issue.

In response to this:

Issue link

RHOAIENG-32532

What changes have been made

• Added local queue label support to RayJobs
• Updated E2E tests
• Made RayJob E2Es support both Kind and Openshift

Verification steps

Be oc logged into an openshift cluster with Kuberay enabled. Kueue should be disabled in the DSC, and you should install the latest version of RHBoK (RedHat Build of Kueue) installed. Search for Kueue in Operator Hub to find this. Run the below:

pip install -e .
poetry run pytest tests/e2e/rayjob/ -v -s -x

You can also test this manually. Try to create a RayJob with the usual Kueue resources created (follow Pat's KS to set up the resources). You should see the local queue admit the job and the cluster get created.

Check the Workloads CR, there should only be one for the job, and no workloads for the RayCluster.

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • Testing is not required for this change

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@kryanbeane: This pull request references RHOAIENG-32532 which is a valid jira issue.

In response to this:

Issue link

RHOAIENG-32532

What changes have been made

• Added local queue label support to RayJobs
• Updated E2E tests
• Made RayJob E2Es support both Kind and Openshift

Verification steps

Be oc logged into an openshift cluster with Kuberay enabled. Kueue should be disabled in the DSC, and you should install the latest version of RHBoK (RedHat Build of Kueue) installed. Search for Kueue in Operator Hub to find this. Run the below:

pip install -e .
poetry run pytest tests/e2e/rayjob/ -v -s -x

You can also test this manually. Try to create a RayJob with the usual Kueue resources created (follow Pat's KS to set up the resources). You should see the local queue admit the job and the cluster get created.

Check the Workloads CR, there should only be one for the job, and no workloads for the RayCluster.

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • Testing is not required for this change

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@kryanbeane: This pull request references RHOAIENG-32532 which is a valid jira issue.

In response to this:

Issue link

RHOAIENG-32532

What changes have been made

• Added local queue label support to RayJobs
• Updated E2E tests
• Made RayJob E2Es support both Kind and Openshift

Verification steps

Be oc logged into an openshift cluster with Kuberay enabled. Kueue should be disabled in the DSC, and you should install the latest version of RHBoK (RedHat Build of Kueue) installed. Search for Kueue in Operator Hub to find this. Run the below:

pip install -e .
poetry run pytest tests/e2e/rayjob/ -v -s -x

You can also test this manually. Try to create a RayJob with the usual Kueue resources created (follow Pat's KS to set up the resources). You should see the local queue admit the job and the cluster get created.

Check the Workloads CR, there should only be one for the job, and no workloads for the RayCluster.

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • Testing is not required for this change

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@kryanbeane: This pull request references RHOAIENG-32532 which is a valid jira issue.

In response to this:

Issue link

RHOAIENG-32532

What changes have been made

• Added local queue label support to RayJobs
• Updated E2E tests
• Made RayJob E2Es support both Kind and Openshift

Verification steps

Be oc logged into an openshift cluster with Kuberay enabled. Kueue should be disabled in the DSC, and you should install the latest version of RHBoK (RedHat Build of Kueue) installed. Search for Kueue in Operator Hub to find this. Run the below:

pip install -e .
poetry run pytest tests/e2e/rayjob/ -v -s -x

You can also test this manually. Try to create a RayJob with the usual Kueue resources created (follow Pat's KS to set up the resources). You should see the local queue admit the job and the cluster get created.

Check the Workloads CR, there should only be one for the job, and no workloads for the RayCluster.

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • Testing is not required for this change

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[codecov]

Codecov Report

❌ Patch coverage is 90.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.04%. Comparing base (8c9dd7d) to head (01c8a7b).
⚠️ Report is 2 commits behind head on ray-jobs-feature.

Files with missing lines	Patch %	Lines
src/codeflare_sdk/ray/cluster/cluster.py	66.66%	2 Missing ⚠️

Additional details and impacted files

@@                 Coverage Diff                  @@
##           ray-jobs-feature     #910      +/-   ##
====================================================
- Coverage             94.17%   94.04%   -0.14%     
====================================================
  Files                    22       22              
  Lines                  1924     1914      -10     
====================================================
- Hits                   1812     1800      -12     
- Misses                  112      114       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[pawelpaszki]

if these tests are to executed during any release testing - can you explain if there are any setup (or other) steps required for the tests execution?

[pawelpaszki]

I have verified the changes on a ROSA cluster by running the tests - looks good. waiting for kind e2e tests to pass

[kryanbeane]

@pawelpaszki

if these tests are to executed during any release testing - can you explain if there are any setup (or other) steps required for the tests execution?

All we'll need is RHBoK installed and RayJob and RayCluster integrations enabled

[kryanbeane]

E2E is still failing. I'll add a fix in a new commit so we only need to re-review that commit 👍🏻

[laurafitzgerald]

/approve
/lgtm

Verified this works as expected.
Observed with RHBOK using SDK I was able to
Submit a RayCluster which was appropriately labelled and admitted.
Submit a RayJob to that existing cluster which was not labelled as expected.
Submit a RayJob to a lifecycled RayCluster which was admitted as expected at the RayJob level.

We do need to ensure that for cluster level kueue that the config map named kueue-manager-config entry has manageJobsWithoutQueueName which allows for RayJobs to be submitted to RayClusters which are already being admitted by kueue.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: laurafitzgerald

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [laurafitzgerald]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment