[Umbrella] Improve RayJob UX, performance, and scalability

[kevin85421]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

• Startup time

  • Light-weight job submitter.

• Stabiility (avoid cross Pod communication) and resource efficiency

  • [Done] Consider injecting an application container for job submission in Ray head Pod. ([Feature] Add Sidecar Mode in Ray Job to avoid cross pod communication #3928)
  • [Won't do] Provide an option that use HTTPMode to create a RayJob and create a submitter to tail logs when the job is in terminal state. ([Feature] RayJob use HTTP Mode to submit and create a K8s job to tail log #3927 (comment))

• Performance

  • Verify multi-threading reconciler stability.
  • Provide configurations for users to configure the frequency to query Ray dashboard instead of sending requests for every reconciliation.
  • Create a background goroutine pool that queries the Ray Dashboard for job statuses and caches the results.

• Benchmark: Create multiple RayJob CRs and each one require autoscaling.

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[Future-Outlier]

take

[Future-Outlier]

take

[kevin85421]

cc @KunWuLuan would you mind leaving a comment so that I can assign it to you? Thanks!

[KunWuLuan]

cc @KunWuLuan would you mind leaving a comment so that I can assign it to you? Thanks!

@kevin85421 ok

[caican00]

I have encountered a problem where RayCluster autoscale up is very slow. I'm very glad to discuss it here.
https://github.com/ray-project/kuberay/issues/ cc @kevin85421
Thanks!

[Future-Outlier]

There are 3 things to do related to the RayJob submitter

1. (Verified with maintainer) Change the Ray Job’s K8s Job mode image to a lightweight image (use HTTP RESTful API and Golang) — target size ideally < 10 MB.
   related issue: [Feature] Light-weight job submitter #2537
2. (Need discussion) Users don’t want to incur the costs for the K8s Job. → We can use HTTP Mode to submit RayJob and create a K8s Job to tail logs when the Ray Job is in a terminal state.
   By default, we will use the same image as the Ray head, but we will allow users to specify their own image and entrypoint for this purpose.
   related issue: [Feature] RayJob use HTTP Mode to submit and create a K8s job to tail log #3927
3. (Need discussion) ApplicationContainer Mode → Add a container (same as the Ray head) in the Ray head Pod to perform job submission and log tailing. (Will open an issue to discuss this.)
   related issue: [Feature] Add Sidecar Mode in Ray Job to avoid cross pod communication #3928