[Feature] RayJob use HTTP Mode to submit and create a K8s job to tail log

[Future-Outlier]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Users don’t want to incur the costs for the K8s Job because the job submitter pod needs to submit jobs and tail logs for a long time. We can use HTTP mode to submit the RayJob and create a K8s Job to tail logs when the RayJob is in a terminal state.

(For the image in K8s Job, please use the same image as the Ray head by default, but also allow users to specify their own image and entrypoint for this purpose.)

Use case

RayJob.
https://docs.ray.io/en/latest/cluster/kubernetes/getting-started/rayjob-quick-start.html#step-3-install-a-rayjob

Related issues

#3907

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[andrewsykim]

We can use HTTP mode to submit the RayJob and create a K8s Job to tail logs when the RayJob is in a terminal state.

Do you view the purpose of the k8s job is to provide a pod where you can run kubectl logs to feftch the Ray job logs?

[Future-Outlier]

We can use HTTP mode to submit the RayJob and create a K8s Job to tail logs when the RayJob is in a terminal state.

Do you view the purpose of the k8s job is to provide a pod where you can run kubectl logs to feftch the Ray job logs?

Hi @andrewsykim,
Yes!
Do you think there’s a better alternative, or is there some use case I might have missed?

[andrewsykim]

The solution seems a bit overkill to address the cost of the submitter job. Alternatively you can just use a significantly smaller resource request for the submitter job if cost is a real problem. I feel that KubeRay's default of 250m CPU more than enough and could probably be lower.

But in production I would't rely on kubectl logs to get job logs, you should use a solution that can persist the logs that you can query afterwards

[Future-Outlier]

Hi @andrewsykim,

Thank you for sharing your thoughts! I have a few questions to clarify:

Alternatively, you could use a significantly smaller resource request for the submitter job if cost is truly a concern. I feel that KubeRay’s default of 250 m CPU is more than sufficient—and could probably be set lower.

Are you saying that the KubeRay operator’s RayJob controller sends a request to the head Pod to retrieve logs and prints them?

But in production, I wouldn’t rely on kubectl logs to access job logs. You should use a solution that persists the logs and allows querying them later.

Thank you—I’ve learned a lot from this use case. Does this approach relate to the History Server?

[smit-kiri]

I agree with @andrewsykim. We use the submitter pod as a quick way to get live logs for a job in production where we don't have access to the ray dashboard, since our log aggregator picks it up. Getting just the job logs from the Ray cluster's pod logs is painful. It would be great if the image size is reduced and the pod uses very little resources, but I don't want the pod to go away completely

[Future-Outlier]

Summary of KubeRay Sync Meeting
1. The primary objective of this issue is to lower the cost associated with RayJobs.
2. Andrew raised concerns that relying on RayJobs for log tailing may not be a sound approach.
3. To reduce Kubernetes job costs, perhaps we could use a lightweight RayJob submitter image with significantly fewer resources (potentially around 10× less CPU) and conduct stress testing to validate its stability.
4. This ties into the ongoing discussion in the following issue: #3907

cc @andrewsykim
I missed recording the concern you mentioned in point 2 during the meeting—if you remember what it was, could you please share it here?