KEP Automatic configuration of GPU requests for TrainJobs

[VassilisVassiliadis]

What you would like to be added?

I would like to add intelligent GPU assignment to Kubeflow Trainer, enabling the controller to dynamically determine the appropriate GPU resources (e.g. GPU count, memory, number of replicas, etc) or even training options (e.g. batch size, tuning method, hyper parameters, etc) based on user-provided training configuration.

Examples of configuration options are:

• model type
• sequence length
• parallelism hints
• batch size
• quality of service hints, etc

The goal is to help users avoid having to manually specify options that can be tricky to come up with like resource requirements.

The TrainJob controller should facilitate the mutation of the job before scheduling/admission (e.g., via scheduling gates or suspension).
The mutated job should contain the final configuration options (resources, other settings, etc) so that the scheduler sees the true resource requirements at admission.

Ideally we'd like a way to support trying out different policies for mutating the Jobs in some kind of plugin fashion. This is because it's likely that it's hard to come up with a one-size-fits-all solution that optimally determines the "optimal" resources/configurations for jobs.

Such a mechanism is consistent with the "separation of concerns" design principles of Kubeflow Trainer v2 which aims to simplify distributed training and improve cluster efficiency.
Dynamically determining resources from user intent aligns with Trainer's design for scalable, efficient training on Kubernetes.

Why is this needed?

This feature is needed because users currently must manually specify GPU resources, which can lead to suboptimal utilization and resource pressure especially in multi tenant environments. Additionally, this would enable systems that continuously monitor the capacity and utilization of hardware to intelligently decide the resources of Jobs based on their configuration, priority, etc. This could result in increased hardware utilization and or queue/execution time reduction of jobs.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[github-actions]

🎉 Welcome to the Kubeflow Trainer! 🎉

Thanks for opening your first issue! We're happy to have you as part of our community 🚀

Here's what happens next:

• Our team will review your issue soon! cc @kubeflow/kubeflow-trainer-team
• If you'd like to contribute to this issue, check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards.

Join the community:

• Slack: Join our #kubeflow-trainer Slack channel.
• Meetings: Attend the Kubeflow AutoML and Training Working Group bi-weekly meetings.

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[andreyvelich]

/remove-label lifecycle/needs-triage
/area devices
/area controller

[VassilisVassiliadis]

In kubernetes-sigs/kueue#9891, a user proposes a Kueue feature that dynamically scales the parallelism of a batch/v1 GPU Job at runtime (up to a max, and down to a minimum) to improve hardware utilisation.

I think it's safe to assume that KEP-3328 will likely start with a much narrower scope.

Automatically set a TrainJob’s resource request before the TrainJob's first execution. At a stretch, maybe the KEP also covers some kind of a pluggable policy mechanism so that users and/or platform engineers can pick and choose between different methods.

That said, as long as we do not explode the complexity of the feature we should consider designing the KEP in a way that it can organically evolve in a future iteration to cover advanced scenarios like the proposed dynamic-parallelism in Kueue. For example, I can imagine having an advanced policy for "capacity" borrowing between Jobs.

[andreyvelich]

Thanks for sharing this @VassilisVassiliadis!
I think, your work in kubernetes-sigs/kueue#9792 would be super beneficial for that use-case as well as @kannon92 mentioned. I agree with @mimowo that scaling decision should be delegated to the underlying controllers, and Kueue should have a mechanism to dynamically adjust quota based on those decisions.

I don't think Kueue should modify numNodes or parallelism values.

[VassilisVassiliadis]

Thank you for the sanity check!

[VassilisVassiliadis]

We were just talking with Andrey about this KEP. A rather nice implementation of the proposed API would be a method for "Predictive GPU Capacity Planning for TrainJobs". I'd like to include a user story on this topic which will essentially help TrainJobs avoid getting stuck in queues due to having "sub-optimal" resource requests.