Multinode SkyPilot example

[alex000kim]

Adding an example on how to run the same job as in multinode_trainer.slurm but using SkyPilot

[meta-cla]

Hi @alex000kim!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at [email protected]. Thanks!

[meta-cla]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

[wwwjn]

Hi @alex000kim Thanks for contributing! I've heard of skypilot and this is a really cool product. However, I haven't used it myself or I'm sure how many users are using it. If it's widely used, we could consider add it main README. Otherwise, I would prefer host the instruction somewhere else (eg, can we have a page in Skypilot docs?) to avoid confusing people.

[alex000kim]

@wwwjn Thanks!
I believe it's very widely used as an alternative to Slurm and AWS Batch.

• It has 8.5k+ GH stars
• I have experience working many AI startups how train on large H100 and H200 clusters on Nebius AI and Lambda Labs. Quite a few of them do so using SkyPilot.
• There's also a very active user community in https://slack.skypilot.co/

Or were you looking for a different kind of evidence.

eg, can we have a page in Skypilot docs?

I can add the same example to SkyPilot docs as well.

[alex000kim]

Following up with some relevant links that mention SkyPilot (partnerships, integrations and research publications):

• https://research.ibm.com/publications/when-skypilot-meets-kubernetes
• https://aws.amazon.com/blogs/machine-learning/streamline-machine-learning-workflows-with-skypilot-on-amazon-sagemaker-hyperpod/
• https://cloud.google.com/blog/topics/hpc/salk-institute-brain-mapping-on-google-cloud-with-skypilot
• https://gke-ai-labs.dev/docs/tutorials/workflow-orchestration/skypilot/cross-region-capacity-chasing/
• https://docs.nvidia.com/nemo-framework/user-guide/24.09/nemorun/guides/execution.html
• https://nebius.com/blog/posts/nebius-ai-cloud-skypilot-integration
• https://docs.runpod.io/integrations/skypilot
• https://vast.ai/article/vast-ai-gpus-can-now-be-rentend-through-skypilot
• https://blog.skypilot.co/covariant/
• https://blog.paperspace.com/skypilot-seamless-integration-paperspace/
• https://a16z.com/announcing-our-latest-open-source-ai-grants/
• https://lambda.ai/blog/kubernetes-cluster-deployment-made-easy-with-lambda-and-skypilot

So as you can see SkyPilot is used by/integrated with most major hyperscalers and neoclouds

[wwwjn]

Following up with some relevant links that mention SkyPilot (partnerships, integrations and research publications):

• https://research.ibm.com/publications/when-skypilot-meets-kubernetes
• https://aws.amazon.com/blogs/machine-learning/streamline-machine-learning-workflows-with-skypilot-on-amazon-sagemaker-hyperpod/
• https://cloud.google.com/blog/topics/hpc/salk-institute-brain-mapping-on-google-cloud-with-skypilot
• https://gke-ai-labs.dev/docs/tutorials/workflow-orchestration/skypilot/cross-region-capacity-chasing/
• https://docs.nvidia.com/nemo-framework/user-guide/24.09/nemorun/guides/execution.html
• https://nebius.com/blog/posts/nebius-ai-cloud-skypilot-integration
• https://docs.runpod.io/integrations/skypilot
• https://vast.ai/article/vast-ai-gpus-can-now-be-rentend-through-skypilot
• https://blog.skypilot.co/covariant/
• https://blog.paperspace.com/skypilot-seamless-integration-paperspace/
• https://a16z.com/announcing-our-latest-open-source-ai-grants/
• https://lambda.ai/blog/kubernetes-cluster-deployment-made-easy-with-lambda-and-skypilot

So as you can see SkyPilot is used by/integrated with most major hyperscalers and neoclouds

Thank you so much for introducing SkyPilot! I do think this is a great project but from torchtitan's point of view, I don't think we want to introduce every too that used to launch torchtitan runs, and we want to make our doc simple and easy to read. We want to keep torchtitan simple focus on training and let user to choose what framework they want to use to launch jobs.

[alex000kim]

Understood. Thanks for taking a look.