[Question] General multi-node support

[sfc-gh-truwase]

We are evaluating torchforge adoption for large-scale RL development, and would appreciate this clarification.

1. Are there plans to provide multi-node support beyond slurm (and MAST) launchers?
2. Do you expect supported remote launchers to take dependency on torchx, just as slum launcher does? If so, does that mean ray launcher would not be available since torchx has dropped support.

Thanks!

[allenwang28]

Hi @sfc-gh-truwase,

Monarch has recently added in a Jobs API that currently supports only SLURM. In parallel, we're also exploring adding a K8s-native Jobs API in Monarch as well.

I am planning on re-writing the current remote host / cluster integration logic to take advantage of Jobs API and provide more of an ergonomic workflow, but you could theoretically write your own Ray-based Job implementation as well

[sfc-gh-truwase]

@allenwang28 thanks for the reply. I will take a look at the Jobs API.

[felipemello1]

closing the issue. If you have more questions, please feel free to reopen or create a new one!

[sfc-gh-mwyatt]

@allenwang28 We have looked at the Jobs API you linked to and in particular we found that the SSHJob may be close to what we are looking for. We've created a little prototype and verified that this works for a simple use case:

import os
import sys
from pprint import pprint

from forge.controller.actor import ForgeActor
from monarch.actor import context, endpoint
from monarch._src.job.job import SSHJob

HOSTS = [os.getenv("HOST0", "node-0"), os.getenv("HOST1", "node-1")]
job = SSHJob(python_exe=sys.executable, monarch_port=22222)
job.add_mesh("trainers", HOSTS)

state = job.state()
hosts = state.trainers
procs = hosts.spawn_procs(per_host={"gpus": 2})

class WhoAmI(ForgeActor):
    @endpoint
    def where(self):
        ctx = dict(context().message_rank)
        ctx['hostname'] = os.uname().nodename
        return ctx

actors = procs.spawn("whoami", WhoAmI)
pprint(actors.where.call().get())

job.kill()

We see output that indicates we are successfully able to launch actors multi-node:

ValueMesh({hosts: 2, gpus: 2}):
  (({'hosts': 0/2, 'gpus': 0/2},
    {'gpus': 0, 'hostname': 'node-0', 'hosts': 0}),
   ({'hosts': 0/2, 'gpus': 1/2},
    {'gpus': 1, 'hostname': 'node-0', 'hosts': 0}),
   ({'hosts': 1/2, 'gpus': 0/2},
    {'gpus': 0, 'hostname': 'node-1', 'hosts': 1}),
   ({'hosts': 1/2, 'gpus': 1/2},
    {'gpus': 1, 'hostname': 'node-1', 'hosts': 1}))

However we are not able to launch the ForgeActor objects using as_actor() or as_service(), which we would very much like to do! Do you have any advice for paths forward? Do we need to write a new Launcher class in torchforge? We would be happy to contribute changes that make this sort of multi-node execution possible.

[allenwang28]

@sfc-gh-mwyatt awesome! Yeah I was planning on re-writing the provisioner and launcher logic altogether in the near future, it's sort of a mess as is.

Until then, I think to unblock yourself you should be able to write your own launcher logic. The abstractions are unfortunately kind of wrong, so you will likely need to change the logic for how provisioner gets host mesh (i.e. by grabbing hostmeshes directly from the launcher) as well.

before I re-write the provisioner / logic, I'll post an RFC and tag you if that sounds good?

[sfc-gh-mwyatt]

Thank you! That sounds great. I'll hack away at the current implementation to see if we can unblock ourselves for the time being.