Mast setup

[Ritesh1905]

• Attempt 2, of the mast setup PR
• Has a read me to help setup the mast env.

[allenwang28]

Can we change the design a bit?

• Let's retain the Provisioner as original
• We introduce scheduler specific classes like this:

class BaseScheduler:
    async def initialize(self):
        pass

    def get_allocator(self):
        pass


class MastScheduler(BaseScheduler):
    async def initialize(self):
        await self._build_appdef(...)
        await self._launch_mast_job(...)
        ...

    async def get_allocator(self):
        allocator = MastAllocator(...)
        return allocator, alloc_constraints


class SlurmScheduler(BaseScheduler):
    async def initialize(self):
        # no op

    async def get_allocator(self):
        appdef = hyperactor.host_mesh(...)
        server_info = commands.get_or_create(...)
        alloc = RemoteAllocator(...)
        return alloc, None

There are other things I'd want cleaned up, but I think if we can make this proposed change that is sufficient for the sake of this PR.

[LucasLLC]

Why do this at all? why not always just return commands.get_or_create... ?

[LucasLLC]

just as an observation, imo we should be able to remove most of this complexity with host mesh. Specifically we should use a single host mesh for the entire job, remove TG specific logic, and go from there.

[LucasLLC]

Don't need to address here, but in the future we shouldn't make this a requirement for launching mast jobs. By leaving this open it's actually possible the remote job uses some locally passed assets

[allenwang28]

almost there! Thanks for your patience @Ritesh1905

[allenwang28]

since there isn't such thing as a local launcher, can we remove LOCAL?

[allenwang28]

I would like to keep OmegaConf constrained to only the apps as it isn't explicit enough. The pattern we've been following throughout Forge is something like this:

@dataclass
class LauncherConfig:
    ...

@dataclass
class SlurmConfig:
    ...

@dataclass
class MastConfig:
    ...

Would also like to get rid of the Local launcher:

def get_launcher(cfg: LauncherConfig | None = None) ->  BaseLauncher | None:
    """Returns the launcher given the configuration."""
    if not cfg:
        return None

    if isinstance(cfg, MastConfig):
        return MastLauncher(cfg)
    elif isinstance(cfg, SlurmConfig):
        return SlurmLauncher(cfg)
    else:
        raise ValueError(f"Unsupported config provided, got {cfg}") 

[allenwang28]

Suggested change

	self.launcher: BaseLauncher = get_launcher(cfg)
	self.launcher: BaseLauncher | None = get_launcher(cfg)
	if not self.launcher:
	logger.warning("Launcher not provided, remote allocations will not work.")

[Ritesh1905]

why? Thoughts on defaulting get_launcher to return Slurm, rather than throwing this exception?

[allenwang28]

because people running on like a dev GPU might set this unknowingly and wonder why they're getting a SLURM error

[allenwang28]

Suggested change

	await self.launcher.initialize()
	if self.launcher is not None:
	await self.launcher.initialize()

[allenwang28]

if not self.launcher:
    raise RuntimeError("You tried to create a remote allocation by specifying the number of hosts on an actor or service, but no launcher was specified.")

[daniellepintz]

nit: wrong cmd