Skip to content

(Randomized) Deferred Model Init#11

Open
pgusto34 wants to merge 5 commits intocomm_schedulingfrom
deferred_init
Open

(Randomized) Deferred Model Init#11
pgusto34 wants to merge 5 commits intocomm_schedulingfrom
deferred_init

Conversation

@pgusto34
Copy link
Collaborator

  • Changed the piper_setup API to use model_configs and constructor arguments instead of passing the model itself
  • Build and compile the model on torch.device("meta") to avoid OOM issues with models that can't fit on a single GPU
  • piper_setup makes a separate API call to materialize the actual tensors on actors after doing a forward pass on the compiled model

@pgusto34 pgusto34 self-assigned this Feb 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant