[docs] Distributed inference #12285
[docs] Distributed inference #12285
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| > [!TIP] | ||
| > You can use `device_map` within a [`DiffusionPipeline`] to distribute its model-level components on multiple devices. Refer to the [Device placement](../tutorials/inference_with_big_models#device-placement) guide to learn more. | ||
| ## Model sharding |
There was a problem hiding this comment.
I removed it because it seems more like a "recipe" for progressively and strategically fitting models on a GPU by loading and removing them. I don't think a user is really learning anything new/useful about device_map here compared to the device placement docs to which there is a link at the bottom.
I would suggest removing it or at least moving it to Resources > Task Recipes.
There was a problem hiding this comment.
Removal doesn't sound right as the content is useful IMO. Including it in "Task Recipes" might also hamper its discoverability.
There was a problem hiding this comment.
Looks like it's still discarded?
stevhliu
force-pushed
the
distributed-inference
branch
from
b1713b2 to
90409dd
Compare
3 participants
Reduces bloat by removing the
## Model shardingsection because it isn't really about distributed inference (its more selectively loading and deleting models) and doesn't show processing multiple prompts in parallel.