set is_sequential_cpu_offload = True only when some component is on cpu and has AlignDevicesHook simultaneously
#8750
Conversation
…ify_is_sequential_off_load
|
Let us know when this PR is ready for a review. |
|
alright, i will modify all stuff that is correspoding to |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
…ify_is_sequential_off_load
zhangvia
force-pushed
the
modify_is_sequential_off_load
branch
from
5ca5291 to
eebed22
Compare
|
i have check all stuff is correspoding to |
|
a gentle ping here @sayakpaul @yiyixuxu |
|
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
|
Hi, sorry for the delay here. I've asked Sayak for a review on this |
|
Hi, Thanks for your PR. Could you demonstrate your use-case with some minimal code for us to understand this better? |
sayakpaul
added
the
needs-code-example
| if is_model_cpu_offload or is_sequential_cpu_offload: | ||
| logger.info( | ||
| "Pipeline offload enabled and Accelerate hooks detected. Since you have called `load_lora_weights()`, the previous offload hooks will be first removed. Then the LoRA parameters will be loaded and the hooks will be applied again." | ||
| ) | ||
| remove_hook_from_module(component, recurse=is_sequential_cpu_offload) |
| return False | ||
|
|
||
| return hasattr(module, "_hf_hook") and ( | ||
| return hasattr(module, "_hf_hook") and hasattr(module,'device') and module.device.type == "cpu" and ( |
There was a problem hiding this comment.
manually add a hook to the model and attempting to move it to another GPU does not mean that sequential_cpu_offload is enabled. but the original code will still return true
|
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
|
sorry for delay.
there are two cases that might get error with the current verison of diffusers: when you want to manually add |
4 participants
What does this PR do?
this pr is talking about when the is_sequential_cpu_offload should be set to True.
before we got device_map feature for pipline, if any component in pipeline has AlignDevicesHook which is used to move input data to the model device, we will set the
is_sequential_cpu_offload = True. but when using device_map, we will also add AlignDevicesHook to model.and besides, if someone want to add AignDevicesHook to model manually, is_sequential_cpu_offload will also be set to True.
that would trigger a bug in
load_lora_weights()methodso maybe we should set is_sequential_cpu_offload=True when some component is on cpu and has AlignDevicesHook simultaneously
Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.