[LoRA] enable LoRA for Mochi-1 #9943
Conversation
| class CogVideoXLoraLoaderMixin(LoraBaseMixin): | ||
| r""" | ||
| Load LoRA layers into [`CogVideoXTransformer3DModel`]. Specific to [`CogVideoX`]. | ||
| Load LoRA layers into [`CogVideoXTransformer3DModel`]. Specific to [`CogVideoXPipeline`]. |
There was a problem hiding this comment.
Unrelated change but doesn't hurt I guess.
| super().unfuse_lora(components=components) | ||
|
|
||
|
|
||
| class Mochi1LoraLoaderMixin(LoraBaseMixin): |
There was a problem hiding this comment.
A copy-paste of the Cog LoRA loader classes, indicated by the "Copied from ..." comments.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| def test_simple_inference_with_text_denoiser_lora_unfused(self): | ||
| super().test_simple_inference_with_text_denoiser_lora_unfused(expected_atol=9e-3) | ||
|
|
||
| @unittest.skip("Not supported in Mochi.") |
There was a problem hiding this comment.
Looks good since not supporting T5 finetuning!
@a-r-r-o-w thanks! Do you think it could make sense to merge regardless with the tests, etc.? Because this will also allow others to experiment from |
|
The tests failing are related to fact that our CI is currently in PyTorch 2.4. #9961 should fix this. Additional info: |
|
Seems like the @yiyixuxu can I merge this PR or should we first investigate the Cc: @BenjaminBossan as well. It's the |
@a-r-r-o-w wants to test it with fine-tune script first, no? |
Well, that is currently done by me here huggingface/finetrainers#90 with Aryan's reviews. However, I don't think the LoRA implementation is dependent on the fine-tuning experiments as it's less likely gonna change (and it's similar to CogVideoX). But okay to wait and will defer to @a-r-r-o-w. |
|
I think should be okay to merge without waiting for finetuning script to work or having an available checkpoint since it will unblock others trying to work on finetuning scripts based on Diffusers (I don't know if there is anyone apart from us yet though). Thanks! |
* feat: add lora support to Mochi-1.
4 participants
What does this PR do?
Needed for fine-tuning.