Skip to content

Conversation

@sayakpaul
Copy link
Member

What does this PR do?

Currently, if we do

scales = {"text_encoder": 0.0, "text_encoder_2": 0.0, "unet": 0.0}
pipe.set_adapters("optimus", adapter_weights=scales)

where pipe is an instance of the FluxPipeline it doesn't error out whereas it should because Flux doesn't have any UNet and its text_encoder_2 component isn't LoRA-loadable:

_lora_loadable_modules = ["transformer", "text_encoder"]

Instead, we silently ignore things. This PR fixes this behavior.

Thanks to @asomoza for the idea in #9542 (comment)!

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Comment on lines +196 to +198
@unittest.skip("Not supported in CogVideoX.")
def test_simple_inference_with_text_denoiser_multi_adapter_block_lora(self):
pass
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we're catching the error appropriately within the code, we should skip this test for unsupported models.

def test_modify_padding_mode(self):
pass

@unittest.skip("Not supported in Flux.")
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gc.collect()
torch.cuda.empty_cache()

@is_flaky
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's absolutely no reason for it to be flaky but I think okay for now.

def test_modify_padding_mode(self):
pass

@is_flaky
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as #9618 (comment).

Copy link
Member

@BenjaminBossan BenjaminBossan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea to check this, LGTM. I have some suggestions for the test, please check if they make sense.

Apart from that, I'm wondering if we still need the checks starting here:

https://github.com/huggingface/diffusers/pull/9618/files#diff-005343920b6fa6c3f4c1f639ee23f180dfa424d6008e487fdc9962a82f498d15R568

Or are those now covered by the newly introduced check and can be removed?

@sayakpaul
Copy link
Member Author

@BenjaminBossan thanks!

Apart from that, I'm wondering if we still need the checks starting here:

Good catch. Resolved in 2714043.

@sayakpaul sayakpaul requested a review from DN6 October 18, 2024 05:59
@sayakpaul
Copy link
Member Author

@DN6 could you give this a look?

@sayakpaul
Copy link
Member Author

@DN6 a gentle ping.

@github-actions
Copy link
Contributor

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

@github-actions github-actions bot added the stale Issues that haven't received updates label Nov 25, 2024
@sayakpaul
Copy link
Member Author

@yiyixuxu @DN6 a gentle ping.

@sayakpaul sayakpaul removed the stale Issues that haven't received updates label Nov 26, 2024
@sayakpaul
Copy link
Member Author

@asomoza @yiyixuxu @DN6 could you give this a look?

@sayakpaul
Copy link
Member Author

@DN6 I have changed from raising error to raising warnings. Additionally, we're removing the invalid components from the adapter_weights. LMK what you think.

@sayakpaul sayakpaul requested a review from DN6 December 16, 2024 10:52
@github-actions
Copy link
Contributor

github-actions bot commented Jan 9, 2025

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

@github-actions github-actions bot added the stale Issues that haven't received updates label Jan 9, 2025
@sayakpaul sayakpaul removed the stale Issues that haven't received updates label Feb 19, 2025
@sayakpaul
Copy link
Member Author

@hlky could you give this a look?

@sayakpaul sayakpaul merged commit 6fe05b9 into main Feb 19, 2025
14 of 15 checks passed
@sayakpaul sayakpaul deleted the set-adapters-robust branch February 19, 2025 09:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants