Fix: Make tokenizer_2 and text_encoder_2 truly optional in StableDiffusionXLPipeline #11606
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…usionXLPipeline Previously, the StableDiffusionXLPipeline did not correctly handle cases where `tokenizer_2` and `text_encoder_2` were None, despite them being listed in `_optional_components`. This could lead to errors during prompt encoding. This commit addresses the issue by: 1. Modifying `encode_prompt` to dynamically build lists of tokenizers and text encoders, only including those that are not None. 2. Adjusting the logic for handling `prompt_2` and `negative_prompt_2` to only use them if the corresponding `tokenizer_2` and `text_encoder_2` are present. 3. Ensuring that pooled embeddings are correctly sourced from the last available text encoder. 4. Updating the determination of `text_encoder_projection_dim` in the `__call__` method to correctly use available text encoder configs or fallback to pooled embedding dimensions if `text_encoder_2` is not present. Unit tests have been added to verify the pipeline's behavior when: - Only the primary `tokenizer` and `text_encoder` are present. - Only `tokenizer_2` and `text_encoder_2` are present. - All tokenizer and text encoder components are present.
…nts in StableDiffusionXLPipeline This commit addresses a NameError in `encode_prompt` and ensures that `tokenizer_2` and `text_encoder_2` are truly optional in the StableDiffusionXLPipeline. Changes: - Corrected a `NameError` in `encode_prompt` where `prompt_embeds_output` was used before assignment. The logic now correctly uses the output variable from the text encoder loop (e.g., `current_encoder_output`). - Modified `encode_prompt` to dynamically build lists of tokenizers and text encoders, only including those that are not None. - Adjusted logic for handling `prompt_2` and `negative_prompt_2` to only use them if the corresponding `tokenizer_2` and `text_encoder_2` are present. - Ensured that pooled embeddings are correctly sourced from the last available text encoder and that `text_encoder_projection_dim` in `__call__` is determined robustly. Unit tests focusing on optional component configurations have been reviewed and confirmed to cover the corrected code paths.
|
bah. found bugs. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Background:
I am working on an SDXL varient that exclusively has only (long)CLIP-L.
cf: https://huggingface.co/opendiffusionai/sdxl-longcliponly
Problem that this PR fixes
I would LIKE to be able to just use StableDiffusionXLPipeline.
The text_encoder_2 component is supposedly already marked as optional. However...
Previously, the StableDiffusionXLPipeline did not correctly handle cases where
tokenizer_2andtext_encoder_2were None, despite them being listed in_optional_components. This could lead to errors during prompt encoding.This commit addresses the issue by:
encode_promptto dynamically build lists of tokenizers and text encoders, only including those that are not None.prompt_2andnegative_prompt_2to only use them if the correspondingtokenizer_2andtext_encoder_2are present.text_encoder_projection_dimin the__call__method to correctly use available text encoder configs or fallback to pooled embedding dimensions iftext_encoder_2is not present.Unit tests have been added to verify the pipeline's behavior when:
tokenizerandtext_encoderare present.tokenizer_2andtext_encoder_2are present.What does this PR do?
Fixes # (issue)
Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
Core library: