[wip] [core] introduce pipeline-specific mixins to hold common methods

[sayakpaul]

What does this PR do?

As discussed internally, this PR introduces pipeline-specific Mixin classes that hold common methods shared across different task-specific pipelines. The actual pipelines (such as StableDiffusionPipeline, QwenImagePipeline, QwenImageImg2ImgPipeline, etc.) would then subclass from these mixins.

Examples of such common methods vary from pipeline to pipeline, but some examples include encode_prompt(), properties such as guidance_scale, etc.

Additionally, we have a couple of methods like retrieve_latents() and retrieve_timesteps() originated in stable_diffusion and we copied them over to other pipelines that use these methods. Same for calculate_shift that originated in flux. This PR also considers this situation and follows a reasonable approach to treat it (more below).

I know we also discussed moving the specific LoraLoaderMixins to their respective pipeline modules. The PR only does that for Stable Diffusion to gather feedback. Please keep on reading.

Note

This PR introduces Mixin classes for four popular pipelines: Flux, Qwen, SDXL, and SD. Once the PR is merged, we could work with the community to do this for other influential pipelines.

---

Guiding principles

A sample structure for src/diffusers/pipelines/flux:

.
├── __init__.py
├── modeling_flux.py
├── pipeline_flux_control_img2img.py
├── pipeline_flux_control_inpaint.py
├── pipeline_flux_control.py
├── pipeline_flux_controlnet_image_to_image.py
├── pipeline_flux_controlnet_inpainting.py
├── pipeline_flux_controlnet.py
├── pipeline_flux_fill.py
├── pipeline_flux_img2img.py
├── pipeline_flux_inpaint.py
├── pipeline_flux_kontext_inpaint.py
├── pipeline_flux_kontext.py
├── pipeline_flux_prior_redux.py
├── pipeline_flux_utils.py <------ holds the Mixin
├── pipeline_flux.py
└── pipeline_output.py

pipeline_flux_utils.py here has the Mixin class, holding the common methods. It additionally has the calculate_shift() method, which is imported by the other pipelines under src/diffusers/pipelines/flux. Those pipelines won't have to maintain # Copied from ... versions of the calculate_shift() method anymore. They can just import like from .pipeline_flux_utils import calculate_shift. This is how we reduce LOCs, further on an intra-pipeline basis.

However, if a new pipeline (other than Flux), wants to leverage calculate_shift(), it should still follow # Copied from ... versions of the calculate_shift() instead of a direct import. This way, we can let individual pipeline-level methods evolve independently without introducing complex dependency patterns. For example, pipeline_flux_utils.py uses the # Copied from ... versions of methods like retrieve_latents().

LoRA

I experimented with moving StableDiffusionLoraLoaderMixin to src/diffusers/pipelines/stable_diffusion/lora_utils.py. However, there are a couple of derivative SD pipelines that subclass from StableDiffusionLoraLoaderMixin (imported from lora_pipeline.py).

So, that will deprecation warnings for those. Two approaches come to mind:

• Do from ..pipelines.stable_diffusion.lora_utils import StableDiffusionLoraLoaderMixin. I don't like it at all because the dependency pattern is bad.
• Do a "# Copied from ... version of StableDiffusionLoraLoaderMixin inside a lora_utils.py the respective pipeline module. src/diffusers/pipelines/animatediff, for example. I prefer this one.

WDYT?

Also, if we decided to deprecate StableDiffusionLoraLoaderMixin from diffusers.loaders.lora_pipeline, it would lead to a circular import problem:
https://github.com/huggingface/diffusers/actions/runs/20021061502/job/57407882632?pr=12322#step:16:80

So, if we decided to go this route of separating the LoRA loaders, this might have to be a breaking change.

---

Notes

• In the Qwen mixin, kept encode_prompt() varying while unifying the _get_qwen_prompt_embeds() method in QwenImageMixin. Comments are in line.
• There are a few methods that are shared by 2/3 pipelines such as prepare_image() is shared by QwenImageControlNetInpaintPipeline and QwenImageControlNetPipeline but not others. We can choose to move them to pipeline_qwen_utils.py but I am not strongly opinionated on this. The same applies to methods like prepare_mask_latents(). My preference here is to keep them as is for now.

We can take similar principles for Flux, too. Flux Control and ControlNet pipelines have a prepare_image() method. So, we could have something like FluxControlMixin(FluxMixin) and include prepare_image() there. But I felt like it was getting complex so, decided to skip.

---

Additional questions

We have

diffusers/src/diffusers/pipelines/pipeline_utils.py

Line 2175 in 6bf668c

class StableDiffusionMixin:

This PR also has a similar Mixin class but without the methods from the above Mixin. To avoid confusion, I have named it SDMixin.

WDYT about collating the two classes (StableDiffusionMixin and SDMixin (introduced in this PR)) into one?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[DN6]

Looking good 👍🏽 Still on the fence on whether we use multiple levels of inheritance with QwenImageMixin. But other than that this looks good to me.

Perhaps we try with to do the same with Flux and SDXL to see if some common patterns show up for these Mixins.

[sayakpaul]

Looking good 👍🏽 Still on the fence on whether we use multiple levels of inheritance with QwenImageMixin. But other than that this looks good to me.

With you. I am also on the fence. Since the mixins for QwenImage and QwenImageEdit primarily differ in encode_prompt(), I decided to design it like that. We can still rewrite the encode_prompt() method in a single mixin so that it can deal with the images, but it could complicate things. It will also end up changing the signature of the method, leaving room for breaking changes.

But we can definitely uniformize the _get_qwen_prompt_embeds() method as it's private anyway. WDYT?

Perhaps we try with to do the same with Flux and SDXL to see if some common patterns show up for these Mixins.

Yes, I will work on that.

[sayakpaul]

Decided to unify it to further reduce LOC. It's a private method anyway, so, we should be safe.

[sayakpaul]

Only differing encode_prompt() to not introduce any breaking change.

[sayakpaul]

@DN6 pushed updates for Flux. LMK what you think.