Add support for guidance-interval during sampling

[Bili-Sakura]

Issue

Describe the solution you'd like.
Integrate source code from kynkaat/guidance-interval into huggingface/diffusers in callbacks.py.

Additional context.

Although, we have a pr#7761 provide a solution as dynamic-classifier-free-guidance, it only support to disable CFG after a certain number of inference steps. However, in the paper of 'Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models', it is stated that it is best to only do guidance in the middle of denoising process.

Moreover, I want it to be natively support for all diffusion models with tasks including text-to-image, image-to-image, text guided image editing, inpainting, etc.

Reference

@inproceedings{Kynkaanniemi2024,
    author = {Tuomas Kynkäänniemi and
              Miika Aittala and
              Tero Karras and
              Samuli Laine and
              Timo Aila and
              Jaakko Lehtinen},
    title = {Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models},
    booktitle = {Proc. NeurIPS},
    year = {2024},
}

[Bili-Sakura]

#7657 #7761 @asomoza Take a look. Great thanks.

[ukaprch]

I would like to point out that what you have above seems to be directed at SDXL models. With the newer DIT models like Flux, WAN, etc. this may not be the case. If I understand this concept correctly, the original author is using the sigma values as his noise variables for determining which steps to include / exclude in the process. For DIT we see that the noise variables (sigmas) are going to be quite different with a range of 1.0 ... .00??. Based on this (as an example) I ran WAN 2.1 using a low / high interval of [0,27] for 40 steps. I did get a slightly better image result. The same was true for WAN 2.2. Again, it was only a marginally better result.

[asomoza]

thanks both, I think the general technique of the paper can be applied to all models but I think this is less relevant for video models. I've seen that using CFG strategies with newer Flow Match models also makes the generation better and even if this is not the case, where the quality is the same, when not using CFG you are making the inference faster.

With general technique I mean to ability to use or not CFG at certain intervals, but IMO using this for specific timesteps would be too much and no one will ever use it because it would be too complex to use.

Back to the issue, we have this ability now out of the box with the guiders in modular diffusers which seems to be the ideal use case for this, but if you want to achieve the same with the normal pipelines, then you can create a custom callback for this, you will need to get the prompt embeddings first though, so you can just enable or disable the CFG in the steps you want, the official callback is just a premade callback to help users achieve this faster.

[Bili-Sakura]

Thanks a lot for both. I appreciate your explanations and I will close this issue.