[feature] implement TaylorSeer

[sayakpaul]

Currently, we support PAB and FasterCache. More details: https://huggingface.co/docs/diffusers/main/en/optimization/cache.

It'd be cool to support Accelerating Diffusion Models with TaylorSeers as it's quite neat and significantly better than others. Code is here:
https://github.com/Shenyi-Z/TaylorSeer

Before proceeding with a PR, please propose a design first in this thread. We can take it from there.

[Aznix07]

Hi @sayakpaul! I am lookin forward to it. But I've a few questions to understand the project:

1. Which pipelines should be prioritized? (SDXL, SD 1.5, Flux or all ?)
2. Should this follow the same pattern as PAB/FasterCache ?
3. Roughly how many files/pipelines would need modification? Is this a small, med or large contribution?
4. Also, can TayloSeer work alongside existing optimizations like PAB, FasterCache?

Based on your answers, I'll propose a design. Thanks!

[sayakpaul]

1. Very good question. In general, our methods tend to be very pipeline-agnostic. For example, FasterCache should work for most models. But we can prioritize the pipelines you mentioned (SDXL, Flux, QwenImage, etc.).
2. Yes, following a uniform structure is always helpful.
3. It's a significant contribution. You can get a sense of the needed modifications from this PR: [core] FasterCache #10163.
4. That's an interesting question. I think we can tackle that later. First priority, IMO, should be to have the method natively intrgrated.