wan2.2 i2v FirstBlockCache fix

[okaris]

What does this PR do?

https://huggingface.co/posts/a-r-r-o-w/278025275110164

currently this cache does not work with the latest diffusers. the reason being is the cache context not being properly initialised in the WanImageToVideoPipeline.

Fixes #12012 (which was prematurely closed)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@yiyixuxu @asomoza @a-r-r-o-w @DN6

[yiyixuxu]

It's not a bug, I think we just hadn't added the support to I2V pipeline yet

cc @a-r-r-o-w, I think it's ok to add here, no? I will merge this PR first #12006 and there will be conflicts though

[a-r-r-o-w]

@yiyixuxu Yeah should be okay to add here after we merge your PR

[okaris]

@yiyixuxu @a-r-r-o-w i rebased my branch onto main so there shouldn't be conflicts anymore, hope it's more helpful. Thanks!

[a-r-r-o-w]

Nice, thanks for looking into this!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[gluttony-10]

@okaris @a-r-r-o-w
I use apply_first_block_cache(pipe.transformer, FirstBlockCacheConfig(threshold=0.2)) on Wan2.2 5B model, it works well.
I try to use apply_first_block_cache(pipe.transformer, FirstBlockCacheConfig(threshold=0.2))
apply_first_block_cache(pipe.transformer_2, FirstBlockCacheConfig(threshold=0.2)) on Wan2.2 A14B model, it works bad.
It can inference, but its speed hasn't increased, and its quality has deteriorated.
How can I fix it?

[okaris]

@gluttony-10 you can check my implementation here: https://github.com/inference-sh/grid/blob/main/video/wan2-2-i2v-a14b/inference.py

overall 0.2 is a bit too much, i find that 0.1 and 0.05 give better results with some time savings. it also depends on the content of the video, if there is a lot of movement the cache might not work a lot or it could deteriorate the results

[gluttony-10]

@okaris
Thanks a lot. It works.
I use pipe.transformer.disable_cache()
pipe.transformer_2.disable_cache()
pipe.transformer.enable_cache(FirstBlockCacheConfig(threshold=0.1))
pipe.transformer_2.enable_cache(FirstBlockCacheConfig(threshold=0.1))

[okaris]

@gluttony-10 we are building this new platform https://inference.sh for local ai generation, it might fit your channels content. lmk if you'd like to try