Kandinsky 5 10 sec (NABLA suport)

[leffff]

This PR adds support for 10 sec Kandinsky 5.0 model herd.

import torch
from diffusers import Kandinsky5T2VPipeline
from diffusers.utils import export_to_video

# Load the pipeline
pipe = Kandinsky5T2VPipeline.from_pretrained(
    "ai-forever/Kandinsky-5.0-T2V-Lite-sft-10s-Diffusers", 
    torch_dtype=torch.bfloat16
)
pipe = pipe.to("cuda")

# Generate video
prompt = [
    "Photorealistic closeup video of two intricately detailed pirate ships locked in a fierce battle, complete with cannon fire and billowing sails, as they sail through the swirling waters of a steaming cup of coffee. The ships are miniature but highly realistic, with wooden textures and flags fluttering in the liquid breeze. Coffee splashes and foam ripple around them as they maneuver through the turbulent surface, dodging each other's attacks. A detailed reflection of the battle appears on the glossy surface of the coffee, adding to the dynamic realism. The camera pans and zooms to capture every dramatic moment of the high-seas clash within this tiny, unexpected world.",
    "Bad quality",
]
negative_prompt = "Static, 2D cartoon, cartoon, 2d animation, paintings, images, worst quality, low quality, ugly, deformed, walking backwards"

pipe.transformer.set_attention_backend("flex")

output = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    height=512,
    width=768,
    num_frames=241,
    num_inference_steps=50,
    guidance_scale=5.0,
    num_videos_per_prompt=1,
    generator=torch.Generator(42)
)

output.12.mp4

[sayakpaul]

Yes, that should cut it!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[leffff]

Okay, this seems to be working:

import torch
from diffusers import Kandinsky5T2VPipeline
from diffusers.utils import export_to_video

# Load the pipeline
pipe = Kandinsky5T2VPipeline.from_pretrained(
    "ai-forever/Kandinsky-5.0-T2V-Lite-sft-10s-Diffusers", 
    torch_dtype=torch.bfloat16
)
pipe = pipe.to("cuda")

pipe.transformer.set_attention_backend("flex")
pipe.transformer.compile(mode="max-autotune-no-cudagraphs", dynamic=True)

# Generate video
prompt = "A cat and a dog baking a cake together in a kitchen."
negative_prompt = "Static, 2D cartoon, cartoon, 2d animation, paintings, images, worst quality, low quality, ugly, deformed, walking backwards"

output = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    height=512,
    width=768,
    num_frames=241,
    num_inference_steps=50,
    guidance_scale=5.0,
).frames[0]

## Save the video
export_to_video(output, "output.mp4", fps=24, quality=9)

from: https://huggingface.co/ai-forever/Kandinsky-5.0-T2V-Lite-sft-10s-Diffusers

[leffff]

@yiyixuxu @sayakpaul
Please review, but I believe, we good. If is ok for this PR - we can close it.
Please check out this PR: #12527 we need to merge it ASAP.
Things left from me: Docs and Tests.

[sayakpaul]

@leffff let's add the tests and docs as well.

[yiyixuxu]

ok, let's just use this PR to add docs and tests?

[leffff]

Okay

[leffff]

Please checkout the docs

[yiyixuxu]

thanks!

[yiyixuxu]

@bot /style

[github-actions]

Style bot fixed some files and pushed the changes.

[leffff]

@yiyixuxu plz check the new docs version!

[yiyixuxu]

looks really good! thanks!

[sayakpaul]

@leffff could you also add kandinsky_v5 to _toctree.yml?

[leffff]

Okay!

[leffff]

@sayakpaul @yiyixuxu done!