New HunyuanVideo-I2V

[a-r-r-o-w]

Checkpoints:

• 33-channel (old): https://huggingface.co/hunyuanvideo-community/HunyuanVideo-I2V-33ch
• 16-channel (new): https://huggingface.co/hunyuanvideo-community/HunyuanVideo-I2V

Code:

import torch
from diffusers import HunyuanVideoImageToVideoPipeline, HunyuanVideoTransformer3DModel
from diffusers.utils import load_image, export_to_video

# Available checkpoints: "hunyuanvideo-community/HunyuanVideo-I2V" and "hunyuanvideo-community/HunyuanVideo-I2V-33ch"
model_id = "hunyuanvideo-community/HunyuanVideo-I2V"
transformer = HunyuanVideoTransformer3DModel.from_pretrained(
    model_id, subfolder="transformer", torch_dtype=torch.bfloat16
)
pipe = HunyuanVideoImageToVideoPipeline.from_pretrained(
    model_id, transformer=transformer, torch_dtype=torch.float16
)
pipe.vae.enable_tiling()
pipe.to("cuda")

prompt = "A man with short gray hair plays a red electric guitar."
image = load_image(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/guitar-man.png"
)

output = pipe(image=image, prompt=prompt).frames[0]
export_to_video(output, "output.mp4", fps=15)

Examples:

16-channel

output-hunyuani2v-3.mp4

33-channel

output-i2v-33ch.mp4

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yiyixuxu]

what's these used for?

[a-r-r-o-w]

In the main transformer, we pass token_replace_emb and first_frame_num_tokens. The original single and double transformer blocks don't use those arguments, so we need to discard them. The args/kwargs is not used, so they are essentially discarded.

The extra arguments are only used by the new token-replace single and double blocks.

We will have to do a lot of if-else in the main transformer model forward otherwise:

[a-r-r-o-w]

Do you prefer something like this?

block_args = [hidden_states, encoder_hidden_states, temb, attention_mask, image_rotary_emb]

if self.config.image_condition_type == "token_replace":
    block_args.extend([token_replace_emb, first_frame_num_tokens])

for block in self.transformer_blocks:
    block_args[0], block_args[1] = block(*block_args)

Then we can remove *args and **kwargs

[yiyixuxu]

ok, sound good! (no need to change)

[yiyixuxu]

what is this argument?

[a-r-r-o-w]

I believe it controls the amount of image embedding sequence tokens seen by the transformer. Previously, we had this defaulted to 2 (in encode_prompt). For the new I2V model, they make this user-configurable and it's different for 33-ch vs 16-ch model

[a-r-r-o-w]

Failing tests seem unrelated

[yiyixuxu]

thanks!

[tin2tin]

Unfortunately, both of these checkpoints, using the demo code, will crash on a RTX 4090:

chan_crash.mp4

[a-r-r-o-w]

There needs to be multiple memory saving optimizations enabled for it to run on 24gb for this one :(

cc @asomoza as he'll be able to help you in this regard with examples. The latest diffusers main provides group offloading (+an option to lower cpu usage) and layerwise casting, which may be helpful for you to run the model without too much additional time overhead: https://huggingface.co/docs/diffusers/main/en/optimization/memory

[eppaneamd]

#12273