Docs for HunyuanVideo LoRA?

[tin2tin]

Describe the bug

As it seems like LoRA loading on HunyuanVideo has been implemented, I wonder where I can find the docs on this? Are they missing?

Reproduction

Search for HunyuanVideo and LoRA

Logs

System Info

As it is the online docs...

Who can help?

@stevhliu @sayakpaul

[nitinmukesh]

See if this work. You can remove GGUF if you want.
.

import torch
from diffusers import HunyuanVideoPipeline, HunyuanVideoTransformer3DModel
from diffusers import GGUFQuantizationConfig
from diffusers.utils import export_to_video
repo_id = "hunyuanvideo-community/HunyuanVideo"
# hunyuan-video-t2v-720p-Q3_K_M.gguf
transformer_path = f"https://huggingface.co/city96/HunyuanVideo-gguf/blob/main/hunyuan-video-t2v-720p-Q3_K_S.gguf"

transformer = HunyuanVideoTransformer3DModel.from_single_file(
    transformer_path,
    quantization_config=GGUFQuantizationConfig(compute_dtype=torch.bfloat16),
    torch_dtype=torch.bfloat16,
)

pipe = HunyuanVideoPipeline.from_pretrained(
    repo_id, 
    transformer=transformer,
    torch_dtype=torch.float16
)

# Lora https://civitai.com/models/1064343/hunyuan-video-lora-animeshots
# rename the lora after manual downloading
pipe.load_lora_weights("lora", weight_name="hunyuan-video-lora-animeshots.safetensors", adapter_name="hunyuan-lora")
pipe.set_adapters("hunyuan-lora", 0.8)

pipe.enable_model_cpu_offload()
pipe.vae.enable_tiling()
pipe.vae.enable_slicing()
video = pipe(
	prompt="anime boy with silver hair and blue eyes, wearing a casual hoodie, sitting on a park bench, feeding pigeons with a gentle smile.",
	height=544,
	width=960,
	num_frames=49,
	num_inference_steps=20,
	generator=torch.Generator(device='cuda').manual_seed(0),
).frames[0]
export_to_video(video, "hun.mp4", fps=8)

[tin2tin]

Thank you. Though the implementation is pretty straight forward, it might be too to mention on the Hunyuan model page that it is possible use LoRAs with it.

If this code should be included in the docs, it might be better to use a LoRA which has been trained on copyright-free material, so I've changed it to a Steamboat Willie LoRA:

import torch
from diffusers import HunyuanVideoPipeline, HunyuanVideoTransformer3DModel
from diffusers import GGUFQuantizationConfig
from diffusers.utils import export_to_video
repo_id = "hunyuanvideo-community/HunyuanVideo"
# hunyuan-video-t2v-720p-Q3_K_M.gguf
transformer_path = f"https://huggingface.co/city96/HunyuanVideo-gguf/blob/main/hunyuan-video-t2v-720p-Q3_K_S.gguf"

transformer = HunyuanVideoTransformer3DModel.from_single_file(
    transformer_path,
    quantization_config=GGUFQuantizationConfig(compute_dtype=torch.bfloat16),
    torch_dtype=torch.bfloat16,
)

pipe = HunyuanVideoPipeline.from_pretrained(
    repo_id, 
    transformer=transformer,
    torch_dtype=torch.float16
)

pipe.load_lora_weights("badayvedat/Steamboat-Willie-Hunyuan-Video-LoRA", weight_name="pytorch_lora_weights.safetensors", adapter_name="hunyuan-lora")
pipe.set_adapters("hunyuan-lora", 0.8)

pipe.enable_model_cpu_offload()
pipe.vae.enable_tiling()
pipe.vae.enable_slicing()
video = pipe(
    prompt="A black and white animated scene featuring a mouse, dressed in a sailor outfit. The mouse is standing at the helm of a ship, holding the steering wheel with both hands.",
    height=544,
    width=960,
    num_frames=49,
    num_inference_steps=20,
    generator=torch.Generator(device='cuda').manual_seed(0),
).frames[0]
export_to_video(video, "willie.mp4", fps=8)

[sayakpaul]

I don't think we specifically mention LoRA loading support from any pipeline official page. @stevhliu is this something we should consider improving?

Currently, I think the easiest way to know if a pipeline supports loading LoRA is by checking its parent class includes *_LoraLoaderMixin.

[SahilCarterr]

Using Lora is mentioned in docs and it is same for all pipelines

[tin2tin]

@SahilCarterr There are so many models and so many features, and unless you on a daily basis reads all commits, it is very hard to keep updated on what features are supported on what models rn, like LoRAs, ControlNets, quantizations, sequential_offloading, and sometimes even img2img, img2vid etc. I don't know what the best solution might be. A supported feature matrix? Which will be a lot of work. Maybe there are other more low-hanging-fruits ways to extract and expose info like this?

[stevhliu]

What do you think about having a badge on the pipeline's API doc that indicates it supports LoRA? We can also add more badges to indicate other supported features (for example, in the Gemma doc below, you can easily see it supports FlashAttention/SDPA).

[a-r-r-o-w]

@stevhliu Was not asked, but I think that's a great idea to have badges!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[a-r-r-o-w]

I think the docs surrounding loras and other features have improved over time, and the badges also work now (very cool!), so will mark as closed.

If there's anything specific you want us to document more on, please let us know!