Skip to content

Conversation

@sayakpaul
Copy link
Member

@sayakpaul sayakpaul commented Jan 14, 2025

What does this PR do?

Similar to how it's done in Flux, this PR adds support for doing CFG in HunyuanVideo.

Here are results: https://wandb.ai/sayakpaul/hunyuanvideo_cfg/runs/41dhd69p

Code
import torch
from diffusers import HunyuanVideoPipeline, HunyuanVideoTransformer3DModel
from diffusers.utils import export_to_video
import argparse

prompt = "A cat walks on the grass, realistic"
negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"

def load_pipeline():
    model_id = "hunyuanvideo-community/HunyuanVideo"
    transformer = HunyuanVideoTransformer3DModel.from_pretrained(
        model_id, subfolder="transformer", torch_dtype=torch.bfloat16
    )
    pipe = HunyuanVideoPipeline.from_pretrained(
        model_id, transformer=transformer, torch_dtype=torch.float16
    ).to("cuda")
    return pipe

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--true_cfg_scale", type=float, default=1.0)
    args = parser.parse_args()

    pipe = load_pipeline()
    output = pipe(
        prompt="A cat walks on the grass, realistic",
        negative_prompt=negative_prompt,
        true_cfg_scale=args.true_cfg_scale,
        height=320,
        width=512,
        num_frames=61,
        generator=torch.manual_seed(0),
    ).frames[0]
    path = f"output_cfg@{args.true_cfg_scale}.mp4"
    export_to_video(output, path, fps=15)

TODOs

  • Add docs
  • Add support for negative embeds in __call__()
  • Add tests

@sayakpaul sayakpaul requested a review from a-r-r-o-w January 14, 2025 05:18
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@sayakpaul
Copy link
Member Author

@a-r-r-o-w a gentle ping.

@a-r-r-o-w
Copy link
Contributor

Hi, thanks for your contribution!

The results seem a little too burnt to me and not very promising. Could we see more examples to determine if there's an improvement to quality considering there's ~2x more time required for generation?

@a-r-r-o-w
Copy link
Contributor

Ohh, the PR is from you as well. I thought you asked me for review on a contributors PR lol

@sayakpaul
Copy link
Member Author

Here are some more results: https://wandb.ai/sayakpaul/hunyuanvideo_cfg/runs/w4xbb90g.

The cat seems to be fishy 🐟

I ran it another subject (rocket): https://wandb.ai/sayakpaul/hunyuanvideo_cfg/runs/kebbcmvi and the benefits seem visually better, especially for this one:

true_cfg_0_d53f2d11117d53cc601c.mp4

Compared to the no CFG case:

true_cfg_0_17717d7d8b10d0ab0944.mp4

WDYT? The effects seem to be dependent on the subjects and the scenes being generated.

@github-actions
Copy link
Contributor

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

@github-actions github-actions bot added the stale Issues that haven't received updates label Feb 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stale Issues that haven't received updates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants