Skip to content

Conversation

@lmxyy
Copy link
Contributor

@lmxyy lmxyy commented Jan 11, 2025

What does this PR do?

LTX-Video pipeline uses the original torch.randn to create a random tensor. torch.randn requires generator and latents on the same device, while the wrapped function randn_tensor does not have this issue.

Fixes # (issue)
Fix the running issue when the generator and latents on different devices. For example, if you use CPU's random seed to generate a video on CUDA, the original pipeline will raise an error.

Before submitting

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@yiyixuxu @asomoza

`torch.randn` requires `generator` and `latents` on the same device, while the wrapped function `randn_tensor` does not have this issue.
@lmxyy
Copy link
Contributor Author

lmxyy commented Jan 11, 2025

A simple script to reproduce the error:

import torch
from diffusers import LTXPipeline
from diffusers.utils import export_to_video

pipe = LTXPipeline.from_pretrained("a-r-r-o-w/LTX-Video-0.9.1-diffusers", torch_dtype=torch.bfloat16)
pipe.to("cuda")

prompt = (
    "A woman with long brown hair and light skin smiles at another woman with long blonde hair. "
    "The woman with brown hair wears a black jacket and has a small, barely noticeable mole on her right cheek. "
    "The camera angle is a close-up, focused on the woman with brown hair's face. The lighting is warm and natural, "
    "likely from the setting sun, casting a soft glow on the scene. The scene appears to be real-life footage"
)
negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"

video = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    width=768,
    height=512,
    num_frames=161,
    decode_timestep=0.03,
    decode_noise_scale=0.025,
    num_inference_steps=50,
    generator=torch.Generator().manual_seed(0),
).frames[0]

export_to_video(video, "output.mp4", fps=24)

Copy link
Contributor

@a-r-r-o-w a-r-r-o-w left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect, thanks! Sorry for the oversight and will merge once tests pass

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@a-r-r-o-w a-r-r-o-w merged commit 5cda8ea into huggingface:main Jan 12, 2025
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants