LTXConditionPipeline Doc example results in poor video

[tin2tin]

Describe the bug

LTXConditionPipeline Doc example results in poor video

Reproduction

Running this code from the documentation: https://huggingface.co/docs/diffusers/main/en/api/pipelines/ltx_video

Results in this video:
https://github.com/user-attachments/assets/374b82de-2d4f-4929-89d0-74ea98121bea

Is this really supposed to be the correct output?

BTW. this feature seems like it could need a more extensive documentation (including the text-prompt-only feature)?

import torch
from diffusers.pipelines.ltx.pipeline_ltx_condition import LTXConditionPipeline, LTXVideoCondition
from diffusers.utils import export_to_video, load_video, load_image

pipe = LTXConditionPipeline.from_pretrained("Lightricks/LTX-Video-0.9.5", torch_dtype=torch.bfloat16)
pipe.to("cuda")

# Load input image and video
video = load_video(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cosmos/cosmos-video2world-input-vid.mp4"
)
image = load_image(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cosmos/cosmos-video2world-input.jpg"
)

# Create conditioning objects
condition1 = LTXVideoCondition(
    image=image,
    frame_index=0,
)
condition2 = LTXVideoCondition(
    video=video,
    frame_index=80,
)

prompt = "The video depicts a long, straight highway stretching into the distance, flanked by metal guardrails. The road is divided into multiple lanes, with a few vehicles visible in the far distance. The surrounding landscape features dry, grassy fields on one side and rolling hills on the other. The sky is mostly clear with a few scattered clouds, suggesting a bright, sunny day. And then the camera switch to a winding mountain road covered in snow, with a single vehicle traveling along it. The road is flanked by steep, rocky cliffs and sparse vegetation. The landscape is characterized by rugged terrain and a river visible in the distance. The scene captures the solitude and beauty of a winter drive through a mountainous region."
negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"

# Generate video
generator = torch.Generator("cuda").manual_seed(0)
# Text-only conditioning is also supported without the need to pass `conditions`
video = pipe(
    conditions=[condition1, condition2],
    prompt=prompt,
    negative_prompt=negative_prompt,
    width=768,
    height=512,
    num_frames=161,
    num_inference_steps=40,
    generator=generator,
).frames[0]

export_to_video(video, "output.mp4", fps=24)

Logs

Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████| 4/4 [00:13<00:00,  3.43s/it]
Loading pipeline components...: 100%|████████████████████████████████████████████████████| 5/5 [00:14<00:00,  2.81s/it]
Token indices sequence length is longer than the specified maximum sequence length for this model (166 > 128). Running this sequence through the model will result in indexing errors

System Info

Diffusers 0.33.0

Who can help?

No response

[a-r-r-o-w]

Thanks for pointing that out @tin2tin. The conditions here are weirdly used, and so this output is expected. The first frame image-conditioning here is completely unrelated to the second video-conditioning. It's effectively trying to interpolate between two very different things. Also, the prompt is relevant only to the first condition.

The example needs to be improved here, but if you were to try with something sensible, it should work as expected. I'll open a PR to improve this

[tin2tin]

I tried with two images (generated by Gemini), and the result was not super great either, but if interpolation is the main thing this offers, I guess you could say it is doing that.

lady_output.mp4

import torch
from diffusers.pipelines.ltx.pipeline_ltx_condition import LTXConditionPipeline, LTXVideoCondition
from diffusers.utils import export_to_video, load_video, load_image

from diffusers import LTXVideoTransformer3DModel
from diffusers import AutoencoderKLLTXVideo

single_file_url = "https://huggingface.co/Lightricks/LTX-Video/blob/main/ltx-video-2b-v0.9.5.safetensors"

transformer = LTXVideoTransformer3DModel.from_single_file(
  single_file_url, torch_dtype=torch.bfloat16
)
vae = AutoencoderKLLTXVideo.from_single_file(single_file_url, torch_dtype=torch.bfloat16)

pipe = LTXConditionPipeline.from_pretrained("Lightricks/LTX-Video-0.9.5", transformer=transformer, vae=vae, torch_dtype=torch.bfloat16)
pipe.to("cuda")

image1 = load_image(
    "1_image.jpg"
)
image2 = load_image(
    "2_image.jpg"
)

# Create conditioning objects
condition1 = LTXVideoCondition(
    image=image1,
    frame_index=0,
)
condition2 = LTXVideoCondition(
    image=image2,
    frame_index=80,
)

prompt = "slow-motion. A lady is walking through the Scandinavian inspired apartment"
negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"

# Generate video
generator = torch.Generator("cuda").manual_seed(0)
# Text-only conditioning is also supported without the need to pass `conditions`
video = pipe(
    conditions=[condition1, condition2], 
    prompt=prompt,
    negative_prompt=negative_prompt,
    width=768,
    height=512,
    num_frames=166,
    num_inference_steps=40,
    generator=generator,
).frames[0]

export_to_video(video, "lady_output.mp4", fps=24)

[asomoza]

you're probably expecting too much of the model, maybe you can test it with some known results, for example this article has an example of the frame interpolation where a car transforms into a cake.

Also they mention something relevant to your video of the woman:

It is too much to ask the LTXV 0.9.5 model to generate motions like a person putting on a jacket or videos with large body part movement. You will be disappointed!

you're trying a lot more complex motion than putting a jacket so I guess your result is expected.

[tin2tin]

@asomoza Okay, thank you. All good then. Closing.