Generated video quality is not up to the mark LTX-Video 0.9.5

[nitinmukesh]

Is it that this model can be used with LTXConditionPipeline only, which mandatorily requires image/video? [The documentation says https://huggingface.co/Lightricks/LTX-Video-0.9.5. LTX Video is compatible with the Diffusers Python library. It supports both text-to-video and image-to-video generation.]

Or It has to do with code or model or am I missing any parameter? Did anyone else tried this model using diffusers.

Using the same prompt and settings, with 0.9.1 the output was good
newgenai79/sd-diffuser-webui#8 (comment)

import torch
from diffusers import LTXPipeline
from diffusers.utils import export_to_video

pipe = LTXPipeline.from_pretrained("Lightricks/LTX-Video-0.9.5", torch_dtype=torch.bfloat16)
pipe.enable_model_cpu_offload()

prompt = "A woman with long brown hair and light skin smiles at another woman with long blonde hair. The woman with brown hair wears a black jacket and has a small, barely noticeable mole on her right cheek. The camera angle is a close-up, focused on the woman with brown hair's face. The lighting is warm and natural, likely from the setting sun, casting a soft glow on the scene. The scene appears to be real-life footage"
negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"

video = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    width=704,
    height=480,
    num_frames=161,
    num_inference_steps=50,
).frames[0]
export_to_video(video, "output.mp4", fps=24)

output.mp4

output1.mp4

[nitinmukesh]

Even tried the image to video

import torch
from diffusers.utils import export_to_video, load_image
from diffusers import LTXImageToVideoPipeline
repo_id = "Lightricks/LTX-Video-0.9.5"
pipe = LTXImageToVideoPipeline.from_pretrained(
    repo_id, 
    torch_dtype=torch.bfloat16
)
pipe.enable_model_cpu_offload()
# pipe.enable_sequential_cpu_offload()
image = load_image("C:\\Users\\nitin\\OneDrive\\Desktop\\img.png")
inference_params = {
    "image": image,
    "prompt": "A beautiful young woman with dark hair, blue eyes and rosy cheeks wearing a cream winter hat with pom pom and white fur-lined jacket, blinking naturally and smiling warmly while snowflakes gently fall around her. Snow-covered trees and soft bokeh lights in the background.",
    "height": 928,
    "width": 640,
    "num_inference_steps": 30,
    "num_frames": 161,
    "generator": torch.Generator(device="cuda").manual_seed(249050527),
}
video = pipe(**inference_params).frames[0]
export_to_video(video, "ltx-image.mp4", fps=25)

ltx-image.mp4

Results with 0.9.1 was good

newgenai79/sd-diffuser-webui#11

[bghira]

can you try the beta schedule for euler, or unipc

[nitinmukesh]

I don't think UniPCMultistepScheduler is supported. Will try Euler

ValueError: The current scheduler class <class 'diffusers.schedulers.scheduling_unipc_multistep.UniPCMultistepScheduler'>'s set_timesteps does not support custom sigmas schedules. Please check whether you are using the correct scheduler.

[yiyixuxu]

hi @nitinmukesh

Yes it is to be used with LTXConditionPipeline only