[core] LTX Video

[a-r-r-o-w]

T2V:

import torch
from diffusers import LTXPipeline
from diffusers.utils import export_to_video

pipe = LTXPipeline.from_pretrained("a-r-r-o-w/LTX-Video-diffusers", torch_dtype=torch.bfloat16)
pipe.to("cuda")

prompt = "A woman with long brown hair and light skin smiles at another woman with long blonde hair. The woman with brown hair wears a black jacket and has a small, barely noticeable mole on her right cheek. The camera angle is a close-up, focused on the woman with brown hair's face. The lighting is warm and natural, likely from the setting sun, casting a soft glow on the scene. The scene appears to be real-life footage"
negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"

video = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    width=704,
    height=480,
    num_frames=161,
    num_inference_steps=50,
).frames[0]
export_to_video(video, "output.mp4", fps=24)

I2V:

import torch
from diffusers import LTXImageToVideoPipeline
from diffusers.utils import export_to_video, load_image

pipe = LTXImageToVideoPipeline.from_pretrained("a-r-r-o-w/LTX-Video-diffusers", torch_dtype=torch.bfloat16)
pipe.to("cuda")

image = load_image(
    "https://huggingface.co/datasets/a-r-r-o-w/tiny-meme-dataset-captioned/resolve/main/images/8.png"
)
prompt = "A young girl stands calmly in the foreground, looking directly at the camera, as a house fire rages in the background. Flames engulf the structure, with smoke billowing into the air. Firefighters in protective gear rush to the scene, a fire truck labeled '38' visible behind them. The girl's neutral expression contrasts sharply with the chaos of the fire, creating a poignant and emotionally charged scene."
negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"

video = pipe(
    image=image,
    prompt=prompt,
    negative_prompt=negative_prompt,
    width=704,
    height=480,
    num_frames=161,
    num_inference_steps=50,
).frames[0]
export_to_video(video, "output.mp4", fps=24)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[a-r-r-o-w]

@DN6 Should I follow your approach with Mochi and create a separate attention class for LTX?

[yiyixuxu]

ok but we want to be more careful, ideally, we do that as part of carefully planned-out refactor
but maybe it would be safe to just inherit form Attention for now? e.g. we wrote code like this with the assumption in mind we only have one attention class
https://github.com/huggingface/diffusers/blob/e47cc1fc1a89a5375c322d296cd122fe71ab859f/src/diffusers/pipelines/pag/pag_utils.py#L57C39-L57C48

cc @DN6 here too

[yiyixuxu]

I guess Attention stays here for now

[stevhliu]

Thanks for adding!

[a-r-r-o-w]

Thanks for the reviews! Waiting for confirmation of where we should host weights, so I can update documentation accordingly. Currently they are under my account, but once we move them, we should be good to merge

[Skquark]

I'm wondering if it might be easy to incorporate STG (Spatiotemporal Skip Guidance) into LTX-Video pipeline. The improvements in video quality look significant, examples are impressive. Here's the links STG Project, STGuidance GitHub, ComfyUI-LTXTricks. Looks like it can also apply to Mochi, SVD and Open-Sora. Could be that missing ingredient... Adds params stg_mode, stg_scale, stg_block_idx, do_rescaling & rescaling_scale.

[a-r-r-o-w]

Hey, thanks for the suggestion!

We do plan to incorporate STG and other methods of guidance by isolating it out into its separate component. I do not like the idea of adding more parameters to the pipeline __call__ because it starts to become very confusing and bloated, so the design for integration is still a WIP on my end, but I plan to open a PR in the coming week or whenever I can get it to work with all our pipelines.

[yoavhacohen]

One request about naming, LTX -> LTXV

[tin2tin]

@a-r-r-o-w

Testing, I get this error:
AttributeError: module diffusers has no attribute LTXTransformer3DModel. Did you mean: 'LatteTransformer3DModel'

[a-r-r-o-w]

@tin2tin I think it should be LTXVideoTransformer3DModel

[tin2tin]

@a-r-r-o-w I just used the test code in the first post, which doesn't specify that name. So, I guess something internally is calling a wrongly named operator?

[a-r-r-o-w]

Just to confirm, you have installed diffusers from the main branch, yes?

[a-r-r-o-w]

Also, the config files on the LTX repo were updated recently. Could you ensure you're using the latest commit of those configs so that the right model names are being pointed to?

https://huggingface.co/Lightricks/LTX-Video/commits/main

[tin2tin]

Yes, I'm on the latest main branch, and I downloaded the LTX repo yesterday, and it seems like the name change was 4 days ago.

Error: Python: Traceback (most recent call last):
  File "C:\Users\xxx\Documents\Blender Projekter\LTX_video.blend\Text", line 5, in <module>
  File "C:\Users\xxx\AppData\Roaming\Python\Python311\site-packages\huggingface_hub\utils\_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\xxx\AppData\Roaming\Python\Python311\site-packages\diffusers\pipelines\pipeline_utils.py", line 902, in from_pretrained
    loaded_sub_model = load_sub_model(
                       ^^^^^^^^^^^^^^^
  File "C:\Users\xxx\AppData\Roaming\Python\Python311\site-packages\diffusers\pipelines\pipeline_loading_utils.py", line 635, in load_sub_model
    class_obj, class_candidates = get_class_obj_and_candidates(
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\xxx\AppData\Roaming\Python\Python311\site-packages\diffusers\pipelines\pipeline_loading_utils.py", line 319, in get_class_obj_and_candidates
    class_obj = getattr(library, class_name)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\xxx\AppData\Roaming\Python\Python311\site-packages\diffusers\utils\import_utils.py", line 861, in __getattr__
    raise AttributeError(f"module {self.__name__} has no attribute {name}")
AttributeError: module diffusers has no attribute LTXTransformer3DModel. Did you mean: 'LatteTransformer3DModel'?

Checking dependencies...

[Abhinay1997]

@a-r-r-o-w minor typo here, cond_mask should use mask_shape rather than shape here. https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/ltx/pipeline_ltx_image2video.py#L493