Why mochi diffusers video output is worse than mochi official code?

[foreverpiano]

Describe the bug

The quality of video is worse.

Reproduction

Run the code with official prompt

Logs

No response

System Info

diffusers@main

Who can help?

@a-r-r-o-w @yiyixuxu

[foreverpiano]

@DN6 #10033
Has this been fixed now?

[SahilCarterr]

Official Diffusers Code

mochi.mp4

PR #10033 Close to Merged

mochi_new.mp4

Quality has much improved , now fixed @foreverpiano

[DN6]

Hi @foreverpiano yes that branch should address the quality issues. We're currently looking into whether we can remove the dependency on torch autocast and still match the original repo.

[jmahajan117]

Related issue, but I don't see MochiPipeline on release 0.31. How are you generating these videos?

[hlky]

@jmahajan117 Install from main. We are preparing for 0.32 release in the next weeks.

[foreverpiano]

I think the issue is still not fixed. The Mochi attention does not consider adding masks (which are needed for text padding). So the result is different from the original Mochi.

The encoder query should have an attention mask because it is padded to 256. Padding with zeros is not equivalent to adding an attention mask when considering the softmax operation in attention.

diffusers/src/diffusers/models/attention_processor.py

Line 3928 in 6324340

hidden_states = F.scaled_dot_product_attention(query, key, value, dropout_p=0.0, is_causal=False)

Although this difference might be negligible since attention has some robustness, there are still factual errors compared to attention with cu_seq_len due to issues in the attention implementation.

cc @hlky @DN6 @a-r-r-o-w

[a-r-r-o-w]

@foreverpiano Could you use this branch? https://github.com/huggingface/diffusers/tree/mochi-quality

diffusers/src/diffusers/models/transformers/transformer_mochi.py

Line 278 in 09fe7ec

mask = attention_mask[idx][None, :]

[foreverpiano]

@a-r-r-o-w The implementation appears better since it takes the mask into consideration. I believe this addresses my previous concern. Let me look into this version video output.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[a-r-r-o-w]

Marking as closed since Dhruv's PR was merged and video quality now matches. Thanks for all the help in testing @foreverpiano 🤗