T2V inference getting bad results

[YanzuoLu]

System Info / 系統信息

Diffusers install from source, latest huggingface weight with fixed transformer sample size

Information / 问题信息

• The official example scripts / 官方的示例脚本
• My own modified scripts / 我自己修改的脚本和任务

Reproduction / 复现过程

default inference code in huggingface repo

Expected behavior / 期待表现

clear videos, the following is the bad results I get of 81 frames (8 fps, 4 videos concated)

000.mp4

[nitinmukesh]

Post pip list here.

Also try this and post result (you can change prompt)
python inference\cli_demo.py --prompt "A butterfly flying over flowers." --num_inference_steps 50 --model_path THUDM/CogVideoX1.5-5b --generate_type "t2v"

[YanzuoLu]

Hi @nitinmukesh
I'm sure the current inference code on t2v repo is problematic without specifying sample size (height and width) since this script now takes more than 200s for one step on H100...

16%|███████████████▎ | 8/50 [31:45<2:46:44, 238.21s/it]

[YanzuoLu]

I'm wondering what this fix means actually...
This make the target sample video to be 300 x 8(vae temporal upsample)=2400p

[lbwang2006]

I have met the same problem with diffusers==0.32.0

[YanzuoLu]

Here is the results after setting height and width...

output.mp4

[a-r-r-o-w]

Could you try again with the latest diffusers commit? A config update was made to the T2V transformer yesterday (the update was to ensure both SAT and Diffusers behaviour was completely consistent). Please take a look at huggingface/diffusers#9963 for more details, and apologies for the inconvenience!

[YanzuoLu]

Hi @a-r-r-o-w
It looks to be working well now. Thank you so much.
Don't apologize haha~ I really love to read codes from diffusers team : )
One thing to note is the max_sequence_length, it seems to be maximum at 224 in CogVideoX-1.5, maybe it is helpful to change that default parameters for pipeline and huggingface model repo.
And I see both use_dynamic_cfg and CogVideoXDPMScheduler enabled in the original repo cli demo, I'm wondering are they suggested settings for 1.5 model since this behavior is also different from the huggingface model repo.
I'm now running inference to see how these parameters affect the results...

[a-r-r-o-w]

One thing to note is the max_sequence_length, it seems to be maximum at 224 in CogVideoX-1.5, maybe it is helpful to change that default parameters for pipeline and huggingface model repo.

This was done in SAT codebase to support context-parallel (so that can be distributed evenly across multiple GPUs more easily) based on my discussion with @zRzRzRzRzRzRzR. The correct max sequence length is 226 I think for all the Cog models.

Thanks for understanding and the kind words!

[zigchang]

@YanzuoLu Have your solved this problem?
I have met the same situation, As the project owner@zRzRzRzRzRzRzR mentioned, a few places that need to be revised:
1 Update huggingface's CogVideoX1.5-5B latest commit regarding the T2V model (Yes, have revised the "sample_height" and "sample_width" to 300)
2 Update the source code and install the diffusers library (Done)
3 Since it's A100, you can use pipe.to("cuda"). (Done)

I have done all these three steps, but still have very slow inference speed like followings:

[YanzuoLu]

@zigchang You need to use the api from_pretrained with torch_dtype=torch.bfloat16
otherwise it runs under fp32 which could be slow

[YanzuoLu]

One more thing, you need to set up height and width to 768 and 1360 yourself.

[zigchang]

One more thing, you need to set up height and width to 768 and 1360 yourself.

I have run it in bf16, but I still don't know the meaning of changes in transformer config:

[YanzuoLu]

No, the fix here is about positional embedding.
You still need to set the height and width in pipeline call.