non-square videos

[MichalGeyer]

Hi!

I would like to run your method on non-square videos. However, it seems simply changing the crop size in the dataset causes dimension errors afterwards. Are non-square videos not supported? are you planning to support this?

Thanks!

[mayuelala]

Thanks for your attention, limited by original stable diffusion, FateZero can not generate non-square videos. This is a valuable question and worth to explore

[ChenyangQiQi]

Sorry that I have not implemented non-square video for my fatezero paper, and the original code only supports 512X512.
I think the simplest way to change is to remove the image cropping operation in dataloader.
https://github.com/ChenyangQiQi/FateZero/blob/9b6cb7b4dea2b2154df71ad7be0f61a66ca46d88/video_diffusion/data/dataset.py#L111-L113

I think there are other points of concern that you may need to consider when testing on non-square videos

1. The overall image quality will degrade because attention and convolution layers are pre-trained on 512x512 squares and are sensitive to the resolution
2. It will take much more memory when tested on larger video (e.g., 1024x1024 ), especially on self-attention maps.
3. Some part of my code write explicit integer that is related to resolution. They will bring unwanted bugs
   a. (e.g., https://github.com/ChenyangQiQi/FateZero/blob/9b6cb7b4dea2b2154df71ad7be0f61a66ca46d88/video_diffusion/prompt_attention/attention_register.py#L112C84-L112C91)