What's the improvement in model

[Hickey8]

Hi! Thanks for your work.
I’ve read your paper and I'm a bit confused about the claimed contribution of relying solely on the self-attention mechanism. From my understanding, this architecture appears just the backbone used in CogVideoX. Additionally, using a VAE and concatenating image latents seems to be common practice.

I might be missing something, but could you clarify what the main improvements or novelties are compared to existing approaches like CogVideoX?

Thanks in advance!

[yongzhong-ruc]

Relying solely on the 3D self-attention mechanism and VAE, our proposed unified framework can seamlessly adapt to the two mainstream Transformer-based SOTA video architectures — for instance, full-attention-based models like CogVideoX and self/cross-attention-based models like Wan2.1. This also means that we do not need to introduce any additional parameters or modules and modify the base model structure.

Our framework was finalized as early as last November. While there have been some image-domain approaches that concatenate reference image features of VAE and noise latents along the sequence dimension, to the best of our knowledge, ours is the first to apply this technique to video generation. Moreover, we further explore the use of multiple reference images in the concatenation. Unfortunately, a number of concurrent works and some submission papers of NIPS have recently appeared, including several short arXiv preprints seemingly aimed at staking claims.

[yongzhong-ruc]

Furthermore, in our most recent model (https://github.com/ML-GSAI/Concat-ID?tab=readme-ov-file#concat-id-wan-adaln), we introduce an additional AdaLN module atop the existing architecture to more effectively incorporate reference images across different temporal steps.

[Hickey8]

@yongzhong-ruc Got it. Thanks!