Some issues when fine-tuning LoRA

[xubeining]

First of all, thank you very much for open-sourcing such an impressive video generation model and training framework. Your work has had a significant positive impact on the community, and it has enabled researchers like us to explore and experiment much more efficiently.“”

I am currently using the pretrained model cdit_xl_100000.pth.tar and conducting LoRA fine-tuning on our own dataset, which lies outside the distribution of the original training data. Although the fine-tuned model has successfully learned the motion patterns in our videos, the generated frames remain quite blurry and lack visual clarity. I would like to ask whether you have experimented with applying LoRA fine-tuning on out-of-distribution datasets using this pretrained model. In your experience, does LoRA tend to limit the model’s representational capacity and lead to blurry outputs, especially when the data distribution differs significantly from the pretraining distribution? Or is this a common issue when using LoRA under large distribution shifts?

Additionally, our dataset is completely unlabeled. Currently, I am using sparse optical flow to sample trajectories from the videos for training. However, I saw in another issue that you suggested mixing labeled and unlabeled data when training on unlabeled datasets. Since I do not have access to labeled data, I would like to ask whether my current approach is acceptable. If mixing labeled data is indeed required, would it be valid to combine my unlabeled dataset with a labeled dataset such as recon or one of the annotated datasets mentioned in your paper?

Thank you very much for your time, and I sincerely appreciate any guidance or suggestions you can provide.

[amirbar]

Hello and thank you for the kind words!

First, I would recommend to start from the checkpoint that was already trained on Ego4D videos, and therefore might adapt more easily to out of distribution environments: https://huggingface.co/facebook/nwm/resolve/main/cdit_xl_ego4d_200000.pth.tar?download=true

Second, I think the issue is most likely because there are no navigation actions in the target dataset. The first thing I would try would be to label the data with navigation actions by obtaining camera poses by running off the shelf VGGT/MONST3R/etc. Then to derive the navigation actions from the camera poses.

Can you try that first? If it doesn't solve the problem we can chat more.

[xubeining]

Thank you for your reply. I have another question: if my current goal is simply to improve the generalization ability of the diffusion model, i.e., to generate higher-quality future frames on my own dataset, is it possible to fine-tune the model without using labels?

I previously saw in an issue that someone mentioned fine-tuning on an unlabeled dataset without relying on labels, so I would like to ask how feasible this is in practice. In that context, does “not relying on labels” mean that only trajectory labels are absent, or that even pose labels are not used?

More specifically, I plan to use LoRA to fine-tune my model. While studying the Ego4D dataset, I noticed that it contains bicycle scenes. I hope that with this kind of fine-tuning, the model can generate higher-quality videos in bicycle scenarios, especially on my own training set (which is also a bicycle dataset).

[HuanLi0311]

First of all, thank you very much for open-sourcing such an impressive video generation model and training framework. Your work has had a significant positive impact on the community, and it has enabled researchers like us to explore and experiment much more efficiently.“”

I am currently using the pretrained model cdit_xl_100000.pth.tar and conducting LoRA fine-tuning on our own dataset, which lies outside the distribution of the original training data. Although the fine-tuned model has successfully learned the motion patterns in our videos, the generated frames remain quite blurry and lack visual clarity. I would like to ask whether you have experimented with applying LoRA fine-tuning on out-of-distribution datasets using this pretrained model. In your experience, does LoRA tend to limit the model’s representational capacity and lead to blurry outputs, especially when the data distribution differs significantly from the pretraining distribution? Or is this a common issue when using LoRA under large distribution shifts?

Additionally, our dataset is completely unlabeled. Currently, I am using sparse optical flow to sample trajectories from the videos for training. However, I saw in another issue that you suggested mixing labeled and unlabeled data when training on unlabeled datasets. Since I do not have access to labeled data, I would like to ask whether my current approach is acceptable. If mixing labeled data is indeed required, would it be valid to combine my unlabeled dataset with a labeled dataset such as recon or one of the annotated datasets mentioned in your paper?

Thank you very much for your time, and I sincerely appreciate any guidance or suggestions you can provide.

Your idea of LoRA fine-tuning for bicycle scene generalization attracts me. Unsupervised fine-tuning on out-of-distribution data is indeed promising, and i also do something familiar with you. I’d love to hear more about your dataset details and progress. Feel free to reach out anytime. If you are willing, will come to contact me via WeChat: 15801553698