Distributed Environment Issue in Training

[YiboLi-4110]

Wonderful work you guys did!

But I have a samll question:
In train.py, from 343 to 360, it can be seen that the precision will be selected according to different distributed training modes
In accelerate_config_4_gpu.yaml, it has specially set up DeepSpeed. However, in train_4_gpu.sh, it sets MULTI_GPU through command-line parameters.
So, I would like to know in which distributed environment the trained weights you provided were actually trained?

Because I found that when I used the repository code and made no modifications, the video quality generated by the two models that I trained on 4 A800 was slightly worse than the weights you provided.
Specifically, the model I trained performed somewhat poorly in terms of its ability to distinguish the magnitude of forces and the unity of objects after being subjected to forces.

Do you think this is caused by the randomness of training? Is it still caused by this distributed environment in the code? Or is it a problem in some other aspect?

Looking forward to your reply. Thanks!

[nate-gillman]

Howdy! Thanks for your question.

Based on what you shared, my guess is that the discrepancy is caused by the randomness of training. (We indeed obtained the weights using the code in the repo.)

I'd suggest trying a slightly earlier + slightly later checkpoint, and seeing if one of those does better. Let me know if this solves the problem...

[YiboLi-4110]

It's glad to see your reply!

After some experiments, it is determined that the cause of the discrepancy is the randomness of the training and inference, especially inference.
The code here is fine.

There is just one minor bug in the code.
In train.py, line 355, the weight_dtype = torch.float16 should be weight_dtype = torch.bfloat16.
However, during the training and inference, the code did not enter this branch, so this error has no impact on the current results.

Thank you very much for your suggestion😄