I want to know the convergence of the point force model during the training process

[Tomy20Wang]

Wonderful work you guys did.
I am amazed at the generalization ability of the model. And I want to know the convergence of the point force model during the training process, thanks!

[nate-gillman]

Howdy! Thanks for your question. We found good convergence of the point force model after 5000 training steps, that's the checkpoint that we released.

[Tomy20Wang]

And I want to know what is the difference when I set num_processes=1 or 4(in scripts/accelerate/accelerate_config_4_gpu.yaml). I found the training time almost the same(about 30 hours).

[Tomy20Wang]

If I train this model with one GPU(num_processes=1), should I set epochs to 20000(4 * 5000) ?

[nate-gillman]

I'm not sure what a proper training configuration would be if you only wanted to train on a single GPU, you would need to experiment with that on your own. (And please let us know what works so that we can add a sentence to the README advising people in a similar situation...)

One thing I want to note though: the time to complete each training step (which has 2 gradient accumulation steps) should be approximately equal, whether you use 1 or 4 GPUs. But it should take 4x as long to train for a single epoch when you only use 1 gpu, because the effective batch size is divided by 4. Also, right now, the training script is set to 40 epochs, not because the model should actually train for 40 epochs, which would be a lot more than 5k steps; the model only trains for 5k steps, and then we just killed the train job after that is done. We just forgot to re-set that to a smaller value.

[Tomy20Wang]

I want to know how you define the model (at a certain epoch) is good?
I found that the convergence is unstable(the loss is below), did you save all the checkpoints, and inference examples to see which one is the best?

[Tomy20Wang]

And I found that training with 2 gpus, 6000 epochs' performance is better than 10000 epochs' performance

[nate-gillman]

In general, the training loss for the ControlNet architecture doesn't give you a lot of information about whether learning is happening (the original ControlNet paper mentions this). So we measure performance by evaluating a given checkpoint on inference examples.