Whether can use 4 GPU and 1/4 stop_batchs to reduce training time and achieve same accuracy.

[roger13231]

Dear Maintainers,
Similar discussion have been mentioned in #3655. I have a further questions.
I want to reduce the training time of DP potential as I have 100+ iterations in DPGEN and mainly training cost most of time. In #3655, Dr. Zeng replied that, double GPU means double epochs, while training time is the same. In this way, if I use stop_batchs of 400k and 1GPU, will it be the same "accuracy" as use stop_batchs of 100k and 4 GPUs

[wanghan-iapcm]

You may increase the number of fp data collected in each iteration to reduce the total number of iterations, thus reducing the total training cost.

[roger13231]

Thank you so much for your reply, Dr. Wang.
I will try to double or even triple the number of fp data.

I still have doubt for that question I mentioned. I am wondering whether stop_batchs of 400k and 1GPU be the same or similar with 100k and 4 GPUs.

[wanghan-iapcm]

using 4 GPU effective increase the batch size by 4 times, but it does not always leads to faster decay of the error by 4 times. One may have to test case by case

for dpa-1 and se_a descriptors, smaller batch size (like auto:32) is usually preferred, while for dpa-2 and dpa-3, larger batch size may speedup the training.

[roger13231]

Thank you, Dr. Wang,
I will test my case.