How to accelerate the training process?

[McAlexDudu]

Hi~Since I use dpgen, the training process seems to cost a lot of time. Every 100 steps cost about 40s of four models. I use cpu version and my computer has two sockets, every socket has 22 cores, and every core has two threads. So I use:

export OMP_NUM_THREADS=20
export TF_INTRA_OP_PARALLELISM_THREADS=16
export TF_INTER_OP_PARALLELISM_THREADS=16

It can work in just one dp train, about 3s every 100 steps; but in dpgen it seem not working. So how can I accelerate the dpgen?

[McAlexDudu]

I try to decrease the init_batch_size (to 5, previous is 15) and the training speed up! Any reason?