CPU Parallel Training Unusual Performance

[hu-lm]

Hello all.
I am writing this discussion because when I was doing the parallel training, each training step time would be surprisingly longer as I increased the mpi tasks assigned node. But isn't that true that more nodes would increase the training speed so the time each step it takes would be shorter?

Here is the info about the HPC hardware: Intel Xeon CPU E5-2690 v2 @ 3.00GHz; Mem 124GB; 20 cores per node available; 49 nodes available.
DeepMD-Kit version: DeePMD-kit v2.0.4.dev1+g96e623d5; Installed with miniconda3 CPU version.

The dataset is the same, and the batch size is all set to be 1 for testing.
When using four nodes 20cpu each, the time for each training step is around 35s;
When the nodes increase to be 8 with 20cpu each, this time increase to 56s;

I have attached the two log files and input scripts, and I really appreciate your help!

log8nodes-20coreseach-btch1.txt
log4nodes-20coreseach-btch1.txt

HPC_Script.txt
input.txt

[njzjz]

MPI training applies data parallelism. The time for each step will not be reduced but the batch size will be increased.

The performance of multiple nodes may be limited by your network.