Lammps-dp lose efficiency in long runs with multiple nodes

[omaraek]

Hello,
I have noticed that when I perform long MD runs, with DeePMD potentials, after a while the simulations gets slower when I employ more than 1 node.

I have installed DeePMD and Lammp-dp with conda. The details about my environment are the following:

[omar@login02 lmp]$ conda list | grep deepmodeling
lammps-dp 2.0.2 h9954876_2 deepmodeling
libdeepmd 2.0.3 0_cudaNone_cpu deepmodeling
libtensorflow_cc 2.5.0 cpu_cudaNone_0 deepmodeling
plumed 2.6.2 h5c0ae08_0 deepmodeling

I have trained the model with a hybrid descriptor. So you can reproduce my results using the hybrid example of water.
In particular, after training a model for water with the example given by you, I modified the in.lammps file adding few new lines: run 50000
Then I have performed two runs. The first one with only 1 node while the second one with two nodes.

performance of single node:

[omar@login02 lmp]$ grep -a ns/day lammps_singlenode.out
Performance: 3.365 ns/day, 7.132 hours/ns, 77.898 timesteps/s
Performance: 3.290 ns/day, 7.294 hours/ns, 76.162 timesteps/s
Performance: 3.382 ns/day, 7.096 hours/ns, 78.293 timesteps/s
Performance: 3.383 ns/day, 7.094 hours/ns, 78.310 timesteps/s
Performance: 3.396 ns/day, 7.068 hours/ns, 78.606 timesteps/s
Performance: 3.418 ns/day, 7.023 hours/ns, 79.109 timesteps/s
Performance: 3.374 ns/day, 7.113 hours/ns, 78.100 timesteps/s
Performance: 3.331 ns/day, 7.205 hours/ns, 77.105 timesteps/s
Performance: 3.391 ns/day, 7.078 hours/ns, 78.486 timesteps/s
Performance: 3.399 ns/day, 7.062 hours/ns, 78.672 timesteps/s

performance of two node:

[omar@login02 lmp]$ grep -a ns/day lammps_two_nodes.out
Performance: 4.284 ns/day, 5.602 hours/ns, 99.170 timesteps/s
Performance: 4.158 ns/day, 5.773 hours/ns, 96.239 timesteps/s
Performance: 4.308 ns/day, 5.571 hours/ns, 99.726 timesteps/s
Performance: 4.305 ns/day, 5.575 hours/ns, 99.647 timesteps/s
Performance: 4.325 ns/day, 5.549 hours/ns, 100.125 timesteps/s
Performance: 1.552 ns/day, 15.466 hours/ns, 35.921 timesteps/s
Performance: 1.576 ns/day, 15.230 hours/ns, 36.478 timesteps/s
Performance: 1.580 ns/day, 15.186 hours/ns, 36.582 timesteps/s

As you can see, in the second case after a while the run gets slower. To check whether this efficiency reduction is due to DeePMD or not, I tested the same lammps employed in all the runs by running a simulation of Silicon with Tersoff potential and two nodes. As you can see I get a very stable performance:

[omar@login02 lmp]$ grep -a ns/day Si_two_nodes.out
Performance: 243.427 ns/day, 0.099 hours/ns, 1408.722 timesteps/s
Performance: 261.280 ns/day, 0.092 hours/ns, 1512.035 timesteps/s
Performance: 262.356 ns/day, 0.091 hours/ns, 1518.264 timesteps/s
Performance: 261.883 ns/day, 0.092 hours/ns, 1515.525 timesteps/s
Performance: 263.261 ns/day, 0.091 hours/ns, 1523.504 timesteps/s
Performance: 251.706 ns/day, 0.095 hours/ns, 1456.633 timesteps/s
Performance: 263.584 ns/day, 0.091 hours/ns, 1525.370 timesteps/s
Performance: 262.420 ns/day, 0.091 hours/ns, 1518.637 timesteps/s
Performance: 264.012 ns/day, 0.091 hours/ns, 1527.847 timesteps/s
Performance: 262.589 ns/day, 0.091 hours/ns, 1519.615 timesteps/s
Performance: 263.879 ns/day, 0.091 hours/ns, 1527.078 timesteps/s
Performance: 260.743 ns/day, 0.092 hours/ns, 1508.930 timesteps/s

So I would like to ask if this is a well known limitation or an anomalous behavior?
Thanks in advance and best regrades.
Omar

[njzjz]

Do you make sure there's no other program occupying the CPUs?
The number of threads can be controlled by three environmental variables: OMP_NUM_THREADS, TF_INTRA_OP_PARALLELISM_THREADS and TF_INTER_OP_PARALLELISM_THREADS.

[omaraek]

I submit the jobs on the cluster with slurm using the following command:
sbatch --exclusive job.bash
which grantee that the CPUs are used exclusively to the operations of the job file. In my job file I have only one operation which is the molecular dynamics simulations. So I would say that only lammps is occupying these CPUs.

I know about the variables controlling the threads, however, for a given set of variables I would expect a fixed performance with time, wouldn't you?