Distributed LAMMPS Simulation on CPU

[npiroozan]

Hello,

I hope you are doing well.

I'm trying to run DeePMD with LAMMPS for a distributed simulation across multiple CPU nodes. When I try to execute, the following error arises:

nodes: 2
ppn: 112
omp: 1
intra: 1
inter: 1
kmp_blk: 1
srun: environment addon enabled
[0] MPI startup(): library kind: release
[0] MPI startup(): File "" not found
[0] MPI startup(): Load tuning file: "/global/panfs01/admin/opt/intel/oneAPI/2023.1.0.46401/mpi/2021.9.0/etc/tuning_spr_shm.dat"
LAMMPS (23 Jun 2022 - Update 1)

/home/anaconda3/envs/deepmd29_py309-Pylmp/bin/lmp: line 11: 149447 Floating point exception(core dumped) /home/anaconda3/envs/deepmd29_py309-Pylmp/bin/_lmp "$@"

I'm not sure what is causing this. Could you please help?

Notes:

1. This error is not observed when using a single node.
2. Everything runs very smoothly on a single node.
3. Training, and the resultant .pb file, was obtained on a single node.
4. Using Tensorflow 2.9 and DeePMD 2.1.5.
5. Built DeePMD with the C++ Tensorflow, and linked to LAMMPS (plugin mode)
6. LAMMPS was built with support for MPI.
7. Within conda environment, downloaded "lammps" from deepmodling

Thank you very much.

[njzjz]

I suggest using gdb to debug it:

gdb lmp

Then type

r -in in.xxx   # here are the LAMMPS arguments

After it crash, type

backtrace

Traceback will be printed.

[npiroozan]

Thank you. I used gdb to diagnose the problem. All that can be determined at this time is that there is an issue with the execution of dpplugin.so.

This problem is not observed on a single node, no matter how many mpi ranks are used. It's only now on 2 nodes that I see this issue.

[npiroozan]

Do you think this is an issue seen through the use of cmake to build lammps? Perhaps it is better to use the make process and avoid having to use dpplugin.so

[npiroozan]

Problem Fixed.

[njzjz]

#2923 should fix the bug.