Questions about the parallel mpirun of lammps using a conda easy installation

[ligangchang]

Hello

I hope you are doing well. I have installed the DeepMD kit (DeePMD-kit v2.2.4) following the easy installation procedure with conda environment. The lammps is running with no issue in a single core but when I try to run lammps parallel, it works wrong. I am using the following command to run lammps parallel: mpirun -np 2 lmp -in in.lammps. The following is the error logs:
#########################################
(deepmd) [ligch@login-1-1 02.lmp]$ mpirun -np 2 lmp -i in.lammps
LAMMPS (2 Aug 2023)
OMP_NUM_THREADS environment is not set. Defaulting to 1 thread. (src/comm.cpp:98)
using 1 OpenMP thread(s) per MPI task
Loaded 1 plugins from /public/data/zhu_wp/ligch/software/miniconda3/envs/deepmd/lib/deepmd_lmp
Reading data file ...
triclinic box = (0 0 0) to (10.114259 10.263124 10.216793) with tilt (0.036749877 0.13833062 -0.056322169)
1 by 2 by 1 MPI processor grid
reading atoms ...
5 atoms
read_data CPU = 0.002 seconds
DeePMD-kit WARNING: Environmental variable TF_INTRA_OP_PARALLELISM_THREADS is not set. Tune TF_INTRA_OP_PARALLELISM_THREADS for the best performance. See https://deepmd.rtfd.io/parallelism/ for more information.
DeePMD-kit WARNING: Environmental variable TF_INTER_OP_PARALLELISM_THREADS is not set. Tune TF_INTER_OP_PARALLELISM_THREADS for the best performance. See https://deepmd.rtfd.io/parallelism/ for more information.
DeePMD-kit WARNING: Environmental variable OMP_NUM_THREADS is not set. Tune OMP_NUM_THREADS for the best performance. See https://deepmd.rtfd.io/parallelism/ for more information.
Summary of lammps deepmd module ...

Info of deepmd-kit:
installed to: /public/data/zhu_wp/ligch/software/miniconda3/envs/deepmd
source: v2.2.4
source branch: HEAD
source commit: 6cf7544
source commit at: 2023-09-01 07:44:01 +0800
surpport model ver.:1.1
build variant: cpu
build with tf inc: /public/data/zhu_wp/ligch/software/miniconda3/envs/deepmd/include;/public/data/zhu_wp/ligch/software/miniconda3/envs/deepmd/include
build with tf lib: /public/data/zhu_wp/ligch/software/miniconda3/envs/deepmd/lib/libtensorflow_cc.so
set tf intra_op_parallelism_threads: 0
set tf inter_op_parallelism_threads: 0
Info of lammps module:
use deepmd-kit at: /public/data/zhu_wp/ligch/software/miniconda3/envs/deepmdDeePMD-kit WARNING: Environmental variable TF_INTRA_OP_PARALLELISM_THREADS is not set. Tune TF_INTRA_OP_PARALLELISM_THREADS for the best performance. See https://deepmd.rtfd.io/parallelism/ for more information.
DeePMD-kit WARNING: Environmental variable TF_INTER_OP_PARALLELISM_THREADS is not set. Tune TF_INTER_OP_PARALLELISM_THREADS for the best performance. See https://deepmd.rtfd.io/parallelism/ for more information.
DeePMD-kit WARNING: Environmental variable OMP_NUM_THREADS is not set. Tune OMP_NUM_THREADS for the best performance. See https://deepmd.rtfd.io/parallelism/ for more information.
DeePMD-kit WARNING: Environmental variable TF_INTRA_OP_PARALLELISM_THREADS is not set. Tune TF_INTRA_OP_PARALLELISM_THREADS for the best performance. See https://deepmd.rtfd.io/parallelism/ for more information.
DeePMD-kit WARNING: Environmental variable TF_INTER_OP_PARALLELISM_THREADS is not set. Tune TF_INTER_OP_PARALLELISM_THREADS for the best performance. See https://deepmd.rtfd.io/parallelism/ for more information.
DeePMD-kit WARNING: Environmental variable OMP_NUM_THREADS is not set. Tune OMP_NUM_THREADS for the best performance. See https://deepmd.rtfd.io/parallelism/ for more information.
2023-09-27 12:30:26.885517: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-09-27 12:30:26.885667: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-09-27 12:30:26.887482: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
2023-09-27 12:30:26.887685: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
2023-09-27 12:30:26.935860: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled
2023-09-27 12:30:26.947556: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled
INVALID_ARGUMENT: Tensor spin_attr/ntypes_spin:0, specified in either feed_devices or fetch_devices was not found in the Graph
INVALID_ARGUMENT: Tensor spin_attr/ntypes_spin:0, specified in either feed_devices or fetch_devices was not found in the Graph
Info of model(s):
using 1 model(s): 2.pb
rcut in model: 6
ntypes in model: 2

CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE

Your simulation uses code contributions which should be cited:

• USER-DEEPMD package:
  The log file lists these citations in BibTeX format.

CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE

Generated 0 of 0 mixed pair_coeff terms from geometric mixing rule
Neighbor list info ...
update: every = 10 steps, delay = 0 steps, check = no
max neighbors/atom: 2000, page size: 100000
master list distance cutoff = 7
ghost atom cutoff = 7
binsize = 3.5, bins = 3 3 3
1 neighbor lists, perpetual/occasional/extra = 1 0 0
(1) pair deepmd, perpetual
attributes: full, newton on
pair build: full/bin/atomonly
stencil: full/bin/3d
bin: standard
Setting up Verlet run ...
Unit style : metal
Current step : 0
Time step : 0.001
Per MPI rank memory allocation (min/avg/max) = 2.566 | 2.566 | 2.566 Mbytes
Step PotEng KinEng TotEng Temp Press Volume
0 -21.034997 0.025852029 -21.009145 50 -10873.951 1060.5429
100 -28.682094 2.2664998 -26.415594 4383.6015 -3670.4352 1060.5429
/public/data/zhu_wp/ligch/software/miniconda3/envs/deepmd/bin/lmp: line 11: 14503 Floating point exception(core dumped) /public/data/zhu_wp/ligch/software/miniconda3/envs/deepmd/bin/_lmp "$@
#########################################
Thanks in advance for your cooperation.

Best
lgc

[Taixi-Feng]

It seems that TensorFlow did not recognize the GPU device

[njzjz]

The error may be the same as #2668, but I cannot reproduce it. Could you share the input files and the environment?

Export the conda environment:

conda env export > environment.yml

[njzjz]

The bug is fixed in #2923.