mpirun for executing neb command

[WJiangH]

Dear community,

After trained a deep learning potential, I tried to use it in LAMMPS but encountered an issue of executing the "neb" command in LAMMPS.

With activate deepmd env, I use the command:
mpirun -np 5 lmp -partition 5x1 -in in.neb.sivac
Here, in.neb.sivac is sourced from the example folder in LAMMPS.

The error I received is:
ERROR: Processor partitions do not match number of allocated processors (src/lammps.cpp:451)
This error suggests that it's running in "serial" mode.

The deepmd-kit package I installed using the "easy install" mode with conda:

conda create -n deepmd deepmd-kit=*=gpu libdeepmd==*gpu lammps cudatoolkit=11.6 horovod -c https://conda.deepmodeling.com -c defaults

Would you have any insights or suggestions to resolve this issue?

Thank you for your assistance.

[njzjz]

Please print package information, MPI information, and LAMMPS information using

conda list
mpirun --version
lmp -h

[WJiangH]

Here is my conda list

_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
abseil-cpp                20211102.0           hd4dd3e8_0  
absl-py                   1.0.0              pyh7a5ff2f_0    https://conda.deepmodeling.com
aiohttp                   3.8.5           py310h5eee18b_0  
aiosignal                 1.2.0              pyhd3eb1b0_0  
astunparse                1.6.3                      py_0  
async-timeout             4.0.2           py310h06a4308_0  
attrs                     22.1.0          py310h06a4308_0  
backrefs                  5.0.1              pyhd8ed1ab_0    https://conda.deepmodeling.com
blas                      1.0                         mkl  
blinker                   1.4             py310h06a4308_0  
bracex                    2.1.1              pyhd8ed1ab_0    https://conda.deepmodeling.com
brotlipy                  0.7.0           py310h7f8727e_1002  
bzip2                     1.0.8                h7b6447c_0  
c-ares                    1.19.1               h5eee18b_0  
ca-certificates           2023.08.22           h06a4308_0  
cachetools                4.2.2              pyhd3eb1b0_0  
certifi                   2023.7.22       py310h06a4308_0  
cffi                      1.15.1          py310h5eee18b_3  
charset-normalizer        2.0.4              pyhd3eb1b0_0  
click                     8.0.4           py310h06a4308_0  
cloudpickle               2.2.1           py310h06a4308_0  
cryptography              41.0.3          py310h130f0dd_0  
cudatoolkit               11.6.0              hd754f66_10    https://conda.deepmodeling.com
cudnn                     8.4.0.27             h5453bde_0    https://conda.deepmodeling.com
dargs                     0.3.5              pyh7a5ff2f_0    https://conda.deepmodeling.com
deepmd-kit                2.2.4           py310_0_cuda11.6_gpu    https://conda.deepmodeling.com
dpdata                    0.2.16                   pypi_0    pypi
fftw                      3.3.9                h27cfd23_1  
frozenlist                1.3.3           py310h5eee18b_0  
gast                      0.4.0              pyhd3eb1b0_0  
giflib                    5.2.1                h5eee18b_3  
google-auth               2.22.0          py310h06a4308_0  
google-auth-oauthlib      0.4.4              pyhd3eb1b0_0  
google-pasta              0.2.0              pyhd3eb1b0_0  
grpc-cpp                  1.48.2               h5bf31a4_0  
grpcio                    1.48.2          py310h5bf31a4_0  
gsl                       2.7.1                h6e86dc7_1  
h5py                      3.7.0           py310he06866b_0  
hdf5                      1.10.6               hb1b8bf9_0  
horovod                   0.27.0          py310h50483eb_0    https://conda.deepmodeling.com
icu                       68.1                 h2531618_0  
idna                      3.4             py310h06a4308_0  
intel-openmp              2021.4.0          h06a4308_3561  
jpeg                      9e                   h5eee18b_1  
jsoncpp                   1.9.4                hdb19cb5_3  
keras                     2.9.0              pyh7a5ff2f_0    https://conda.deepmodeling.com
keras-preprocessing       1.1.2              pyhd3eb1b0_0  
krb5                      1.20.1               h568e23c_1  
lammps                    20230802.0           hf349665_1    https://conda.deepmodeling.com
ld_impl_linux-64          2.38                 h1181459_1  
libcurl                   7.88.1               h91b91d3_2  
libdeepmd                 2.2.4            1_cuda11.6_gpu    https://conda.deepmodeling.com
libedit                   3.1.20221030         h5eee18b_0  
libev                     4.33                 h7f8727e_1  
libffi                    3.4.4                h6a678d5_0  
libgcc-ng                 11.2.0               h1234567_1  
libgfortran-ng            7.5.0               ha8ba4b0_17  
libgfortran4              7.5.0               ha8ba4b0_17  
libgomp                   11.2.0               h1234567_1  
libnghttp2                1.52.0               ha637b67_1  
libopenblas               0.3.18               hf726d26_0  
libpng                    1.6.39               h5eee18b_0  
libprotobuf               3.20.3               he621ea3_0  
libssh2                   1.10.0               h37d81fd_2  
libstdcxx-ng              11.2.0               h1234567_1  
libtensorflow_cc          2.9.0           cuda116h4bf587c_0    https://conda.deepmodeling.com
libuuid                   1.41.5               h5eee18b_0  
markdown                  3.4.1           py310h06a4308_0  
markupsafe                2.1.1           py310h7f8727e_0  
mkl                       2021.4.0           h06a4308_640  
mkl-service               2.4.0           py310h7f8727e_0  
mkl_fft                   1.3.1           py310hd6ae3a3_0  
mkl_random                1.2.2           py310h00e6091_0  
monty                     2023.9.5                 pypi_0    pypi
mpi                       1.0                       mpich  
mpi4py                    3.1.3           py310h21fafdc_1    https://conda.deepmodeling.com
mpich                     3.3.2                hc856adb_0  
multidict                 6.0.2           py310h5eee18b_0  
nccl                      2.18.3.1             hbf7919f_0    https://conda.deepmodeling.com
ncurses                   6.4                  h6a678d5_0  
numpy                     1.22.3          py310hfa59a62_0  
numpy-base                1.22.3          py310h9585f30_0  
oauthlib                  3.2.0              pyh7a5ff2f_0    https://conda.deepmodeling.com
openssl                   1.1.1w               h7f8727e_0  
opt_einsum                3.3.0              pyhd3eb1b0_1  
packaging                 23.1            py310h06a4308_0  
pip                       23.2.1          py310h06a4308_0  
plumed                    2.8.0                h5980d18_1    https://conda.deepmodeling.com
protobuf                  3.20.3          py310h6a678d5_0  
psutil                    5.9.0           py310h5eee18b_0  
pyasn1                    0.4.8              pyhd3eb1b0_0  
pyasn1-modules            0.2.8                      py_0  
pycparser                 2.21               pyhd3eb1b0_0  
pyjwt                     2.4.0              pyh7a5ff2f_0    https://conda.deepmodeling.com
pylammps                  20230802.0         pyh29e1a88_1    https://conda.deepmodeling.com
pyopenssl                 23.2.0          py310h06a4308_0  
pysocks                   1.7.1           py310h06a4308_0  
python                    3.10.13              h7a1cb2a_0  
python-flatbuffers        2.0                pyhd3eb1b0_0  
python-hostlist           1.21               pyh9f0ad1d_0    https://conda.deepmodeling.com
pyyaml                    6.0             py310h5eee18b_1  
re2                       2022.04.01           h295c915_0  
readline                  8.2                  h5eee18b_0  
requests                  2.31.0          py310h06a4308_0  
requests-oauthlib         1.3.0                      py_0  
rsa                       4.7.2              pyhd3eb1b0_1  
scipy                     1.7.3           py310hfa59a62_0  
setuptools                68.0.0          py310h06a4308_0  
six                       1.16.0             pyhd3eb1b0_1  
snappy                    1.1.9                h295c915_0  
sqlite                    3.41.2               h5eee18b_0  
tensorboard               2.9.0              pyh7a5ff2f_0    https://conda.deepmodeling.com
tensorboard-data-server   0.6.1           py310h52d8a92_0  
tensorboard-plugin-wit    1.8.1           py310h06a4308_0  
tensorflow                2.9.0           cuda116py310h5dcf97d_0    https://conda.deepmodeling.com
tensorflow-estimator      2.9.0           cuda116py310h81bfea3_0    https://conda.deepmodeling.com
termcolor                 2.1.0           py310h06a4308_0  
tk                        8.6.12               h1ccaba5_0  
typing_extensions         4.7.1           py310h06a4308_0  
tzdata                    2023c                h04d1e81_0  
urllib3                   1.26.16         py310h06a4308_0  
wcmatch                   8.2                pyhd8ed1ab_0    https://conda.deepmodeling.com
werkzeug                  2.2.3           py310h06a4308_0  
wheel                     0.38.4          py310h06a4308_0  
wrapt                     1.14.1          py310h5eee18b_0  
xz                        5.4.2                h5eee18b_0  
yaml                      0.2.5                h7b6447c_0  
yarl                      1.8.1           py310h5eee18b_0  
zlib                      1.2.13               h5eee18b_0

mpirun --version:
mpirun (Open MPI) 4.1.4
and lmp -h:

Large-scale Atomic/Molecular Massively Parallel Simulator - 2 Aug 2023

Usage example: /home/wenjiang0716/anaconda3/envs/deepmd/bin/_lmp -var t 300 -echo screen -in in.alloy

List of command line options supported by this LAMMPS executable:

-echo none/screen/log/both  : echoing of input script (-e)
-help                       : print this help message (-h)
-in none/filename           : read input from file or stdin (default) (-i)
-kokkos on/off ...          : turn KOKKOS mode on or off (-k)
-log none/filename          : where to send log output (-l)
-mdi '<mdi flags>'          : pass flags to the MolSSI Driver Interface
-mpicolor color             : which exe in a multi-exe mpirun cmd (-m)
-cite                       : select citation reminder style (-c)
-nocite                     : disable citation reminder (-nc)
-nonbuf                     : disable screen/logfile buffering (-nb)
-package style ...          : invoke package command (-pk)
-partition size1 size2 ...  : assign partition sizes (-p)
-plog basename              : basename for partition logs (-pl)
-pscreen basename           : basename for partition screens (-ps)
-restart2data rfile dfile ... : convert restart to data file (-r2data)
-restart2dump rfile dgroup dstyle dfile ... 
                            : convert restart to dump file (-r2dump)
-reorder topology-specs     : processor reordering (-r)
-screen none/filename       : where to send screen output (-sc)
-skiprun                    : skip loops in run and minimize (-sr)
-suffix gpu/intel/opt/omp   : style suffix to apply (-sf)
-var varname value          : set index style variable (-v)

OS: Linux "Red Hat Enterprise Linux" 3.10.0-1160.90.1.el7.x86_64 x86_64

Compiler: GNU C++ 7.5.0 with OpenMP 4.5
C++ standard: C++11
MPI v3.1: MPICH Version:	3.3.2
MPICH Release date:	Tue Nov 12 21:23:16 CST 2019
MPICH ABI:	13:8:1

Thanks

[njzjz]

MPICH and Open MPI are not ABI compatible. LAMMPS was compiled with MPICH, so Open MPI should not be used.
MPICH has been listed in the output of conda list, so you may need to activate the environment?

[WJiangH]

Thank you for providing the solution. I've successfully run the "neb" command.

However, I've noticed that the computation speed is considerably slower compared to the standard LAMMPS version I've previously used. Additionally, I encountered several warnings:

DeePMD-kit WARNING: The environmental variable TF_INTRA_OP_PARALLELISM_THREADS is not set. For optimal performance, please adjust the TF_INTRA_OP_PARALLELISM_THREADS value. More details can be found at https://deepmd.rtfd.io/parallelism/.
DeePMD-kit WARNING: The environmental variable OMP_NUM_THREADS is not set. For optimal performance, please adjust the OMP_NUM_THREADS value. More details can be found at https://deepmd.rtfd.io/parallelism/.
DeePMD-kit WARNING: The environmental variable TF_INTRA_OP_PARALLELISM_THREADS is not set. For optimal performance, please adjust the TF_INTRA_OP_PARALLELISM_THREADS value. More details can be found at https://deepmd.rtfd.io/parallelism/.
INVALID_ARGUMENT: The tensor spin_attr/ntypes_spin:0, specified in either feed_devices or fetch_devices, was not found in the Graph.

I apologize for any inconvenience, as I'm not well-versed in multi-process operations. For your reference, I'm currently utilizing only the CPUs.

Thank you again!

[njzjz]

As shown in the message, refer to https://deepmd.rtfd.io/parallelism/ for more information.

[White-Charles]

Base on these discussions, I finally understand how to deal this issue.

conda activate your_env # activate your environment
conda install mpich

add path in ~/.bashec
delete the old mpi path_

export CONDA_ENV_PATH="your_env_path"
export PATH="${CONDA_ENV_PATH}/bin:$PATH"         
export LD_LIBRARY_PATH="${CONDA_ENV_PATH}/lib:$LD_LIBRARY_PATH" 

activate modification and testing

source ~/.bashrc
which mpirun  #  your_env_path/bin/mpirun
mpirun --version