Best way to install Deepmd and LAMMPS with custom plug-in for CUDA GPU system?

[kimiaGF]

I am trying to install Deepmd and LAMMPS with options available for running on a GPU node on a HPC. I have gone through the documentation for both "Easy Install" and "Build from source" over and over again and keep getting the same result:

(1) when I "Easy Install" via conda, everything works as intended BUT I need the LAMMPS VORONOI plug-in which cannot be enabled when LAMMPS was compiled with Conda.

(2) I finally got LAMMPS to work after building from source, BUT it is INSANELY slow even on a GPU (10,000 steps in a 96k atom system in FIVE DAYS). I have made sure that tensorflow "sees" the GPU it is running on, but I notice that the Conda-installed tensorflow is optimized with cuDNN, while the built-from-source tensorflow is not. Could this be the problem?

My question is: what is the best way to install Deepmd-kit and LAMMPS optimized to run on GPUs while also enabling custom lammps packages?

[njzjz]

I notice that the Conda-installed tensor flow is optimized with cuDNN, while the built-from-source tensor flow is not.

In this case, you need to export the directory of cuDNN library to LD_LIBRARY_PATH (assume you have downloaded it).

[kimiaGF]

I checked the contents of the LD_LIBRARY_PATH variable and the cuDNN library is included in it, here is the result of echo $LD_LIBRARY_PATH when the installation environment is active:

/blue/subhash/kimia.gh/envs/test/lib/python3.11/site-packages/nvidia/cudnn/lib:/blue/subhash/kimia.gh/envs/test/lib/:/blue/subhash/kimia.gh/envs/test/lib/python3.11/site-packages/nvidia/cudnn/lib:/blue/subhash/kimia.gh/envs/test/lib/:/apps/mpi/gcc/12.2.0/openmpi/4.1.5/lib64/openmpi:/apps/mpi/gcc/12.2.0/openmpi/4.1.5/lib64:/opt/slurm/lib64:/apps/compilers/gcc/12.2/lib64::

but when I run python -c "import tensorflow as tf" I still get the message:

This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.

[njzjz]

TensorFlow always loads the CPU device, regardless the GPU device exists or not.
You can use tf.config.list_physical_devices() to list all devices.

[kimiaGF]

Tensorflow definitely sees the GPU as it's listed as one of the available devices. Despite this my lammps runs are excruciatingly slow. I see on the LAMMPS documentation I should be installing KOKKOS to aid in GPU optimization, could this be the problem?

[njzjz]

The LAMMPS log file should contain information about how much percent of the time the pair section (provided by DeePMD-kit) costs. Other sections are provided by LAMMPS.