Running lammps with GPU

[namtranlab]

Hi Everyone,

I installed the GPU version of Deepmd using conda:
conda create -n deepmd deepmd-kit=*=*gpu libdeepmd=*=*gpu lammps-dp cudatoolkit=11.3 horovod -c https://conda.deepmodeling.org

I wonder if the installed version of lammps is also GPU?

If I would like to run this version of lammps on a HPC node of 48 CPUs and 4 GPUs. What command should I use

mpirun -np 48 lmp -i input.lammps OR mpirun -np 4 lmp -i input.lammps ???

Thank you for your help!

[wanghan-iapcm]

mpirun -np 4 lmp -i input.lammps is the right way

[namtranlab]

Dear Dr. Wang,

Thank you for your response.

I just did some tests and found that the parallel simulation seems not to work for multiple nodes.
I found that by using two nodes (8 GPUs, mpirun -np 8 ...) the wall-time is similar to one node (4 GPUs, mpirun -np 4 ...).

How can I optimize the simulation for multiple nodes???

Also, I found that using one node (4 GPUs) and running with 12 MPI (mpirun -np 12 ...) tasks provide the best performance for some reason.

About our HPC nodes: Each node contains four Nvidia V100 GPUs and two 24-core Intel Xeon Scalable 'Cascade Lake' processors.
About the testing system: Hybrid MD/MC simulation of a system of 93312 atoms. Every 20 MD steps there are 600 MC swaps.

Please check the attachments for the log files.

1 node + 4 mpi tasks
deepmd_1node_4gpus_48cpus.log
deepmd_1node_4gpus_48cpus_script.txt

2 node + 8 mpi tasks
deepmd_2nodes_8gpus_96cpus.log
deepmd_2nodes_8gpus_96cpus_script.txt

1 node + 12 mpi tasks
deepmd_1node_4gpus_48cpus_mpirun12.log
deepmd_1node_4gpus_48cpus_mpirun12_script.txt

Any suggestion is very appreciated

Bests

[wanghan-iapcm]

I only see stderr in your logfiles. Could you please provide stdout? or log.lammps?

[namtranlab]

Dear Dr. Wang,

Please check the attachment for the log.lammps and also input.lammps for the case of 2node, 8GPUs

log.lammps
log.lammps.txt

input.lammps
input.lammps.txt

Bests

[njzjz]

You did not set host on the command line, did you?

[Johnsyisme]

Hi！
I also ran into this kind of problem !!
Node info: 4 GPUs (A100 80G) / 2 Gold 6226R(2*16 cores) / both deepmd and lammps are from conda install (gpu version)
Task info: a compressed model contains 380 ops / 5404 carbon atoms in nvt assemble

I have test lammps with different mpi task (mpirun -np $i lmp -i in.lammps) that $i range from 1 to 24 on 1 node
The 12 mpi tasks one has the best performance

And when the number of mpi task is larger than 12, there will be 2 cards working in 0% utility

Here is the summary of this test
***1 mpi tasks*** Performance: 2.650 ns/day, 9.056 hours/ns, 153.375 timesteps/s ***2 mpi tasks*** Performance: 3.841 ns/day, 6.249 hours/ns, 222.272 timesteps/s ***3 mpi tasks*** Performance: 4.961 ns/day, 4.838 hours/ns, 287.106 timesteps/s ***4 mpi tasks*** Performance: 4.358 ns/day, 5.507 hours/ns, 252.213 timesteps/s ***5 mpi tasks*** Performance: 2.643 ns/day, 9.080 hours/ns, 152.955 timesteps/s ***6 mpi tasks*** Performance: 2.207 ns/day, 10.873 hours/ns, 127.733 timesteps/s ***7 mpi tasks*** Performance: 2.541 ns/day, 9.444 hours/ns, 147.061 timesteps/s ***8 mpi tasks*** Performance: 2.451 ns/day, 9.793 hours/ns, 141.831 timesteps/s ***12 mpi tasks*** Performance: 6.117 ns/day, 3.924 hours/ns, 353.991 timesteps/s ***16 mpi tasks*** Performance: 1.386 ns/day, 17.313 hours/ns, 80.222 timesteps/s ***20 mpi tasks*** Performance: 1.055 ns/day, 22.758 hours/ns, 61.030 timesteps/s ***24 mpi tasks*** Performance: 0.807 ns/day, 29.728 hours/ns, 46.721 timesteps/s

Is there anything i can do to improve the parallel efficiency！
Appreciate！！

Bests

[wanghan-iapcm]

Dear njzjz

I only load the miniconda, deepmd-gpu env then run it with mpirun. Then submit the job normally with qsub + script command.

#!/bin/bash
#PBS -q gpuvolta
#PBS -l walltime=2:00:00
#PBS -l ncpus=96
#PBS -l ngpus=8
#PBS -l mem=700GB
#PBS -l jobfs=10GB
#PBS -l wd
#PBS -l software=deepmd
#PBS -P oc95
#PBS -j eo
#PBS -e deepmd.err
#PBS -N dpmd_0
#PBS -m abe

source /scratch/oc95/nt7934/Softwares/miniconda3/etc/profile.d/conda.sh
conda activate deepmd-gpu

mpirun -n 8 lmp -i input.lammps > lammps.log

I will try to set the host in the mpirun command.

Bests Nam

Could you please check whether the MPI tasks run on the same node or two different nodes.

Usually one provides a host file to force 4 tasks running on node 1 and other 4 running on node 2.

[namtranlab]

Dear Dr. Wang,

I have solved the problem by recompiling the DeepMD and Lammps from the source code.
I use the loaded TensorFlow and horovod from the HPC rather than installing miniconda into my account.
The wall-time is now reduced by half when using two nodes.

Thank you for your support.

Best

[wanghan-iapcm]

Dear Dr. Wang,

I have solved the problem by recompiling the DeepMD and Lammps from the source code. I use the loaded TensorFlow and horovod from the HPC rather than installing miniconda into my account. The wall-time is now reduced by half when using two nodes.

Thank you for your support.

Best

@njzjz do you have any hint why it happens?

[njzjz]

The administrator may configure PBS and MPI to make them work together by passing host. Otherwise, one needs to manually provide hosts.