Running DeepMD on HPC with multiple GPU

[namtranlab]

Hi,

I am using an HCP system that has 2 CPUs (2x24 cores) and 4 GPUs. I found that requesting more GPUs does not speed up the training. The MeepMD-kit was installed through Miniconda with the GPU version. What I have done wrong with the running script?

Here are the bash scripts, the logs can be found in the attached files.

#!/bin/bash
#PBS -q gpuvolta
#PBS -l walltime=1:00:00
#PBS -l ncpus=48
#PBS -l ngpus=4
#PBS -l mem=150GB
#PBS -l jobfs=1GB
#PBS -l wd
#PBS -l software=deepmd
#PBS -P bj26
#PBS -j eo
#PBS -e deepmd.err
#PBS -N deepmd-1
#PBS -m abe


# Add environment variables for shared libraries
source /scratch/bj26/nt7934/Softwares/miniconda3/etc/profile.d/conda.sh
conda activate deepmd

export OMP_NUM_THREADS=4
export TF_INTRA_OP_PARALLELISM_THREADS=4
export TF_INTER_OP_PARALLELISM_THREADS=1

dp train input.json

And

#!/bin/bash
#PBS -q gpuvolta
#PBS -l walltime=1:00:00
#PBS -l ncpus=24
#PBS -l ngpus=2
#PBS -l mem=150GB
#PBS -l jobfs=1GB
#PBS -l wd
#PBS -l software=deepmd
#PBS -P bj26
#PBS -j eo
#PBS -e deepmd.err
#PBS -N deepmd-1
#PBS -m abe


# Add environment variables for shared libraries
source /scratch/bj26/nt7934/Softwares/miniconda3/etc/profile.d/conda.sh
conda activate deepmd

export OMP_NUM_THREADS=2
export TF_INTRA_OP_PARALLELISM_THREADS=2
export TF_INTER_OP_PARALLELISM_THREADS=1

dp train input.json

Thank you for your help.
Best regards,

deepmd_2xGPUs.log
deepmd_4GPUs.log

[njzjz]

Only the first card will be used, as usually, GPU utilization is less than 100% (you can see it using nvidia-smi).

[namtranlab]

Thank you for your answer.

[matr1x-1]

I am a bit late to the discussion but does this mean that the number of GPU used depends on how high the GPU utilisation is on a single GPU? What would make "dp train" use more GPUs?

[njzjz]

Only when you explicitly use multiple GPUs. See https://docs.deepmodeling.com/projects/deepmd/en/master/train/parallel-training.html