how to improve the efficiency of dp model?

[Guancred]

Dear developers,

I have trained a model (dpa1 descriptor) based on the DFT data, but the speed of MD (lammps) is slow (a100, 7000 MD steps need 7 min), and the structure only include 24 atoms, so there are some strategies to improve the speed?

Thanks very much.

[wanghan-iapcm]

1. have you tried model compression? https://docs.deepmodeling.com/projects/deepmd/en/r2/freeze/compress.html
2. there's a limit of strong scaling: i.e. on one GPU, you would not gain any acceleration as the system size decrease if your system size is smaller than ~500 atoms.

[Guancred]

this model is trained with 2 atten layer, so it is uncompressable. I also run MD on CPU with the same strucuture (24 atom), which shows the 32 min for 1000000 steps MD, maybe this speed is also slow?

[wanghan-iapcm]

could you plz provide the input.json for training the model?

[Guancred]

Here is my input,
{
"model": {
"descriptor": {
"type": "se_atten",
"sel": 140,
"rcut_smth": 0.5,
"rcut": 7,
"neuron": [
25,
50,
100
],
"resnet_dt": false,
"axis_neuron": 16,
"attn":128,
"attn_layer":2,
"attn_mask":false,
"attn_dotr":true,
"type_one_side": true,
"seed": 1541804686,
"_activation_function": "tanh"
},
"fitting_net": {
"neuron": [
240,
240,
240
],
"resnet_dt": true,
"seed": 1477278577
},
"type_map": [
"Li",
"P",
"S"
]
},
"learning_rate": {
"type": "exp",
"start_lr": 0.001,
"stop_lr":3.51e-8,
"decay_steps": 40000
},
"loss": {
"start_pref_e": 0.02,
"limit_pref_e": 1,
"start_pref_f": 1000,
"limit_pref_f": 1,
"start_pref_v": 0,
"limit_pref_v": 0
},
"training": {
"seed": 1136108454,
"numb_steps": 8000000,
"_comment": "that's all",
"disp_file": "lcurve.out",
"disp_freq": 2000,
"numb_test": 1,
"save_freq": 2000,
"save_ckpt": "model.ckpt",
"disp_training": true,
"time_training": true,
"profiling": false,
"profiling_file": "timeline.json",
"training_data": {
"systems": [
"data/iter.000014_data.240",
"data/iter.000038_data.247",
"data/iter.000026_data.377",
"data/iter.000050_data.020",
"data/iter.000043_data.281",
"data/iter.000029_data.029",
"data/iter.000016_data.328",
"data/iter.000060_data.074",
"data/iter.000026_data.054",
"data/iter.000005_data.061",
"data/iter.000020_data.223",
"data/iter.000043_data.117",

[njzjz]

Could you use replicate command to see if the same performance applies to more atoms?

[Guancred]

Tue Jun 18 10:49:40 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.23.08 Driver Version: 545.23.08 CUDA Version: 12.3 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA A100-SXM4-40GB On | 00000000:4B:00.0 Off | 0 |
| N/A 47C P0 96W / 400W | 16045MiB / 40960MiB | 98% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 1272575 C ...h-2/.conda/envs/deepmd-gpu/bin/_lmp 988MiB |
| 0 N/A N/A 1272576 C ...h-2/.conda/envs/deepmd-gpu/bin/_lmp 988MiB |
| 0 N/A N/A 1272577 C ...h-2/.conda/envs/deepmd-gpu/bin/_lmp 988MiB |
| 0 N/A N/A 1272578 C ...h-2/.conda/envs/deepmd-gpu/bin/_lmp 988MiB |
| 0 N/A N/A 1272579 C ...h-2/.conda/envs/deepmd-gpu/bin/_lmp 988MiB |
| 0 N/A N/A 1272580 C ...h-2/.conda/envs/deepmd-gpu/bin/_lmp 988MiB |
| 0 N/A N/A 1272581 C ...h-2/.conda/envs/deepmd-gpu/bin/_lmp 988MiB |
| 0 N/A N/A 1272582 C ...h-2/.conda/envs/deepmd-gpu/bin/_lmp 988MiB |
| 0 N/A N/A 1272583 C ...h-2/.conda/envs/deepmd-gpu/bin/_lmp 988MiB |
| 0 N/A N/A 1272584 C ...h-2/.conda/envs/deepmd-gpu/bin/_lmp 988MiB |
| 0 N/A N/A 1272585 C ...h-2/.conda/envs/deepmd-gpu/bin/_lmp 988MiB |
| 0 N/A N/A 1272586 C ...h-2/.conda/envs/deepmd-gpu/bin/_lmp 1116MiB |
| 0 N/A N/A 1272587 C ...h-2/.conda/envs/deepmd-gpu/bin/_lmp 988MiB |
| 0 N/A N/A 1272588 C ...h-2/.conda/envs/deepmd-gpu/bin/_lmp 988MiB |
| 0 N/A N/A 1272589 C ...h-2/.conda/envs/deepmd-gpu/bin/_lmp 988MiB |
| 0 N/A N/A 1272590 C ...h-2/.conda/envs/deepmd-gpu/bin/_lmp 988MiB |
+---------------------------------------------------------------------------------------+

the gpu is operating normally, and in this case, the model is from the https://bohrium.dp.tech/notebooks/71679486918 without any excess training, which shows the 14000 MD steps need 15 mins.

[CaRoLZhangxy]

It seems there are too many lmp processes on your single GPU. I guess the resource contention may be the reason for low efficiency.

[Guancred]

Thanks very much, now i set the #SBATCH --ntasks-per-node=1 in my sbatch file and the updated speed is " 20000 MD steps (5 mins)", and this rate is normal? Following is my sbatch file:

#!/bin/bash
#SBATCH --job-name=gch_lammps
#SBATCH --partition=a100
#SBATCH --nodes=1
#SBATCH --gres=gpu:1
#SBATCH --ntasks-per-node=1
#SBATCH --output=%j.out
#SBATCH --error=%j.err

ulimit -s unlimited
ulimit -l unlimited

module purge
module load oneapi
#export PATH=$PATH:/dssg/home/acct-umjzhh/umjzhh-3/chaohong/Scripts/MTP/interface-lammps-mlip-2
mpirun lmp -i in.zz

[CaRoLZhangxy]

I think this is acceptable.

[njzjz]

Note that if the same performance applies to more atoms, the GPU is not 100% utilized.