OOM, even with 72 GB GPU memory for simulating 5000 atoms

[J-You]

Dear developers,

I have successfully trained a DP model, and its accuracy is high based on the testing. It is able to reproduce the results of an AIMD simulations (200 atoms simulation, 10ps, is ok). However, when I use it for a large-scale MD simulation with 5000 atoms, it throws an OOM error.

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 71730 RUNNING AT g0003
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
srun: error: g0003: task 0: Out Of Memory
slurmstepd: error: Detected 2 oom-kill event(s) in StepId=357787.0 cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
slurmstepd: error: Detected 2 oom-kill event(s) in StepId=357787.batch cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.

It is strange that the error occurs consistently at the 90000th step each time. I am using Tesla V100 GPU with 36GB memory. Previously, I only needed one GPU card (for simulating 10000 atoms), but now even when I use two cards, this error still occurs. I have even tried the latest version of deepmd-kit, as well as the previous version 2.01 (I used before, without this error), but the issue can not be fixed. Can you provide me with some solutions to resolve this problem?

[njzjz]

This error message talks about CPU memory instead of GPU memory.
Check if the program runs on the GPU and CPU memory is enough.

[J-You]

Thanks for your response. I run on one GPU card with memory 32 GB, and 8 cores CPU with memory 36GB. Two GPU cards, 16 CPU cores, still not work. This is my job submit script. Could you please have a look if environment variables are okay? I install DP-kit using offline package. Thanks in advance.

#!/bin/bash
#SBATCH --gpus=1

export TF_INTER_OP_PARALLELISM_THREADS=3
export OMP_NUM_THREADS=2
export TF_INTRA_OP_PARALLELISM_THREADS=2

mpirun -np 8 lmp -in in.cooling

[njzjz]

The program is killed by Slurm, so you have to check your default Slurm memory limitation. It's not related to how much memory the hardware has.