out of memory during the midway of training

[robinzyb]

Dear all,
I used deepmd-kit to train a potential without out of memory(OOM) warning at first.
During the midway of training, train.log report the OOM error and quit.
It is weird, since the OOM should be reported before the beginning of training when allocating matrix.

Here is my log.
train.log

[J-You]

I also have a similar problem.

[J-You]

error log is here:

DEEPMD INFO    batch   88000 training time 30.27 s, testing time 0.02 s
2021-09-01 22:17:26.409584: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at save_restore_v2_ops.cc:135 : Resource exhausted: /data/run01/scv0645/deepmd/alfesi/train/model.ckpt-88000.data-00000-of-00001.tempstate2138035191106795435; No space left on device
Traceback (most recent call last):
  File "/data/home/scv0645/run/software/deepmd-kit/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1375, in _do_call
    return fn(*args)
  File "/data/home/scv0645/run/software/deepmd-kit/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1359, in _run_fn
    return self._call_tf_sessionrun(options, feed_dict, fetch_list,
  File "/data/home/scv0645/run/software/deepmd-kit/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1451, in _call_tf_sessionrun
    return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
  (0) Resource exhausted: /data/run01/scv0645/deepmd/alfesi/train/model.ckpt-88000.data-00000-of-00001.tempstate2138035191106795435; No space left on device
         [[{{node save/SaveV2}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted: /data/run01/scv0645/deepmd/alfesi/train/model.ckpt-88000.data-00000-of-00001.tempstate2138035191106795435; No space left on device
         [[{{node save/SaveV2}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

         [[save/SaveV2/_1332]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

[njzjz]

The reason has been given in the message

No space left on device