Estimate Memory Usage (GPU) Roughly

[Chenghao-Wu]

I met problems of out of memory on GPU (11GB) for a system with 880 beads (1 batch of 3750 frames) and with 3 layers DNN (basically using the descriptor setting up of water_se_a.json for testing).
Is there a simple way to estimate the memory usage for the training system?

[njzjz]

deepmd-kit/source/train/env.py

Line 46 in bc01883

return tf.ConfigProto(intra_op_parallelism_threads=intra, inter_op_parallelism_threads=inter)

change to

    return tf.ConfigProto(intra_op_parallelism_threads=intra, inter_op_parallelism_threads=inter, 
            gpu_options=tf.GPUOptions(allow_growth=True)
        ) 

See tensorflow/tensorflow#1578 (comment). Then you will see the real memory usage from nvidia-smi.

[Chenghao-Wu]

deepmd-kit/source/train/env.py

Line 46 in bc01883

return tf.ConfigProto(intra_op_parallelism_threads=intra, inter_op_parallelism_threads=inter)

change to

    return tf.ConfigProto(intra_op_parallelism_threads=intra, inter_op_parallelism_threads=inter, 
            gpu_options=tf.GPUOptions(allow_growth=True)
        ) 

See tensorflow/tensorflow#1578 (comment). Then you will see the real memory usage from nvidia-smi.

Thank you very much for your reply. May I ask that how many particles are normally taken for training (concerning less memory available on GPU)?
This method is really general for building modeling models. One more thing I am curious about: I see the point that for some metal or crystalline systems not a large amount of particles needed for training so it is trainable at present. Is there anyone using this method for building macromolecular systems, or maybe coarse-grained polymer models?