Memory depends on the data size?

[jtyang-chem]

My system has 7 elements and 663 atom on every frame, and when I put over 500 frames in my "./data", dp would spit the memory exceed error:
2021-07-15 16:52:30.160138: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 21290256000 exceeds 10% of free system memory. 2021-07-15 16:52:37.312288: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 14462800000 exceeds 10% of free system memory. 2021-07-15 16:52:38.078152: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 14462800000 exceeds 10% of free system memory. ^C^C^C2021-07-15 16:53:42.886454: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at concat_op.cc:158 : Resource exhausted: OOM when allocating tensor with shape[9614000,50] and type double on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu Kill
if with greater frames, the last failed msg missed, process killed either.
With frames 100, dp runs well.
Smaller parameters like batch_size, neuron didn't make memory use smaller.
Any hints?

[njzjz]

I don't think it's surprising to see the memory is related to the data size...

[jtyang-chem]

Thanks for the reply, I don't think a ML program have to build all data to be used at the first, and 100 is a small number in MD process, 140GB which the program request, is a very big number for a usual computer memory.
To make the job continue, I have to decrease the memory footprint by dividing my data into parts. It should be better if I keep the data in one system, am I right?