How to predict the amount of memory (VRAM) consumed by forward pass?

[marloquemegusta]

Hi!
I understand that performing the forward pass on a network during training consumes GPU memory as it needs to store the activations of the neurons. If I am not wrong, memory consumption should be proportional to the batch size, as it needs to store the activations for each data point.
I am trying to estimate the memory consumption vs batch size, and while measuring it, I see that doubling the batch size does not make the memory consumption double.
Is there something I am missing?
Thanks in advance, you all make a great community!

[rohitgr7]

I think that's not the case always and it depends upon the model and computation done during forward pass. For eg., if you have batch norm layers then I believe they have some states too which don't scale with batch_size so even if you change the batch_size this constant factor will still be there.