FP16 does not decrease much GPU memory #13252
-
|
hi, I just tried mix precision training with precision=16 set in the trainer. I found the training speed does increase by around 30%, but the GPU memory merely decreases. Should it be half of the raw memory? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 2 replies
-
|
Hi @icoz69 AFAIK, the memory usage depends on your model architecture, specifically, the ratio of the model size to the size of activations. This is because, with amp, your model always stays in fp32 while some operations in your model are done in fp16. |
Beta Was this translation helpful? Give feedback.
-
|
which operations specifically are done in fp16? |
Beta Was this translation helpful? Give feedback.
-
|
See https://pytorch.org/docs/stable/amp.html#autocast-op-reference. |
Beta Was this translation helpful? Give feedback.
Hi @icoz69 AFAIK, the memory usage depends on your model architecture, specifically, the ratio of the model size to the size of activations. This is because, with amp, your model always stays in fp32 while some operations in your model are done in fp16.