Optimal use of GPU #46
-
|
After dropping the learning rate from 0.006 to 0.003 because a significant proportion of trainings failed, I now use 4.5 GB of my 8 GB RAM GPU instead of 6.5 GB, and the Windows monitor reports a use at 30%-60°C (instead of 50%-70°C). As I want to go as fast as possible (and heat my office during the night), I'm wondering what parameter I could pitch up to use some of the RAM and processing power I don't use with these new settings? # of workers is stuck to 0 (as more doesn't work under Windows), but I could either use more torch threads (currently 4) or a larger batch size (currently 64)? But is there a sense in using 6 threads or 96 as batch size? Would that make training faster? |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments
-
|
I haven't done rigorous testing on training speeds but I don't think there's much to gain there. Increasing the batch size is never a bad idea, but you will see that computation time/batch goes up even if you have free memory. You will also get faster convergence over iterations, but overall training time stays about the same in my experience. |
Beta Was this translation helpful? Give feedback.
-
|
As a side note: I wonder what ur CPU utilisation is. 100% (for one core) Or less? This gives an indication whether the CPU is an actual bottleneck. |
Beta Was this translation helpful? Give feedback.
I haven't done rigorous testing on training speeds but I don't think there's much to gain there. Increasing the batch size is never a bad idea, but you will see that computation time/batch goes up even if you have free memory. You will also get faster convergence over iterations, but overall training time stays about the same in my experience.