GPU memory persistence and process concurrency

[makafre]

Hello everyone !

I am just starting with Whisper, sorry if this seems obvious and basic; was actually wondering how Whisper scales when used with concurrent calls; for instance, if I have a 16GB GPU and I test whisper's large model which uses ~10GB, am I likely to get an Out of memory error if I initiate a second Whisper command, this, if one instance has already been started and is still running?
Does the model persist in memory in between calls? If so, is it done by default?

On the other hand, if I add a second GPU, how does it scale? Is all the compute melted in one pot and becomes transparent to the way the model and Whisper are handled? Or else? How this all work out with Whisper's concurrent calling?

Thanks for any insights !

[phineas-pta]

1 instance per gpu 👌

[makafre]

okay and do you know if the model resides in memory in between executions or if everything is loaded from scratch?

[phineas-pta]

depends on how u use it

[makafre]

Can you expand?

[phineas-pta]

depends on how u write your code to use it persistently or unload-reload every time

[makafre]

Thank you for all of your answers, it was really helpful