Keep the whisper cached/ loaded into GPU/ RAM memory to save time

[abCods]

Hi,

I am new to the world of the AI and models and this question might sound stupid so apologies for that in advance.

I have noticed that on a GPU enabled system the Whisper performs reasonably well in terms of the actual transcription process. However, most of the time is taken by it in the model initialisation step where it gets loaded into the memory ( in GPU and RAM ). I have heard about the batch processing but that is not suitable to my architecture for now as it requires fundamental changes which is not ideal in the bigger picture.

I was wondering if there is a way we can keep the model cached into the memory in order to save myself from the initialisation cost. To keep the perspective, I am using the model into a JavaScript container using its CLI.

[phineas-pta]

unfortunately CLI is a kind of one-time use, you have to find a way to use python as your backend

[abCods]

@phineas-pta

Since I am not familiar with Python, I would appreciate any advice you might have to offer in the context of memory management ( RAM and CUDA based GPU ) to the first timer.

[phineas-pta]

CLI is one-time use, no caching no persistence, there's no advice i can offer; you may try tinkering with cuda cache, but next time running cli it'll reload the model again, because of the way it's programmed

you have several options:

• learn to use python as backend, and communicate to frontend js via api, you can inspire from Upload audio/video to browser and generate subtitles that display as media is played #1323
• paid for openai api (so no worry about memory gpu)
• if you absolutely stay with js, you can search for onnx or webassembly version, but there's no gpu support for now