Keep model in VRAM?

[ScissorSnips]

I'm calling whisper from powershell, with --device cuda
It takes me something like 18 seconds to load the Large model into VRAM
Is there a way the model can stay loaded?
Or is that just how it is, when you call something from the command line/powershell.

Would I be better off writing a python app? Can it load it once then sit in memory until getting called externally to do the transcription?

[mathdons]

In a python app, you have more control since you explicitely load the model. For instance I use a little flask service with a simple API that keeps the model loaded in between calls.

LoadedModel = ""
DeviceMode = ""

@app.route('/loadModel/<requestModel>/<useGpu>')
def loadModel(requestModel, useGpu):
    global LoadedModel
    global Model
    global DeviceMode

    device = 'cuda'
    if useGpu == 'False': device = 'cpu'

    if LoadedModel != requestModel or DeviceMode != device:
        LoadedModel = requestModel
        DeviceMode = device
        Model = whisper.load_model(LoadedModel, DeviceMode)