Is using load_model in_memory supposed to be faster?

[arabcoders]

Hi,

I am noticing something off with my setup, using the python script with the following parameters

 {
  "model": "large",
  "device": "cuda",  
  "kargs": {
    "verbose": false,
    "task": "translate",
    "language": "Japanese",
    "best_of": 5,
    "beam_size": 5,
    "patience": null,
    "length_penalty": null,
    "suppress_tokens": "-1",
    "initial_prompt": null,
    "condition_on_previous_text": true,
    "fp16": true,
    "compression_ratio_threshold": 2.4,
    "logprob_threshold": -1.0,
    "no_speech_threshold": 0.6
    "temperature": [
    0.0,
    0.2,
    0.4,
    0.6000000000000001,
    0.8,
    1.0
  ]
}

loading and transcribing using the following model parameters self.model = whisper.load_model( self.model, self.device, in_memory=True) is slower than having in_memory=False i tested on 45min audio file the results as the following

with in_memory=False
translate: [/files/test_45min.mp4] is complete. Took [0:14:47.214375]

while with in_memory=True
translate: [/files/test_45min.mp4] is complete. Took [0:20:02.661830]

PC in question

Whisper: running inside WSL2
CPU: 12th Gen Intel Core i7-12700K
GPU: GeForce RTX 3090 ZOTAC 
RAM: 32GB DDR4 VENGEANCE RGB PRO
DISK: WD_BLACK 500GB SN750 NVMe Internal Gaming SSD Solid State Drive - Gen3 PCIe, M.2 2280, 3D NAND, Up to 3,430 MB/s - WDS500G3X0C

[vivianamarquez]

I am seeing the same behavior on a g4dn.12xlarge ec2 instance.

[Pareek-Yash]

Did you find any optimization for this?

[Thomcle]

Here is the explanation of the parameter "in_memory" in the github:

in_memory: bool
whether to preload the model weights into host memory

Hope it answer