High CPU utilization even though it is using the GPU

[ForrestBakerIV]

Hi,

So I am processing a bunch of little files ( about 3 - 10 seconds each ) on 4 separate threads which pegs the CPU utilization at close to 80 - 100% whereas the GPU is only hitting about 10%.

Any thoughts as to why the CPU is my bottle neck? The audio I am sending over is 16k pcm wave where I am taking the head off.

import numpy as np
import torch
import sys

print("Model Loading", datetime.datetime.now())
torch.cuda.is_available()
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
print(DEVICE)
model = whisper.load_model("tiny", device=DEVICE)

audio = file_data[59:] # Extract the remaining bytes as binary data
data_s16 = np.frombuffer(audio, dtype=np.int16).flatten().astype(np.float32) / 32768.0
text = model.transcribe(data_s16)["text"]

Is there something I am doing that is an obvious additional work load onto the CPU? I know it is running on cuda based on the DEVICE...

Thanks!

[phineas-pta]

change model size ?

[glangford]

Since you have lots of little files, the CPU is probably busy just loading them while the GPU waits to run the tiny model serially (which is quick for each file) then is idle again until the next file is loaded.

See also

• Long transcription not maxing out GPU or CPU #34

[ForrestBakerIV]

Nope the model size did not change the CPUutilization ( if anything it increased it but that is open to debate )

The question I have is what is the CPU doing to load? This is an in memory stream - no IO. It should not need to be transformed in anyway I can come up with that would peg the CPU utilization so high...

The question I have is there anything I can do to 'pre-process' to not have this happen? I am trying to figure out what it is doing in the first place that is pegging it so high!

[p-h-c-s]

@ForrestBakerIV Did you come to any conclusion here? I'm facing a similar problem.