Replies: 1 comment 3 replies
-
|
16 GB is probably not enough to run two whisper large models in two separate processes. There are a few implementations that support such as https://github.com/m-bain/whisperX which supports batching using single process and would require less memory. |
Beta Was this translation helpful? Give feedback.
-
|
How much VRAM would be sufficient per GPU to run 2 such instances without issue, using openai/whisper ? |
Beta Was this translation helpful? Give feedback.
-
|
It depends on the batch size (greedy vs beam size and best-of-N parameters), but a 24 GB GPU should be enough for running two instances running greedy decoding and maybe beam size 5. The third-party implementation like the above uses more memory-efficient implementation and should consume much lower memory. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I tried sending the whisper model to any GPU other than GPU0 Using the following code: def load_model(
model: str, in_memory: bool = True, device: str = "cpu", threads: int = 8
):
global whisper_model
if utils.check_gpu():
gpu_idx = 2
device = f"cuda:{gpu_idx}"
logger.info(f"Sending transcription model to GPU {gpu_idx}")
else:
device = "cpu"
logger.info("Sending transcription model to CPU")
torch.set_num_threads(threads)
logger.info(f"Loading transcription model: '{model}'")
whisper_model = whisper.load_model(model, in_memory=in_memory, device=device)def transcribe(
input: str,
language: Optional[str] = None,
model: str = "base",
beam_size: Optional[int] = None,
condition_on_previous_text: bool = True,
word_timestamps: bool = True,
load_model_in_memory: bool = True,
threads: int = 8,
) -> Dict[str, Any]:
"""Transcribe audio file with Whisper
Args:
input (str): input audio file
language (str, optional): language of the audio file, if known. Defaults to None.
model (str, optional): model to use for transcription. Defaults to "base".
word_timestamps (bool, optional): include word timestamps in output. Defaults to False.
"""
logger.info(f"Transcribing {input} with Whisper")
logger.info("Starting transcription")
logger.debug(f"language: '{language}'")
logger.debug(f"word_timestamps: {word_timestamps}")
global whisper_model
if whisper_model is None:
load_model(model, in_memory=load_model_in_memory, threads=threads)
result = whisper_model.transcribe(
input,
language=language,
word_timestamps=word_timestamps,
verbose=False,
beam_size=beam_size,
condition_on_previous_text=condition_on_previous_text,
)
logger.info("Transcription complete")
return resultEven when there are no other jobs running on the machine, I get the same error. Whisper only works on When observing Does the transcribe method load anything else into GPU VRAM? |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
I have 4 A16s available, I am trying to run 2 simultaneous instances of Whisper parallelly, to speed up batch processing a large number of audio files. I am able to run 1 instance of whisper without issue. But when I try to launch another whisper instance, I get the following error:
I am also trying to diarize my audio (using pyannote-audio), which I also run in parallel. And this causes no issues and am able to spin up multiple instances without issue. This uses Torch as well.
So I am not sure what could be causing this issue. Ideally, I would like to run 3 transcription instances on 3 individual A16s. I have around 1000 audio files to transcribe. Each instance of my
23_transcribe.py, will load a file to transcribe from my DB and transcribe it, continuously until there are no audio files left to transcribe. If I am able to run multiple23_transcribe.pyinstances, it will dramatically cut down processing times.This is from
nvidia-smi, WhereGPU0is runningwhisperandGPU1is runningpyannote-audio. When I try sending another whisper toGPU2it crashes with the above error, But I am able to send anotherpyannote-audioinstance to it without issue.What could be limiting the number of parallel Whisper instances? Will I need to do anything more to achieve what I am trying to do?
Beta Was this translation helpful? Give feedback.
All reactions