is mps supported on whisper?

[khalid4294]

Hey!
I've tried using Whisper with device=mps with no luck, I ran into different issues and I couldn't find anything helpful online.
When I run any operation on mps, it works fine, but with Whisper I get different errors. Here's my implementation:

model = whisper.load_model("tiny.en", device="mps")
segment = model.transcribe(vid_url, fp16=False)

And I get these different errors:

• resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

• Error: buffer is not large enough. Must be 19200 bytes'

Can you guide on this?

[glangford]

Not an mps user myself, but here is a discussion of a similar unresolved issue over in PyTorch, fyi

• MPSNDArray.mm:782: failed assertion; bufer is not large enough Mac M1 MPS pytorch/pytorch#86152

[glangford]

...also worth following this thread

• Uses MPS (Mac acceleration) by default when available #382 (comment)

[khalid4294]

unfortunately there's no solution to the errors above.

[khalid4294]

thanks for referencing the threads through 🙌

[sanchit-gandhi]

Hey! You should be able to use mps with the Hugging Face implementation:

import torch
from transformers import pipeline
from datasets import load_dataset

pipe = pipeline(
  "automatic-speech-recognition",
  model="openai/whisper-tiny",
  chunk_length_s=30,
  device=torch.device("mps"),
)

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
sample = ds[0]["audio"]

prediction = pipe(sample.copy())["text"]

# we can also return timestamps for the predictions
prediction = pipe(sample, return_timestamps=True)["chunks"]

[jstoone]

@sanchit-gandhi first of all, thank you for the Whisper event earlier, it was amazing!

I'm in the process to port my use of whisper to the Hugging Face implementation, but I currently run on a fork of this repo, which adds callbacks for when segments or "chunks" 0have been completed. Is there a way to do this with the Hugging Face implementation?

Here's a commit that shows how it works:
-> jstoone@a12b852

[sanchit-gandhi]

Hey @jstoon! Very glad to hear you enjoyed the event! If I understand correctly, you want to have the "chunked" output from the Whisper model? You can pass "return_timestamps=True" to the pipeline and get the chunks this way? See https://huggingface.co/openai/whisper-large-v2#long-form-transcription, namely the line:

>>> prediction = pipe(sample, return_timestamps=True)["chunks"]  # <- we get the 'chunks' of each time segment here
[{'text': ' Mr. Quilter is the apostle of the middle classes and we are glad to welcome his gospel.',
  'timestamp': (0.0, 5.44)}]

[Jonathan-Asher]

Is there a way to use the Hugging Face implementation, and get the "segments" as in the whisper python package?

[sanchit-gandhi]

You can do so with the following code-snippet:

from transformers import WhisperProcessor, WhisperForConditionalGeneration
from datasets import load_dataset, Audio

model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny.en", low_cpu_mem_usage=True)
processor = WhisperProcessor.from_pretrained("openai/whisper-tiny.en")

dataset = load_dataset("distil-whisper/librispeech_long", "clean", split="validation")
dataset = dataset.cast_column("audio", Audio(16_000))

sample = next(iter(dataset))
inputs = processor(sample["audio"]["array"], padding=True, truncation=False, return_attention_mask=True, return_tensors="pt")

outputs = model.generate(**inputs, return_segments=True)

print(outputs)