Pass AudioSegment type form pydub to the model transcribe method

[Hadi2525]

Hello,

I am currently running the whisper model on my machine (Apple M2). Here is a code snippet that I am currently running:

import whisper
from pydub import AudioSegment
import numpy as np
import torch

audio_path = "./my_audio.wav"
audio_array = AudioSegment.from_wav(audio_path).get_array_of_samples()
audio_np_array = np.array(audio_array)
audio_torch_tensor = torch.from_numpy(audio_np_array)

model_type = "base"
model = whisper.load_model(model_type)
result = model.transcribe(audio_path)

The above code works since it's passing a file path from a physical audio. I want to pass a AudioSegment type to the transcribe method.

Based on the information, I can pass in numpy array or torch tensor. But I get the following errors when I pass in the np.array or torch.tensor

Traceback (most recent call last):
  File "~/test.py", line 16, in <module>
    result = model.transcribe(audio_np_array)
  File "~/lib/python3.10/site-packages/whisper/transcribe.py", line 121, in transcribe
    mel = log_mel_spectrogram(audio, padding=N_SAMPLES)
  File "~/lib/python3.10/site-packages/whisper/audio.py", line 148, in log_mel_spectrogram
    stft = torch.stft(audio, N_FFT, HOP_LENGTH, window=window, return_complex=True)
  File "~/lib/python3.10/site-packages/torch/functional.py", line 650, in stft
    return _VF.stft(input, n_fft, hop_length, win_length, window,  # type: ignore[attr-defined]
RuntimeError: stft(torch.ShortTensor[1440400], n_fft=400, hop_length=160, win_length=400, window=torch.FloatTensor{[400]}, normalized=0, onesided=None, return_complex=1) : expected a tensor of floating point or complex values

Can you please help me how I can get an AudioSegment object pass to the transcription method?

[phineas-pta]

#983