Is it possible to upload an audio file (mp3, mp4) directly through code ?

[yousefriven]

Hello,

I want to upload an audio file (mostly through flask fileStorage) to whisper.

How do I go on about that ?

I keep running into errors like :

1. UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 142: invalid start byte

when tried to upload content = request.files['file'].read() to whisper.load_audio(content)

2. TypeError: expected np.ndarray (got bytes)

3. _pickle.UnpicklingError: Failed to interpret file <_io.BytesIO object at 0x000001E896834680> as a pickle

This code only accepts the path for the file rather than the actual file.
result = model.transcribe("audio.mp3")

I am a novice programmer. 😅
This is so hard for me.

thanks.

[besimali]

From what I've seen, the load_audio() function need a file path so using a file is not supported at the moment. It could be possible to pipe the file to stdin if you pass "pipe:" as the input file name.

Another solution would be to use the API webservice I co-authored. You can upload a file directly to the webservice without saving it to the filesystem.

[jianfch]

See comment for the updated answer.

model.transcribe() accepts numpy array

# content should be bytes of ffmpeg output
aud_array = np.frombuffer(content, np.int16).flatten().astype(np.float32) / 32768.0 
model.transcribe(aud_array)

[Shivansh-yadav13]

@jianfch how can we convert video file to numpy array ??

[jianfch]

@Shivansh-yadav13 the aforementioned comment should work for video files too. Setting output format to s16le for ffmpeg will have it use only the audio stream of the file.

[Shivansh-yadav13]

@jianfch yes, I tried it but it gives me this error

I used vn=True

def load_audio(file: (str, bytes), sr: int = 16000):
    """
    Open an audio file and read as mono waveform, resampling as necessary

    Parameters
    ----------
    file: (str, bytes)
        The audio file to open or bytes of audio file

    sr: int
        The sample rate to resample the audio if necessary

    Returns
    -------
    A NumPy array containing the audio waveform, in float32 dtype.
    """

    if isinstance(file, bytes):
        inp = file
        file = 'pipe:'
    else:
        inp = None

    try:
        # This launches a subprocess to decode audio while down-mixing and resampling as necessary.
        # Requires the ffmpeg CLI and `ffmpeg-python` package to be installed.
        out, _ = (
            ffmpeg.input(file, threads=0)
            .output("-", format="s16le", acodec="pcm_s16le", ac=1, ar=sr, vn=True)
            .run(cmd="ffmpeg", capture_stdout=True, capture_stderr=True, input=inp)
        )
    except ffmpeg.Error as e:
        raise RuntimeError(f"Failed to load audio: {e.stderr.decode()}") from e

    return np.frombuffer(out, np.int16).flatten().astype(np.float32) / 32768.0

(check discussion here: #1572 (reply in thread))

INFO:     127.0.0.1:52872 - "POST /tcv HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "N:\audio-to-text\main.py", line 58, in load_audio
    .run(cmd="ffmpeg", capture_stdout=True, capture_stderr=True, input=inp)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "N:\audio-to-text\venv\Lib\site-packages\ffmpeg\_run.py", line 325, in run
    raise Error('ffmpeg', out, err)
ffmpeg._run.Error: ffmpeg error (see stderr output for detail)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "N:\audio-to-text\venv\Lib\site-packages\uvicorn\protocols\http\httptools_impl.py", line 426, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "N:\audio-to-text\venv\Lib\site-packages\uvicorn\middleware\proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "N:\audio-to-text\venv\Lib\site-packages\fastapi\applications.py", line 289, in __call__
    await super().__call__(scope, receive, send)
  File "N:\audio-to-text\venv\Lib\site-packages\starlette\applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "N:\audio-to-text\venv\Lib\site-packages\starlette\middleware\errors.py", line 184, in __call__
    raise exc
  File "N:\audio-to-text\venv\Lib\site-packages\starlette\middleware\errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "N:\audio-to-text\venv\Lib\site-packages\starlette\middleware\cors.py", line 91, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "N:\audio-to-text\venv\Lib\site-packages\starlette\middleware\cors.py", line 146, in simple_response
    await self.app(scope, receive, send)
  File "N:\audio-to-text\venv\Lib\site-packages\starlette\middleware\exceptions.py", line 79, in __call__
    raise exc
  File "N:\audio-to-text\venv\Lib\site-packages\starlette\middleware\exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "N:\audio-to-text\venv\Lib\site-packages\fastapi\middleware\asyncexitstack.py", line 20, in __call__
    raise e
  File "N:\audio-to-text\venv\Lib\site-packages\fastapi\middleware\asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
  File "N:\audio-to-text\venv\Lib\site-packages\starlette\routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "N:\audio-to-text\venv\Lib\site-packages\starlette\routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "N:\audio-to-text\venv\Lib\site-packages\starlette\routing.py", line 66, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "N:\audio-to-text\venv\Lib\site-packages\fastapi\routing.py", line 273, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "N:\audio-to-text\venv\Lib\site-packages\fastapi\routing.py", line 192, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "N:\audio-to-text\venv\Lib\site-packages\starlette\concurrency.py", line 41, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "N:\audio-to-text\venv\Lib\site-packages\anyio\to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "N:\audio-to-text\venv\Lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "N:\audio-to-text\venv\Lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "N:\audio-to-text\main.py", line 71, in tcv
    mel = whisper.log_mel_spectrogram(load_audio(video_file))
                                      ^^^^^^^^^^^^^^^^^^^^^^
  File "N:\audio-to-text\main.py", line 61, in load_audio
    raise RuntimeError(f"Failed to load audio: {e.stderr.decode()}") from e
RuntimeError: Failed to load audio: ffmpeg version 6.0-essentials_build-www.gyan.dev Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 12.2.0 (Rev10, Built by MSYS2 project)
  configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-lzma --enable-zlib --enable-libs
    encoder         : Lavf58.29.100
  Duration: 00:00:48.55, start: 0.000000, bitrate: N/A
  Stream #0:0[0x1](und): Video: h264 (avc1 / 0x31637661), none, 1080x1920, 1891 kb/s, SAR 1:1 DAR 9:16, 30 fps, 30 tbr, 90k tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
  Stream #0:1[0x2](und): Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]
File 'True' already exists. Exiting.