How can we directly use a video (mp4) for transcribe() in code??

[Shivansh-yadav13]

I'm using NextJS to send a video file to my FAST API server,
the server receives it in bytes and I don't want to save the file and then use it to transcribe.

Is there a way to transcribe directly from code without saving file?

@app.post("/url")
def transcribe(video_file: Annotated[bytes, File()]):
    model = whisper.load_model("base")
    # video_array = np.frombuffer(video_file, np.complex64).flatten().astype(np.float32) / 32768.0
    result = model.transcribe(video_file, word_timestamps=True, fp16=False)
    return {"result": result}

[glangford]

Hopefully this discussion here is what you need -

• Using ndarray as input to transcribe method #380 (comment)

and note this comment

• Using ndarray as input to transcribe method #380 (reply in thread)

[Shivansh-yadav13]

@glangford

The solution provided

def load_audio(file: (str, bytes), sr: int = 16000):
    """
    Open an audio file and read as mono waveform, resampling as necessary

    Parameters
    ----------
    file: (str, bytes)
        The audio file to open or bytes of audio file

    sr: int
        The sample rate to resample the audio if necessary

    Returns
    -------
    A NumPy array containing the audio waveform, in float32 dtype.
    """
    
    if isinstance(file, bytes):
        inp = file
        file = 'pipe:'
    else:
        inp = None
    
    try:
        # This launches a subprocess to decode audio while down-mixing and resampling as necessary.
        # Requires the ffmpeg CLI and `ffmpeg-python` package to be installed.
        out, _ = (
            ffmpeg.input(file, threads=0)
            .output("-", format="s16le", acodec="pcm_s16le", ac=1, ar=sr)
            .run(cmd="ffmpeg", capture_stdout=True, capture_stderr=True, input=inp)
        )
    except ffmpeg.Error as e:
        raise RuntimeError(f"Failed to load audio: {e.stderr.decode()}") from e

Does not work for video bytes

RuntimeError: 2D or 3D (batch mode) tensor expected for input, but got: [ torch.FloatTensor{1,1,0} ]

[glangford]

Hmmm...haven't tried this, but does using the -vn option to ffmpeg help?

-vn (input/output)
As an input option, blocks all video streams of a file from being filtered or being automatically selected or mapped for any output. See -discard option to disable streams individually.

As an output option, disables video recording i.e. automatic selection or mapping of any video stream. For full manual control see the -map option.

[Shivansh-yadav13]

@glangford
Modified the code

    try:
        # This launches a subprocess to decode audio while down-mixing and resampling as necessary.
        # Requires the ffmpeg CLI and `ffmpeg-python` package to be installed.
        out, _ = (
            ffmpeg.input(file, threads=0)
            .output("-", format="s16le", acodec="pcm_s16le", ac=1, ar=sr, vn=True) #here
            .run(cmd="ffmpeg", capture_stdout=True, capture_stderr=True, input=inp)
        )

but I got this

RuntimeError: Failed to load audio: ffmpeg version 6.0-essentials_build-www.gyan.dev Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 12.2.0 (Rev10, Built by MSYS2 project)
  configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-lzma --enable-zlib --enable-libs
rt --enable-libssh --enable-libzmq --enable-avisynth --enable-sdl2 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-libass --enable-libfreetype --e
nable-libfribidi --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-libg
me --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libtheora --enable-libvo-amrwbenc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-librubberband
  libavutil      58.  2.100 / 58.  2.100
  Duration: 00:00:48.55, start: 0.000000, bitrate: N/A
  Stream #0:0[0x1](und): Video: h264 (avc1 / 0x31637661), none, 1080x1920, 1891 kb/s, SAR 1:1 DAR 9:16, 30 fps, 30 tbr, 90k tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
  Stream #0:1[0x2](und): Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]
[NULL @ 000002097fa1e400] Unable to find a suitable output format for 'pipe:'
pipe:: Invalid argument