Transcribe error for webm file - Error: expected np.ndarray (got bytes)

[miesepeter]

Hi! I am attempting to use transcribe() but I am running into two (probably connected) issues. Disclaimer: I am a beginner, so the mistake I made might be obvious.

Setting: I am locally (Ubuntu 22.04.2) running a flask app which records audio. This audio is then processed using transcribe - which currently fails. To solve this I would greatly appreciate any input on how to transcribe a webm file recorded with the mediaRecorder

Below my code to record the audio:

    mediaRecorder.addEventListener("stop", () => {
        // create blob from recordedChunks array
        const audioBlob = new Blob(recordedChunks, { type: options.mimeType });
        // Reset the recorded chunks array
        recordedChunks = []; 

        //file reader to read blob as array buffer for submission
        const reader = new FileReader();
        // listen for load event of file reader (reading blob)
        reader.onload = (event) => {
            // send audio data via socket
            socket.emit("audio", event.target.result);
        };
        reader.readAsArrayBuffer(audioBlob);
    });

The data is received and supposedly handled by:

@socketio.on("audio")
def handle_audio(audio):
    whisper_model = whisper.load_model("tiny")
    result = whisper_model.transcribe(audio)
    socketio.emit("response", result)

This fails - the transcribe method throws the following error: "TypeError: expected np.ndarray (got bytes)"

To solve this I attempted to convert to numpy myself, changing the "handle_audio()" to the following:

@socketio.on("audio")
def handle_audio(audio):
    # Load the audio data into AudioSegment object
    audio_segment = AudioSegment.from_file(BytesIO(audio), format="webm")
    # Convert the audio into format
    audio_segment = audio_segment.set_channels(1).set_frame_rate(16000)
    # Convert the audio data to NumPy array
    audio_data = np.frombuffer(audio_segment.get_array_of_samples(), dtype=np.int16)

    whisper_model = whisper.load_model("tiny")
    result = whisper_model.transcribe(audio_data)
    socketio.emit("response", result)

This unfortunately fails as well with: "Segmentation fault (core dumped)". I have not managed to find this coredump unfortunately.

[phineas-pta]

audio inputs to .transcribe must be either a string as path to audio file or numpy array or pytorch tensor, it cannot be directly audio data from javascript

you can either save your audio as file or convert it to array/tensor

[phineas-pta]

what error did you get with this code ?

[miesepeter]

I simply get a: "Segmentation fault (core dumped)" unfortunately

[phineas-pta]

this error is sometimes related to having not enough RAM, how big is your audio ?

[miesepeter]

My self recorded test file is less than 10 seconds long. To crosscheck I downloaded a random mp3 I found online which I was able to transcribe without problems. That makes me thing that there is some problem with my recording or transformation into a usable format.

[phineas-pta]

it is possible to save the audio somewhere in javascript then load in python ?