A pipeline for converting speech to text from noisy environment

[payamsash]

Dear

I am working on converting the speech of a specific individual into text from audio that contains both background noise and speech from other people. Could you advise on the appropriate pipeline to achieve clear and accurate transcription? Should I perform steps such as noise reduction and speaker isolation, or can advanced models handle the raw audio data effectively?

Best,
Payam

[samarasimhapeyala]

You should preprocess the audio files with noise reduction or suppression before passing to the whisper. There is no such perfect pipeline. Better mechanisms are demucs https://github.com/facebookresearch/demucs for separation of tracks, noise reduce https://github.com/timsainb/noisereduce for noise suppression and https://github.com/haoheliu/voicefixer for speech enhancement.

[gongouveia]

@samarasimhapeyala @payamsash from my experience I don't agree with this comment. Pre processing audio with demucs highly degrades the initial signal, although hearing it is great, the signal is degraded and with some artifacts in phase.
I recommend to fine tune it with noisy audio.

[samarasimhapeyala]

@gongouveia I agree with the point that the audio signal degradation and artifacts with the pre-processing steps but hoping this mayn't affect much to the transcription.