Whisper Audio dual channel problem?

[vopbs]

Our audio files are dual-channel files, divided into left and right channels. At present, I use whisper for speech recognition, if the wave library for processing, split into the left and right channels for recognition, but it will affect the recognition results. I have observed a speech recognition model of whisper context association. What I want to ask is that if the recognition is carried out in the disassembly track, it will greatly affect the recognition accuracy. Is there any other way?

[coastal45]

How are you splitting left/right channels? For example, left only --> whisper? Or, left --> mono (left side audio only) --> whisper?

[vopbs]

Because the audio software input is developed by ourselves, it is a two-person dialogue mode, so it can be guaranteed to be dual-track.

[coastal45]

I can't envision your method, but my idea is to split left and right channels first (by ffmpeg), then process each channel in whisper as mono separately.
Finally, combine the whisper text results.
Would that help in your situation?

[dgoryeo]

Hi @coastal45 , when you combine two text results, how do you resolve situations when / if same line is repeated in both L/R channels?

[coastal45]

@dgoryeo OP never made it clear what the actual channel contents are (distinct or concurrent speech). Though "two-person dialogue" might not indicate concurrent speech.
In my case, they are distinct content (languages).
Probably content of channels needs to be determined before "combine" is considered.

[vopbs]

Hi @dgoryeo What I recognize is a phone call recording file, then the track is split, and the left and right channels are clearly separated, that is, two people are dialing the phone.

[oguangxili]

I also have the same problem. have you resolved it?

[Redozator]

import whisper
from pydub import AudioSegment
import pandas as pd

model = whisper.load_model("large-v2")

filenum = '4'
audio = AudioSegment.from_file(filenum + '.wav')
stereo_sound = audio.split_to_mono()
stereo_sound[0].export(filenum + '_left.wav', format="wav")
stereo_sound[1].export(filenum + '_right.wav', format="wav")
result1 = model.transcribe(filenum + '_left.wav',  word_timestamps=True, verbose=True )
result2 = model.transcribe(filenum + '_right.wav', word_timestamps=True, verbose=True )

df = pd.DataFrame(columns = ['speaker', 'segment', 'start', 'text'])
for i in result1['segments']:
  if len(i['words']) > 0:
    newdf = pd.DataFrame({'speaker':[1], 'segment':[i['id']], 'text':[i['text']], 'start':[i['words'][0]['start']]});
    df = pd.concat([df, newdf])
for i in result2['segments']:
  if len(i['words']) > 0:
    newdf = pd.DataFrame({'speaker':[2], 'segment':[i['id']], 'text':[i['text']], 'start':[i['words'][0]['start']]});
    df = pd.concat([df, newdf])
df = df.sort_values(['start'])

texts = []
for i, row in df.iterrows():
  # texts.append('[' + str(round(row['start'])) + '] ' +str(row['speaker']) + ') ' + row['text'])
  texts.append(str(row['speaker']) + ') ' + row['text'])

with open(filenum + '.txt', 'w') as f:
  for t in texts: 
    f.write(t + "\n")