VAD Issues using medium.en

[peardox]

As described in #3139 grab the audio for Aladdin from Archive.org and run it through whisper-cli to get the hang of things (I rename the horrid filename to aladdin.mp3)

Now run it through VAD and output to json so something like this...

whisper-cli -m ggml-medium.en.bin -f aladdin.mp3 --vad -vm ggml-silero-v5.1.2.bin -ojf -of aladdin

Examine the json and note that the first word is VERY long... 16 seconds to say 'there'

{
	"text": " There",
	"timestamps": {
		"from": "00:00:01,190",
		"to": "00:00:17,320"
	},
	"offsets": {
		"from": 1190,
		"to": 17320
	},
	"id": 1318,
	"p": 0.156763,
	"t_dtw": -1
},

In reality what's happened is that medium.en has cut a load of text at the start (credit stuff) and included it in 'there'

This happens with varying degrees depending on which model is used. The smaller the model the less de-crediting occurs

[danbev]

I've tried aladdin.mp3 with the latest master which includes #3173, and I get the following ouput (taken from aladdin.json):

 "transcription": [                                                            
    {                                                                           
      "timestamps": {                                                           
        "from": "00:00:01,120",                                                 
        "to": "00:00:25,960"                                                    
      },                                                                        
      "offsets": {                                                              
        "from": 1120,                                                           
        "to": 25960                                                             
      },                                                                        
      "text": " There once lived a poor Taylor, who had a son called Aladdin, a careless idle-boy, who",
      "tokens": [                                                               
         ...

Would you be able to re-run this using the latest on the master branch and see if you still getting the incorrect output?

[peardox]

Yeah, did a manual conflict-resolution yesterday so I now have that stuff in my fork

The issue is still there - your json doesn't go quite far enough down. The word 'there' actually occurs about 20s in if you listen to the track.

Check lines 56-69 in this json

aladdin.json.zip

[danbev]

Thanks for that! I see that these are the token level timestamps and I'm looking into this as part of #3174.

[peardox]

If you re-run the same test with one of the very small models you'll get different output. The problem stems from the fact that Whisper edits out credits from an input stream (mentioned before in another issue - got ignored)

[danbev]

After looking into this I don't think this is specifically related to VAD...

If we look at how OpenAI's Whisper process this audio file:

(whisper-venv) $ whisper samples/aladdin.mp3 --model medium.en --word_timestamps True
100%|█████████████████████████████████████| 1.42G/1.42G [07:07<00:00, 3.57MiB/s]t
[00:00.000 --> 00:25.960]  There once lived a poor tailor, who had a son called Aladdin, a careless idle boy, who
[00:25.960 --> 00:31.760]  would do nothing but play all day long in the streets with little idle boys like himself.
[00:31.760 --> 00:34.960]  This so grieved the father that he died.
[00:34.960 --> 00:40.200]  Yet in spite of his mother's tears and prayers, Aladdin did not mend his ways.
[00:40.200 --> 00:45.720]  One day, when he was playing in the streets as usual, a stranger asked him his age, and
[00:45.720 --> 00:48.320]  if he was not the son of Mustapha the tailor.
[00:48.320 --> 00:53.840]  "'I am, sir,' replied Aladdin, but he died a long while ago.
[00:53.840 --> 01:00.200]  On this the stranger, who was a famous African magician, fell on his neck and kissed him,
[01:00.200 --> 01:01.200]  saying,
[01:01.200 --> 01:05.360]  "'I am your uncle, and knew you from your likeness to my brother.
[01:05.360 --> 01:09.840]  Go to your mother and tell her I am coming.'
[01:09.840 --> 01:13.360]  Aladdin ran home and told his mother of his newly found uncle.
[01:13.360 --> 01:15.680]  "'Indeed, child,' she said.
...

And the first token timestamp:

  "segments": [
    {
      "id": 0,
      "seek": 0,
      "start": 20.059999999999995,
      "end": 28.98,
      "text": " There once lived a poor tailor, who had a son called Aladdin, a careless idle boy, who",
      "tokens": [
        50363,
        1318,
        1752,
        5615,
        257,
        3595,
        35280,
        11,
        508,
        550,
        257,
        3367,
        1444,
        978,
        46782,
        11,
        257,
        36138,
        21696,
        2933,
        11,
        508,
        51661
      ],
      "temperature": 0.0,
      "avg_logprob": -0.22713401794433594,
      "compression_ratio": 1.075,
      "no_speech_prob": 0.0505792535841465,
      "words": [
        {
          "word": " There",
          "start": 20.059999999999995,
          "end": 20.639999999999997,
          "probability": 0.1740105301141739
        },

And using whisper-cli:

[00:00:00.000 --> 00:00:25.960]   There once lived a poor tailor, who had a son called Aladdin, a careless idle boy, who
[00:00:25.960 --> 00:00:31.760]   would do nothing but play all day long in the streets with little idle boys like himself.
[00:00:31.760 --> 00:00:34.960]   This so grieved the father that he died.
[00:00:34.960 --> 00:00:40.200]   Yet in spite of his mother's tears and prayers, Aladdin did not mend his ways.
[00:00:40.200 --> 00:00:45.720]   One day, when he was playing in the streets as usual, a stranger asked him his age, and
[00:00:45.720 --> 00:00:48.400]   if he was not the son of Mustapha the tailor.
[00:00:48.400 --> 00:00:53.840]   "I am, sir," replied Aladdin, "but he died a long while ago."
[00:00:53.840 --> 00:01:00.160]   On this, the stranger, who was a famous African magician, fell on his neck and kissed him,
[00:01:00.160 --> 00:01:07.000]   saying, "I am your uncle, and knew you from your likeness to my brother; go to your mother,
[00:01:07.000 --> 00:01:09.860]   and tell her I am coming."
[00:01:09.860 --> 00:01:13.360]   Aladdin ran home and told his mother of his newly found uncle.
[00:01:13.360 --> 00:01:20.800]   "Indeed, child," she said, "your father had a brother, but I always thought he was dead."

And the first token timestamp:

 "transcription": [
    {
      "timestamps": {
        "from": "00:00:00,000",
        "to": "00:00:25,960"
      },
      "offsets": {
        "from": 0,
        "to": 25960
      },
      "text": " There once lived a poor tailor, who had a son called Aladdin, a careless idle boy, who",
      "tokens": [
        {
          "text": "[_BEG_]",
          "timestamps": {
            "from": "00:00:00,000",
            "to": "00:00:00,000"
          },
          "offsets": {
            "from": 0,
            "to": 0
          },
          "id": 50363,
          "p": 0.781686,
          "t_dtw": -1
        },
        {
          "text": " There",
          "timestamps": {
            "from": "00:00:01,120",
            "to": "00:00:21,120"
          },
          "offsets": {
            "from": 1120,
            "to": 21120
          },
          "id": 1318,
          "p": 0.374393,
          "t_dtw": -1
        },

Notice that this differs from OpenAI's whisper in that the from is 00:00:01,120
in our case and 00:00:20,059 in OpenAI's Whisper.

And using whisper-cli with VAD enabled:

[00:00:01.120 --> 00:00:25.960]   There once lived a poor Taylor, who had a son called Aladdin, a careless idle-boy, who
[00:00:25.960 --> 00:00:31.750]   would do nothing but play all day long in the streets, with little idle-boys like himself.
[00:00:31.750 --> 00:00:37.790]   Aladdin so grieved the father that he died; yet in spite of his mother's tears and prayers,
[00:00:37.790 --> 00:00:40.180]   Aladdin did not mend his ways.
[00:00:40.180 --> 00:00:45.670]   One day, when he was playing in the streets as usual, a stranger asked him his age, and
[00:00:45.670 --> 00:00:48.140]   if he was not the son of Mustapha the Taylor.
[00:00:48.140 --> 00:00:53.830]   "I am, sir," replied Aladdin; but he died a long while ago.
[00:00:53.830 --> 00:01:00.070]   On this the stranger, who was a famous African magician, fell on his neck and kissed him,
[00:01:00.070 --> 00:01:05.160]   saying, "I am your uncle, and knew you from your likeness to my brother.
[00:01:05.160 --> 00:01:09.820]   Go to your mother, and tell her I am coming."
[00:01:09.820 --> 00:01:13.480]   Aladdin ran home, and told his mother of his newly found uncle.
[00:01:13.480 --> 00:01:19.430]   "Indeed, child," she said, "your father had a brother, but I always thought he was dead."

And the first token timestamp:

  "transcription": [
    {
      "timestamps": {
        "from": "00:00:01,120",
        "to": "00:00:25,960"
      },
      "offsets": {
        "from": 1120,
        "to": 25960
      },
      "text": " There once lived a poor Taylor, who had a son called Aladdin, a careless idle-boy, who",
      "tokens": [
        {
          "text": "[_BEG_]",
          "timestamps": {
            "from": "00:00:01,120",
            "to": "00:00:01,120"
          },
          "offsets": {
            "from": 1120,
            "to": 1120
          },
          "id": 50363,
          "p": 0.925896,
          "t_dtw": -1
        },
        {
          "text": " There",
          "timestamps": {
            "from": "00:00:01,120",
            "to": "00:00:21,080"
          },
          "offsets": {
            "from": 1120,
            "to": 21080
          },
          "id": 1318,
          "p": 0.179201,
          "t_dtw": -1
        },

Notice here that we have the same from and to timestamp issue as with non VAD whisper.cpp.

Edit: I'm taking notes related to this issue here.

[peardox]

Looking at the --word_timestamps True output then this will potentially affect lots of audio in varoius ways and I just happen to have stumbled accross one edge case

[danbev]

@peardox While looking into this further I noticed that it is possible to set a timestamp offset parameter -ot/--offset-t which you might be able to us. Like I mention in the linked section, OpenAI's Whisper seems to be able to do this automatically and I'll look into this further, but just in case using this parameter is an option then that might unblock you.

[peardox]

Yeah, been thinking about it. An alternative may be to scan the audio with base (which doesn't censor audio) to get the 'shape' of the audio then use medium / large for better results.

[ggerganov]

Since the reference OpenAI implementation produces the same results as whisper.cpp, I don't think this bug report is valid. It is just a limitation of the model.