Newer model file is performing worse than older one with same settings??

[FurkanGozukara]

I am using large-v2.pt and it is performing worse for the following strategy

This is the Python command I use

beam_size=10
initial_prompt=Welcome to the Software Engineering Courses channel
best_of=10
verbose=True
temperature=0.0
language="en"
model=large-v2

    model = whisper.load_model("../input/whisper2/large-v2.pt")
    
        result = model.transcribe("../input/whisper2/lecture_"+str(lectureId)+".mp3",language="en",beam_size=10,initial_prompt="Welcome to the Software Engineering Courses channel.",best_of=10,verbose=True,temperature=0.0)

Here I attaching older vs new

This is the video I have transcribed

https://www.youtube.com/watch?v=rz64JM1H20k&list=PL_pbwdIyffskoSXySh0MdiayPJsBZ7m2o&index=8

model_v2.txt is the newer model transcribe and model_v1.txt is the previous one

model v1.txt

model v2.txt

In the image below left one is model 2 which one is supposed to be better but performs worse

[FurkanGozukara]

And here is the comparison of vtt file. left is newer and right is older one

[FurkanGozukara]

I have done multiple more tests. version 1 works better on multiple voice and worse quality audio. version 2 works better on more clear audio.

[jongwook]

Thanks for sharing your tests. The large-v2 model performs better on average, but the improvement is not consistent across different audio, just like in some cases the medium model often performs better than large on their audio.

The first example seems to have gotten stuck at a repetition loop, a failure mode of low-temperature decoding. We haven't systematically evaluated the difference in tendency to go repetition looping between large-v1 and large-v2, so I can't recommend specific parameters. In your case, I'd check the return value of transcribe() for the "compression_ratio" field of the repeating segments, and try setting the --compression_ratio_threshold value slightly lower than that.