Most accurate model for English?

[wenchaoliu-93]

Medium English only model or large multi-language model?
"The .en models for English-only applications tend to perform better."

[ryanheise]

Go to the README page and click on the "Paper" link. Section 4.3 answers your question including a diagram:

Our results visualized in Figure 9 show that for small models
trained with moderate amounts of compute, there is indeed
negative transfer between tasks and languages: joint mod-
els underperform English-only models trained for the same
amount of compute. However, multitask and multilingual
models scale better and for our largest experiments outper-
form their English-only counterparts demonstrating positive
transfer from other tasks. For our largest experiments, joint
models also slightly outperform English-only models even
when not adjusting for compute spent per task.

[amrreda225]

hello
whisper ai does not work for me can you figure out why ?
pelase check my last question in discussion

[wenchaoliu-93]

The models available for downland, based on my understanding, have been trained. This is, however, talking about training, so it hardly answers my question.

[ryanheise]

The models available for downland, based on my understanding, have been trained.

ALL models (by definition) have been trained. I'm not sure if you think that there are other kinds of models that have not been trained, but that doesn't really make sense. An untrained model wouldn't really be a model of anything, and so the training of it is part and parcel of models. Or if you think that the paper is maybe talking about secret models that they haven't made available for download as opposed to the ones that ARE available for download and that these are somehow different, then no, the paper explicitly mentions the models that they have published and which are available for download. The paper tells you that the larger multilingual/multitask models outperform the English-only models, and so it does answer your question.

[wenchaoliu-93]

The left part of diagram shows the English model can outperform.

…

On Fri, Jul 21, 2023 at 11:44 PM ryanheise ***@***.***> wrote: The models available for downland, based on my understanding, have been trained. ALL models (by definition) have been trained. I'm not sure if you think that there are other kinds of models that have not been trained, but that doesn't really make sense. An untrained model wouldn't really be a model of anything, and so the training of it is part and parcel of models. Or if you think that the paper is maybe talking about secret models that they haven't made available for download as opposed to the ones that ARE available for download and that these are somehow different, then no, the paper explicitly mentions the models that they have published and which are available for download. The paper tells you that the larger multilingual/multitask models outperform the English-only models, and so it does answer your question. — Reply to this email directly, view it on GitHub <#1540 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A47TGBYPCKJYIIKL7KLDG33XRNECPANCNFSM6AAAAAA2TQH2AE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ryanheise]

Yes. The quoted part of the paper explains that. For the smaller models, the multilingual/multitask versions perform worse, for the larger models, the transfer effect kicks in and the multilingual/multitask versions start to outperform. But the fact that you only just noticed this now suggests to me that you didn't actually read Section 4.3 before telling me that I "hardly" answered your question. It's all there in the section that I quoted.

[wenchaoliu-93]

I was confused. Apologies. That said, I did read the paper after you quoted it, particularly that section.
How much is medium vs. large? I wasn't able to find that out in the paper. If the large training set is on the left of the intersection, medium English would be better.

[ryanheise]

That's a good question, although I don't have any information on where exactly the crossover point is.

[Tex2002ans]

Most accurate model for English?

largev2 is the most accurate for all languages, even English.

English WER:

• 10.9% = medium
• 9.9% = largev2

So, for English, largev2 is only marginally better than medium.

On page 11 of the Whisper paper (PDF) is this graph and table showing the stats:

---

The way that I read @ryanheise 's graph/reference (Section 4.3 + Figure 9), is:

IF you have English audio, then:

• The .en models will will outperform the multi-language models.
  • tiny.en will be better than tiny.
  • base.en will be better than base.
  • small.en will be better than small.
  • medium.en will be better than medium.

A tipping point happened at large though. In that case:

• Whisper's multi-lingual model (large) became more accurate than the English-only training.
  • You can see this in Figure 9, where the orange line crosses, then starts going below the blue.
  • This is why there's no such thing as "large.en", because it performed worse than large!

---

I'll repost the relevant section here. It explains the technical details behind WHY the larger model, when fed more data, began outperforming the single-language English-only version.

4.3. Multitask and Multilingual Transfer

A potential concern with jointly training a single model on many tasks and languages is the possibility of negative transfer where interference between the learning of several tasks results in performance worse than would be achieved by training on only a single task or language. To investigate whether this is occurring, we compared the performance of models trained on just English speech recognition with our standard multitask and multilingual training setup and measured their average performance across our suite of zero-shot English speech recognition benchmarks. We adjust for the amount of FLOPs spent training on the task of English speech recognition as only 65% of compute is spent on this task in a joint training setup; analysis would otherwise be confounded by under-training on the task when compared to a same-sized English-only model.

Our results visualized in Figure 9 show that for small models trained with moderate amounts of compute, there is indeed negative transfer between tasks and languages: joint models underperform English-only models trained for the same amount of compute. However, multitask and multilingual models scale better and for our largest experiments outperform their English-only counterparts demonstrating positive transfer from other tasks. For our largest experiments, joint models also slightly outperform English-only models even when not adjusting for compute spent per task.

FIGURE 9 IMAGE

Figure 9. Multitask and multilingual transfer improves with scale. For small models, performance on English speech recognition degrades when trained jointly in a multitask and multilingual setup. However, multilingual and multitask models benefit more from scale and eventually outperform models trained on English data only. 95% bootstrap estimate confidence intervals are shown.

[wenchaoliu-93]

Good answer. That said, I want to point out that the table talks about "data size" while in the graph it is "FLOPS training." Certainly, it makes sense that they are positively correlated. "large.en performed worse than large" seems like a reasonable interpretation, but an interpretation, nevertheless. Note that it is, on GitHub, large, not large-v2 as shown in the paper.
I also want to point out the distinction between large and large-v2. It's clearly distinguished here but not on GitHub. I'd assume it is the original version (large) on GitHub.

[ryanheise]

Note that it is, on GitHub, large, not large-v2 as shown in the paper.

When version 2 of large was released, the original large (on GitHub) was renamed to large-v1, the new large-v2 was introduced, large was made as an alias pointing to large-v2, and the paper was updated to reflect the numbers from the new large model all at the same time. The announcement was here: #661

[Tex2002ans]

The announcement was here: #661

Yep. Exactly. :)

After December 2022:

• They released an updated paper with the new v2 data.
• large by default now means large-v2, and the original large model was relegated to large-v1.

---

I do not have the "original original" release of the paper, so I'm unsure exactly what graphs/data/tables were changed, but I'm suspecting all references in the updated paper's large instantly meant large-v2.

[RazyRo]

would you mind adding the required RAM / VRAM needed for each model?

[glangford]

would you mind adding the required RAM / VRAM needed for each model?

@RazyRo The approximate sizes of each model are listed in the README

Available models and languages
https://github.com/openai/whisper?tab=readme-ov-file#available-models-and-languages