Finetuning whisper with a specific language dataset degrades other language's accuracies

[yilmazay74]

Hi Everybody,
Recently we are playing around to fune tune whisper's base, small and medium models with our own Turkish dataset. Currently, the accuracy level we reached is not bad. However, whisper's multi language feature is the most attractive of all. Because, we don't want to loose performance in other languages' after finetuning in a specific language, to be specific, lets say, Turkish. When we evaluate Chinese, Russian or Arabic audios with the fine tuned model, we see that the accuracy degraded significantly. For example, with Arabic audios the resulting text mostly has Latin characters instead of Arabic. We see the same stuation with with Russian and Chinese as well.
So, my question is:
Is it possible to do a finetuning with a specific language only, without any negative side effects on other languages in the model?
a) - If yes: How do we do finetuning without negatively effecting other languages?
b) - If no: Then, does not it mean that fine tuning whisper does not have multilingual support ?

Appreciate your answers and suggestions
REgards

[phineas-pta]

u can ask @andrespimartin he has tried something related #1432 #1454