Vits vs FastPitch

[bzp83]

Has anyone tried to train Vits and FastPitch with the same dataset and compared the results?

Which one is better?

I need to train a new TTS model and wanted to hear about other's experiences doing that.

Thanks!

[yukiarimo]

Updates?

[bzp83]

I ended up testing both...

Fastpitch doesn't seem to get all the nuances of my dataset. For example, I have audio clips with single words ending with an !, like "Stop!" and also "Stop." where the difference is that "Stop!" is spoken with much more energy then "Stop." but fastpitch doesn't seem to catch those... when I run inference with "Stop!" and "Stop.", it sounds the same for both. I couldn't find any pronunciation problems whatsoever and it also fails badly on long phrases... I think I could work around it by splitting long texts by ,.; etc, but if you try to infer a long text, it starts failing midway.

Vits on the other hand seems to catch those nuances pretty well! It also handles long text really well and as long as you have clips with short words and single letters, it also works nice with short texts. However, it has some very annoying random pronunciation problems when you you sdp, if you don't, then it seems this problem is gone, but the output audio sounds very robotic and not natural at all. Also, as vits is end to end, I don't think it's possible to finetune the decoder to improve the output audio quality. I'd love to know if/how to do it. Depending on how picky you are, you can notice differences between the generated audio and gt and it seems like you can never match the quality. Maybe you can if you tweak the params but I didn't.

I really wish fastpitch worked as well as vits did so I could just use any other vocoder and also fine tune it. I also loved the pitch adjustments that fastpitch lets you do, for example, unless you have tons of question phrases, generated audios of question will sound like a simple sentence and you can write some code to increase the pitch in the end of phrases with fastpitch so it sounds like a question and it's really cool! Not sure if there's any other way to do the same with vits.

That's my experience. I would love to create a new model where I just joined the best of both models but that's beying my knowledge. :)