Potentially useful pretrained models

[gkucsko]

Hey friends, we recently release some text-to-audio models that are very similar to the great work you all are doing here. Maybe some of them can be used as a starting point for finetuning within this repo (esp semantic and coarse/fine).
https://github.com/suno-ai/bark

[lucidrains]

@gkucsko what are your plans around Encodec's restrictive non-commercial license?

[gkucsko]

great question. that's the main reason that we had to release ours non-commercially as well. Internally we have our own stuff, but for now that's too tied into other projects for us to release. I've been reading up a bit on the status of your soundstream implementation and it looks like it's getting pretty close? happy to see if i can help there in any way in some free time. btw in my experience you can get pretty far in a codec with standard objectives like demucs etc meaning SpectralConvergengeLoss and LogSTFTMagnitudeLoss. The adversarial stuff is nice for fudging higher quality, but not strictly needed to get decent results and obviously way easier to train and get to converge

[lucidrains]

@gkucsko yea, i think it is nearly there 😄 various researchers have emailed me saying they are using it, but we could use some open sourced model in different domains

re: losses, thanks for sharing that tidbit! i'll make sure the adversarial losses can be turned off later this week

[lucidrains]

@gkucsko just so you know! #175

[jamesparsloe]

Hey @gkucsko great work with Bark! I see you are using SEMANTIC_VOCAB_SIZE = 10_000 - is this from increasing the number of kmeans clusters for Hubert or vq-wav2vec (or something else)? Does this significantly help the model compared to using something smaller like 500? Thanks

[gkucsko]

we use our own embeddings but yeah similar idea. in my experience 200-500 is enough to capture most speech. 10k captures a bit more which can be good or bad depending on what you are aiming for

[jamesparsloe]

Thanks