TTS and STS Models to port to MLX-Audio (Roadmap)

[Blaizzy]

Overview

This issue outlines our roadmap for integrating additional text-to-speech (TTS) and speech-to-speech (STS) models into the MLX-Audio library to expand our offerings beyond the current Kokoro model.

Text-to-Speech (TTS) Models

Planned TTS Models

• Nari Labs Dia 1.6B
• OuteTTS v1
• Orpheus
• BARK
• SparkTTS 0.5B
• Sesame CSM-1B
• IndexTTS
• ChatterBox
• VibeVoice
• VyoTTS
• MegaTTS
• Zonos
• CosyVoice2
• StyleTTS2
• Parler TTS
• ibm-granite/granite-speech-3.2-8b
• LLMVoX
• MeloTTS
• bosonai/higgs-audio-v2

Speech-to-Speech (STS) Models

Planned STS Models

• Kyutai-Labs Moshi
• Kyutai-Labs Moshi-vis

Speech-to-text (STT)

• Whisper
• Parakeet
• Wav2vec
• Voxtral
• Canary

Technical Considerations

• All models will need MLX-specific optimizations
• Quantization support should be implemented for each model
• Documentation and examples will be created for each new model
• Performance benchmarks will be established

Instructions:

1. Select the model and comment below with your selection
2. Create a Draft PR titled: "Add support for X"
3. Read Contribution guide
4. Check existing models
5. Tag @Blaizzy for code reviews and questions.

Community Input

We welcome community feedback on prioritization and additional model suggestions. Please comment on this issue with your thoughts.

[dwohlfahrt]

As always, thanks a MILLION for all the work you do @Blaizzy. You are a legend in the truest sense of the word 🙏

And now, of course, I have to chime in with my own selfish requests 😄

1. For TTS, adding on a +1 for Zonos
2. For STS, RVC would be huge, as I run kokoro outputs through it using RVC-generated fine tunes of custom voices to basically get the best of both worlds (aka kokoro with true voice cloning). Results are absolute 🔥 😃

[Blaizzy]

Thanks a lot, it's my pleasure!

Yes, Zonos is on the way 🚀

Could you share this RVC + Kokoro example?

[szafranek]

I found this project through your awesome demo.

Would you consider supporting StyleTTS2?

[chigkim]

This is amazing!!!
@Blaizzy when do you sleep? VLM, now Audio?
Anyways, there's Outes. It's based on text LLMs, so even llama.cpp and ExLlamaV2 can run it.
Thanks!

[chigkim]

Orpheus-TTS is just released, and it sounds really good!
https://github.com/canopyai/Orpheus-TTS

[lin72h]

@chigkim the KING already start working on it: #47

[chigkim]

Another great sounding model: nari-labs/dia
Demo: https://yummy-fir-7a4.notion.site/dia
Github: https://github.com/nari-labs/dia
HF: https://huggingface.co/nari-labs/Dia-1.6B

[Blaizzy]

Indeed @chigkim that's a great suggestion!

@lucasnewman is working on it 🚀

[chigkim]

Here's another good one. :)
https://github.com/resemble-ai/chatterbox
https://huggingface.co/ResembleAI/chatterbox
https://huggingface.co/spaces/ResembleAI/Chatterbox

[Blaizzy]

Thanks @chigkim!

@BenLumenDigital is on it 🚀

[mseri]

For STT, there are also voxtral small and mini that seem to be quite good

[niklasmato]

For STT, there are also voxtral small and mini that seem to be quite good

I'm actually trying to get this running but it's not that easy.