can provide instructions for how to use deepspeed

[roymina]

everything is working like a charm, great job! Except, when I use xtts_v2 for Chinese voice synthesis, it's slower than with English. I've installed DeepSpeed, but I'm not sure if it's working cause it's still slow. Also, I saw in the docs that "activating DeepSpeed replaces the TTS class with a custom one, FakeTTSWithRawXTTS2. It's like a simple wrapper around the raw XTTS v2.0 with the same API, and it's supposed to enable free voice cloning. ". How do I set that up? Could you maybe write up some instructions for it? By the way, setting streaming_enabled : True didn't seem to help.

[Scthe]

I've updated the code to not require DeepSpeed for streaming. Probably will not make much difference in your case. I've also added debug logs to explain why some settings are ON/OFF. Screenshot from my settings at the end.

Streaming

Streaming works best with longer sentences. It may cause artifacts between chunks, which is why it's disabled by default.

Requires model tts_models/multilingual/multi-dataset/xtts_v2 (the default) and tts.streaming_enabled in the yaml config.

DeepSpeed

See the install and usage file. Compare it with the program output I've attached below (as a screenshot).

Voice cloning

1. Turn ON streaming: streaming_enabled: True
   1. There should be a way to voice clone in XTTS2 without streaming, but I did not look much into it.
2. Uncomment "sample_of_cloned_voice_wav" setting that points to your WAV file. The current line expects "voice_to_clone.wav" in the root of the repo.
3. Make sure "voice_to_clone.wav" exists in the root of the repo.

IIRC Audacity export options are:

• Format: WAV
• Channels: Mono
• Sample Rate: 44100 Hz
• Encoding: Signed 16-bit PCM

XTTS2 docs had more info somewhere.

Chinese voice synthesis

I'm not surprised, XTTS2 seems to handle English best. You would have to find a TTS that was developed for Chinese from ground up. Quick GitHub search returns ChatTTS - "The main model is trained with Chinese and English audio data of 100,000+ hours". It's also relatively new. I do not have any plans to support it, this app is just a proof of concept.

The Unity app requires WAV file send over WebSocket, so it does not care about what you use on backend. I'm also not sure how Oculus Lipsync works with Chinese. I assume that it was also trained on predominately English sounds.

As for speed, there is always a question of hardware. ElevenLabs spends a lot of time/money optimizing its software and runs it on best-in-class GPUs. All the announcements in the news also often just throw the money at the problem. XTTS2 is an open-source project and I run it on RTX 3060, a $350 GPU.

Sample console output