Chatterbox-Turbo support finetuning

[tuteishygpt]

Do you plan to add fine-tuning support for Chatterbox-Turbo?

[gokhaneraslan]

Yes, absolutely! I've just pushed an update that adds full fine-tuning support for Chatterbox-Turbo.
You can now control which model you want to train by using the is_turbo flag in the src/config.py file. The setup.py script will even automatically prepare the multi-language tokenizer for you when Turbo mode is enabled.
Please pull the latest changes and check out the updated README for instructions.

[tuteishygpt]

Thank you very much!

Can you please provide detailed instructions if I want to add a new language to Chatterbox-Turbo (is_turbo: bool = True)?

Which files do I need to edit?

1. Do I need to add my language token, e.g. BE - Belarusian, to "grapheme_mtl_merged_expanded_v1.json" or to another one?

Also add letters from my language as separate tokens? The large vocabulary (~50,000+ tokens) does not have 1 character from my language.

2. Check and update NEW_VOCAB_SIZE in both src/config.py and inference.py. It matches the number of tokens in tokenizer.json.

3. Run setup.py?

4. Run train (where I should set my training language)?
   Or do I need to set the language in metadata.csv like this: "recording_001|Hello world.| [BE] hello world

[gokhaneraslan]

Hello!
Set the mode to Turbo from src/config.py
Run the python setup.py command.

python setup.py

Step 1: Add Your Language Characters (ONE Manual Step)
This is the only file you need to edit manually.
Open the Grapheme File: Open the file pretrained_models/grapheme_mtl_merged_expanded_v1.json. This file contains all the unique characters (graphemes) for the 23 languages ​​we currently support.
Add Your Characters: Add the missing characters from your language (e.g., Belarusian) to this file. You need to add them as new key-value pairs, continuing to number them. For example, if the last entry is "token": 2453, you can add your new characters starting from 2454.
Example "a":
code
JSON
// inside the pretrained_models/grapheme_tokenizer.json file
// find the end of the "vocab" dictionary
...
"ž": 2453,
"a": 2454 // <-- Add your new character here
// Add any other missing characters...
}
Make sure you add all the unique characters your dataset uses.

Step 2: Run Automatic Setup
Now that you've notified the system of your new characters, the setup script will handle the rest.

Run the python setup.py command again.

python setup.py

It will read the updated grapheme_mtl_merged_expanded_v1.json file and intelligently integrate your new characters into the underlying GPT-2 tokenizer.

Step 3: Update Dictionary Size
The setup script will be complete, giving you a new dictionary size.
Get the New Size: At the end of the script you will see a message like this:
Please update the 'new_vocab_size' value in the 'src/config.py' file to: 52265
Update Configuration: Copy this number exactly and paste it into the new_vocab_size variable in the src/config.py file.

Step 4: Prepare and Train Your Dataset
Now you are ready to train!
No Need for Custom Language Tags: You don't need to add language tags like [BE] to your metadata.csv file or add languages ​​elsewhere. Just type the text normally in your language. The tokenizer now understands your characters.

Run Preprocessing and Training: Set preprocess = True in your configuration file (at least for the first run) and start training as described in the README file.

python train.py

That's it! You don't need to edit any other files. The system is designed so that you only need to define the characters, and the automation handles the complex parts.

[tuteishygpt]

Thanks for the instructions.
One question about step 4: Preparing and training the dataset:
If we're using the multilingual vocabulary from grapheme mtl_merged expanded v1.json, why aren't we setting language_id the same as in Chatterbox-Multilingual-TTS?

After fine-tuning, will we get a model with only one target language?
Can we fine-tune by adding a new language and keeping all the old languages?

[gokhaneraslan]

There's a difference between fine-tuning for voice cloning (this repository is designed for that) and training a large multilingual model from scratch.
Multilingual models (like Chatterbox-Multilingual) require a large, balanced dataset containing all those languages ​​when training a model to speak multiple languages ​​from the ground up. Labels like [EN], [DE], [BE] are crucial because they teach the model to change its pronunciation, accent, and grammar according to the desired language. The model learns to be generalist.
With Fine-Tuning for Voice Cloning (This Repository), our goal is different. We take a strong, pre-trained model (Normal or Turbo) and tailor all its knowledge to specialize in a new voice and target language in your dataset. We don't teach it to switch languages; we refocus its entire "brain" to specialize in sounding like your reference speaker in your target language.
Since the entire dataset is in a single language (e.g., Belarusian), there's no need to tell the model which language to use. It needs to learn this implicitly from the data you provide.

And yes, the finely tuned model will master your target language using the voice you provide, but it won't forget English, of course. Especially in the Turbo model, while it might retain some weak "memories" from the original English training, its primary skill and best performance will be in the language and voice it's being fine-tuned in. It will become proficient in a language that will no longer be a reliable multilingual model.
Regarding your last question, this isn't the repository you're looking for. To do that, you need a balanced dataset across all target languages, not just one. Then you need to use language tags ([BE], [EN], etc.) in your metadata.csv file for all instances. Also, the training script and model architecture must be modified to understand and use these language IDs to condition the output. The answer you're looking for is LoRA. If you train the Chatterbox-Multilingual-TTS model with your specially prepared and labeled dataset, it won't lose any of its features and will learn the new 24th language. But you need to strike a good balance here, otherwise you'll break the model and lose quality in the other 23 languages ​​as well.

[tuteishygpt]

Thank you very much for the detailed answer.

I tried your code Setup.py
Why did the files pretrained_models/vocab.json and merges.txt remain unchanged?
I was tuning the XTTS V2 model and there the creation of these files was based on the text of the dataset. Check out this code
https://github.com/anhnh2002/XTTSv2-Finetuning-for-New-Languages/blob/main/extend_vocab_config.py

[gokhaneraslan]

The main difference lies in the architecture initialization.
XTTS V2 typically trains a tokenizer from scratch based on the dataset to cover specific languages/characters efficiently. However, Chatterbox Turbo is initialized from the GPT-2 Medium checkpoints.
To utilize the pre-trained knowledge/weights of GPT-2, we must preserve the original token mappings (indices). If we were to regenerate vocab.json and merges.txt based solely on the dataset (like in XTTS), we would break the alignment with the pre-trained weights, rendering the initialization useless.
Instead of replacing the tokenizer, my script extends the existing GPT-2 tokenizer. It loads the standard gpt2-medium tokenizer and uses .add_tokens() to inject the specific symbols/phonemes required for Chatterbox (from grapheme_mtl_merged_expanded_v1.json).
That is why the base merges.txt usually remains unchanged standard; the new custom tokens are typically handled via added_tokens.json or appended to the tokenizer.json structure, rather than altering the base BPE merge rules.

[sorenastar]

Hi @gokhaneraslan
First of all, thank you very much for this great repository and for sharing such a clean and well-structured fine-tuning pipeline. It’s been really helpful.

I’m currently working on fine-tuning the model specifically for Persian (Farsi), and I wanted to ask if you have any recommendations or best practices for this language.
For example:

tokenizer / vocabulary extension tips for Persian

dataset characteristics (text normalization, diacritics, punctuation, etc.)

any hyperparameter tweaks you’ve found useful for low-resource or morphologically rich languages

Any guidance would help us achieve the best possible configuration for Persian fine-tuning.
Thanks again for your awesome work!

[gokhaneraslan]

Hi there, and thank you for words! Persian (Farsi) is a so challenge due to its script and morphology.
The standard GPT-2 tokenizer handles Persian characters very inefficiently. You must extend the tokenizer. Collect all unique characters from your Persian dataset. You can adapt the token extension logic in my repository. Instead of replacing vocab.json, use the .add_tokens() method from the transformers library to inject Persian characters and save the new tokenizer.
Since these will be "new" tokens, their embeddings will be initialized randomly. The model will need the first few thousand steps just to learn what these characters "sound" like.
I recommend using a library like Hazm or Dadmatools for normalization. Unless your audio strictly follows them, remove all diacritics to prevent confusion. Map Persian punctuation to their Latin equivalents or add them explicitly to the tokenizer.
Since the model is learning a new language/script from scratch
effective Batch Size keep it around 32. Learning Rate stick to 5e-5. If the loss doesn't drop after the first few epochs, you might try a slight warmup, but 5e-5 is good.
Training Time unlike fine-tuning on English, the model first needs to learn the new embeddings. You might need more steps 20k - 30k steps compared to a standard English fine-tune.

[sorenastar]

Thanks a lot for the detailed reply — this was super helpful.

I’m already following your recommendations and digging deeper into your repo. Your work is something I genuinely follow, and it’s really motivating to see such a clean and thoughtful approach, especially for challenging languages like Persian.

Appreciate you taking the time to share your experience and insights. It definitely gives us a lot of motivation to keep going.

Thanks again and keep up the great work!

[dumanzhusupov]

@tuteishygpt @sorenastar Hello guys, could you share with results or experience after training on new language?
For now i have fine tuned orpheus on new language(Kazakh), and it's not bad. I have already tried to fine-tune orpheus on Kazakh language, and it's not bad. I think if we can share our results it will be usefull for all of us to get best results with best model.

[tuteishygpt]

@tuteishygpt @sorenastar Hello guys, could you share with results or experience after training on new language? For now i have fine tuned orpheus on new language(Kazakh), and it's not bad. I have already tried to fine-tune orpheus on Kazakh language, and it's not bad. I think if we can share our results it will be usefull for all of us to get best results with best model.

Hello.
I haven't tried it yet.
I don't think the results will be very good because gpt2-medium tokenizer doesn't have Cyrillic characters.

[sorenastar]

@tuteishygpt @sorenastar Hello guys, could you share with results or experience after training on new language? For now i have fine tuned orpheus on new language(Kazakh), and it's not bad. I have already tried to fine-tune orpheus on Kazakh language, and it's not bad. I think if we can share our results it will be usefull for all of us to get best results with best model.

Hello.
I haven't tried it yet, too.
I am currently preparing a high-quality dataset.