MMS-TTS

NOTE : The Repo is based following REPO

Table of contents

• 1. Description
• 2. Current Support Platform
• 3. Build Python Environment
• 4. Convert to RKNN
• 5. Python Demo
• 6. Android Demo
  • 6.1 Compile and Build
  • 6.2 Push demo files to device
  • 6.3 Run demo
• 7. Linux Demo
  • 7.1 Compile && Build
  • 7.2 Push demo files to device
  • 7.3 Run demo
• 8. Expected Results

1. Description

MMS-TTS (Massively Multilingual Speech, Text to Speech) is part of Facebook's Massively Multilingual Speech project, aiming to provide speech technology across a diverse range of languages.

The model used in this example comes from the following open source projects:

https://huggingface.co/facebook/mms-tts-eng

2. Current Support Platform

RK3566, RK3568, RK3588, RK3562, RK3576

3. Build Python Environment

git clone https://github.com/airockchip/rknn-toolkit2/
cd /rknn-toolkit2/packages/
# cd arm64  # If you are using ARM based system
cd x86_64 

I have test in Python version 3.10. Faced issue with version 3.11

conda create -n rknn python==3.10
conda activate rknn
pip install -r requirements_cp310-2.3.0.txt
pip install rknn_toolkit2-2.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

To create ONNX model read EXPORT_ONNX.md

python export_onnx.py --max_length 200

3. Pretrained Model

Note: There is issue with following Model. Need to create the models manualy

Download link:

mms_tts_eng_encoder_200.onnx
mms_tts_eng_decoder_200.onnx

Download with shell command:

cd model
./download_model.sh

Note: For exporting mms_tts onnx models, please refer to export_onnx.md

4. Convert to RKNN

Usage:

cd python
python convert.py <onnx_model> <TARGET_PLATFORM> <dtype(optional)> <output_rknn_path(optional)>

# such as: 
python convert.py ../model/mms_tts_eng_encoder_200.onnx rk3588
# output model will be saved as ../model/mms_tts_eng_encoder_200.rknn

python convert.py ../model/mms_tts_eng_decoder_200.onnx rk3588
# output model will be saved as ../model/mms_tts_eng_decoder_200.rknn

Description:

• <onnx_model>: Specify ONNX model path.
• <TARGET_PLATFORM>: Specify NPU platform name. Support Platform refer here.
• <dtype>(optional): Specify as i8 or fp. i8 for doing quantization, fp for no quantization. Default is fp.
• <output_rknn_path>(optional): Specify save path for the RKNN model, default save in the same directory as ONNX model.

5. Python Demo

Usage:

cd python
# Inference with ONNX model
python mms_tts.py --encoder_model_path <onnx_model> --decoder_model_path <onnx_model>

# Inference with RKNN model
python mms_tts.py --encoder_model_path <rknn_model> --decoder_model_path <rknn_model> --target <TARGET_PLATFORM>

python mms_tts.py --encoder_model_path ../model/mms_tts_eng_encoder_17_200.rknn --decoder_model_path ../model/mms_tts_eng_decoder_17_200.onnx

Description:

• <TARGET_PLATFORM>: Specify NPU platform name. Support Platform refer here.
• <onnx_model / rknn_model>: Specify model path.

6. Android Demo

6.1 Compile and Build

Usage:

# go back to the rknn_model_zoo root directory
cd ../../
export ANDROID_NDK_PATH=<android_ndk_path>

./build-android.sh -t <TARGET_PLATFORM> -a <ARCH> -d mms_tts

# such as 
./build-android.sh -t rk3588 -a arm64-v8a -d mms_tts

Description:

• <android_ndk_path>: Specify Android NDK path.
• <TARGET_PLATFORM>: Specify NPU platform name. Support Platform refer here.
• <ARCH>: Specify device system architecture. To query device architecture, refer to the following command:

# Query architecture. For Android, ['arm64-v8a' or 'armeabi-v7a'] should shown in log.
adb shell cat /proc/version

6.2 Push demo files to device

With device connected via USB port, push demo files to devices:

adb root
adb remount
adb push install/<TARGET_PLATFORM>_android_<ARCH>/rknn_mms_tts_demo/ /data/

6.3 Run demo

adb shell
cd /data/rknn_mms_tts_demo

export LD_LIBRARY_PATH=./lib
./rknn_mms_tts_demo model/mms_tts_eng_encoder_200.rknn model/mms_tts_eng_decoder_200.rknn '"Mister quilter is the apostle of the middle classes and we are glad to welcome his gospel."'

• After running, the result was saved as output.wav. To check the result on host PC, pull back result referring to the following command:

  adb pull /data/rknn_mms_tts_demo/output.wav

7. Linux Demo

7.1 Compile and Build

usage

# go back to the rknn_model_zoo root directory
cd ../../

# if GCC_COMPILER not found while building, please set GCC_COMPILER path
(optional)export GCC_COMPILER=<GCC_COMPILER_PATH>

./build-linux.sh -t <TARGET_PLATFORM> -a <ARCH> -d mms_tts

# such as 
./build-linux.sh -t rk3588 -a aarch64 -d mms_tts

Description:

• <GCC_COMPILER_PATH>: Specified as GCC_COMPILER path.

• <TARGET_PLATFORM> : Specify NPU platform name. Support Platform refer here.

• <ARCH>: Specify device system architecture. To query device architecture, refer to the following command:

  # Query architecture. For Linux, ['aarch64' or 'armhf'] should shown in log.
  adb shell cat /proc/version

7.2 Push demo files to device

• If device connected via USB port, push demo files to devices:

adb push install/<TARGET_PLATFORM>_linux_<ARCH>/rknn_mms_tts_demo/ /data/

• For other boards, use scp or other approaches to push all files under install/<TARGET_PLATFORM>_linux_<ARCH>/rknn_mms_tts_demo/ to data.

7.3 Run demo

adb shell
cd /data/rknn_mms_tts_demo

export LD_LIBRARY_PATH=./lib
./rknn_mms_tts_demo model/mms_tts_eng_encoder_200.rknn model/mms_tts_eng_decoder_200.rknn '"Mister quilter is the apostle of the middle classes and we are glad to welcome his gospel."'

• After running, the result was saved as output.wav. To check the result on host PC, pull back result referring to the following command:

  adb pull /data/rknn_mms_tts_demo/output.wav
  

8. Expected Results

This example will generate a wav file corresponding to the input text: "Mister quilter is the apostle of the middle classes and we are glad to welcome his gospel.", such as:

output.wav

• Note: Different platforms, different versions of tools and drivers may have slightly different results.