Skip to content

Commit 6dbe108

Browse files
ZDisketZDisket
committed
Initial commit
1 parent fca393a commit 6dbe108

File tree

7 files changed

+940
-0
lines changed

7 files changed

+940
-0
lines changed

‎examples/multiband_pwgan/README.md‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
# Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech (With ParallelWaveGAN discriminator)
2+
Based on the script [`train_multiband_pwgan.py`](https://github.com/dathudeptrai/TensorflowTTS/tree/master/examples/multiband_pwgan/train_multiband_pwgan.py).
3+
4+
## Training Multi-band MelGAN from scratch with LJSpeech dataset.
5+
This example code show you how to train MelGAN from scratch with Tensorflow 2 based on custom training loop and tf.function. The data used for this example is LJSpeech, you can download the dataset at [link](https://keithito.com/LJ-Speech-Dataset/).
6+
7+
### Step 1: Create Tensorflow based Dataloader (tf.dataset)
8+
Please see detail at [examples/melgan/](https://github.com/dathudeptrai/TensorflowTTS/tree/master/examples/melgan#step-1-create-tensorflow-based-dataloader-tfdataset)
9+
10+
### Step 2: Training from scratch
11+
After you re-define your dataloader, pls modify an input arguments, train_dataset and valid_dataset from [`train_multiband_pwgan.py`](https://github.com/dathudeptrai/TensorflowTTS/tree/master/examples/multiband_pwgan/train_multiband_pwgan.py). Here is an example command line to training melgan-stft from scratch:
12+
13+
First, you need training generator with only stft loss:
14+
15+
```bash
16+
CUDA_VISIBLE_DEVICES=0 python examples/multiband_pwgan/train_multiband_pwgan.py \
17+
--train-dir ./dump/train/ \
18+
--dev-dir ./dump/valid/ \
19+
--outdir ./examples/multiband_pwgan/exp/train.multiband_melgan.v1/ \
20+
--config ./examples/multiband_pwgan/conf/multiband_pwgan.v1.yaml \
21+
--use-norm 1 \
22+
--generator_mixed_precision 1 \
23+
--resume ""
24+
```
25+
26+
Then resume and start training generator + discriminator:
27+
28+
```bash
29+
CUDA_VISIBLE_DEVICES=0 python examples/multiband_pwgan/train_multiband_pwgan.py \
30+
--train-dir ./dump/train/ \
31+
--dev-dir ./dump/valid/ \
32+
--outdir ./examples/multiband_pwgan/exp/train.multiband_melgan.v1/ \
33+
--config ./examples/multiband_pwgan/conf/multiband_pwgan.v1.yaml \
34+
--use-norm 1 \
35+
--resume ./examples/multiband_pwgan/exp/train.multiband_melgan.v1/checkpoints/ckpt-200000
36+
```
37+
38+
IF you want to use MultiGPU to training you can replace `CUDA_VISIBLE_DEVICES=0` by `CUDA_VISIBLE_DEVICES=0,1,2,3` for example. You also need to tune the `batch_size` for each GPU (in config file) by yourself to maximize the performance. Note that MultiGPU now support for Training but not yet support for Decode.
39+
40+
In case you want to resume the training progress, please following below example command line:
41+
42+
```bash
43+
--resume ./examples/multiband_pwgan/exp/train.multiband_melgan.v1/checkpoints/ckpt-100000
44+
```
45+
46+
**IMPORTANT NOTES**:
47+
48+
- If Your Dataset is 16K, upsample_scales = [2, 4, 8] worked.
49+
- If Your Dataset is > 16K (22K, 24K, ...), upsample_scales = [2, 4, 8] didn't worked, used [8, 4, 2] instead.
50+
51+
### Step 3: Decode audio from folder mel-spectrogram
52+
To running inference on folder mel-spectrogram (eg valid folder), run below command line:
53+
54+
```bash
55+
CUDA_VISIBLE_DEVICES=0 python examples/multiband_pwgan/decode_mb_melgan.py \
56+
--rootdir ./dump/valid/ \
57+
--outdir ./prediction/multiband_melgan.v1/ \
58+
--checkpoint ./examples/multiband_pwgan/exp/train.multiband_melgan.v1/checkpoints/generator-940000.h5 \
59+
--config ./examples/multiband_pwgan/conf/multiband_pwgan.v1.yaml \
60+
--batch-size 32 \
61+
--use-norm 1
62+
```
63+
64+
## Finetune MelGAN STFT with ljspeech pretrained on other languages
65+
Just load pretrained model and training from scratch with other languages. **DO NOT FORGET** re-preprocessing on your dataset if needed. A hop_size should be 256 if you want to use our pretrained.
66+
67+
## Learning Curves
68+
Here is a learning curves of melgan based on this config [`multiband_pwgan.v1.yaml`](https://github.com/dathudeptrai/TensorflowTTS/tree/master/examples/multiband_pwgan/conf/multiband_pwgan.v1.yaml)
69+
70+
<img src="fig/eval.png" height="300" width="850">
71+
72+
<img src="fig/train.png" height="300" width="850">
73+
74+
## Pretrained Models and Audio samples
75+
| Model | Conf | Lang | Fs [Hz] | Mel range [Hz] | FFT / Hop / Win [pt] | # iters |
76+
| :------ | :---: | :---: | :----: | :--------: | :---------------: | :-----: |
77+
| [multiband_melgan.v1](https://drive.google.com/drive/folders/1Hg82YnPbX6dfF7DxVs4c96RBaiFbh-cT?usp=sharing) | [link](https://github.com/tensorspeech/TensorFlowTTS/tree/master/examples/multiband_pwgan/conf/multiband_pwgan.v1.yaml) | EN | 22.05k | 80-7600 | 1024 / 256 / None | 940K |
78+
| [multiband_melgan.v1](https://drive.google.com/drive/folders/199XCXER51PWf_VzUpOwxfY_8XDfeXuZl?usp=sharing) | [link](https://github.com/dathudeptrai/TensorflowTTS/tree/master/examples/multiband_pwgan/conf/multiband_pwgan.v1.yaml) | KO | 22.05k | 80-7600 | 1024 / 256 / None | 1000K |
79+
80+
## Reference
81+
82+
1. https://github.com/kan-bayashi/ParallelWaveGAN
83+
2. [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://arxiv.org/abs/1910.11480)
84+
3. [Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech](https://arxiv.org/abs/2005.05106)

‎examples/multiband_pwgan/conf/multiband_pwgan.v1.yaml‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
2+
# This is the hyperparameter configuration file for Multi-Band MelGAN with PWGAN discriminator.
3+
# Please make sure this is adjusted for the LJSpeech dataset. If you want to
4+
# apply to the other dataset, you might need to carefully change some parameters.
5+
# This configuration performs 1000k iters.
6+
7+
###########################################################
8+
# FEATURE EXTRACTION SETTING #
9+
###########################################################
10+
sampling_rate: 22050
11+
hop_size: 256 # Hop size.
12+
format: "npy"
13+
14+
15+
###########################################################
16+
# GENERATOR NETWORK ARCHITECTURE SETTING #
17+
###########################################################
18+
model_type: "multiband_melgan_generator"
19+
20+
multiband_melgan_generator_params:
21+
out_channels: 4 # Number of output channels (number of subbands).
22+
kernel_size: 7 # Kernel size of initial and final conv layers.
23+
filters: 384 # Initial number of channels for conv layers.
24+
upsample_scales: [8, 4, 2] # List of Upsampling scales.
25+
stack_kernel_size: 3 # Kernel size of dilated conv layers in residual stack.
26+
stacks: 4 # Number of stacks in a single residual stack module.
27+
is_weight_norm: false # Use weight-norm or not.
28+
29+
###########################################################
30+
# DISCRIMINATOR NETWORK ARCHITECTURE SETTING #
31+
###########################################################
32+
parallel_wavegan_discriminator_params:
33+
out_channels: 1 # Number of output channels.
34+
kernel_size: 3 # Number of output channels.
35+
n_layers: 10 # Number of conv layers.
36+
conv_channels: 64 # Number of chnn layers.
37+
use_bias: true # Whether to use bias parameter in conv.
38+
nonlinear_activation: "LeakyReLU" # Nonlinear function after each conv.
39+
nonlinear_activation_params: # Nonlinear function parameters
40+
alpha: 0.2 # Alpha in LeakyReLU.
41+
42+
###########################################################
43+
# STFT LOSS SETTING #
44+
###########################################################
45+
stft_loss_params:
46+
fft_lengths: [1024, 2048, 512] # List of FFT size for STFT-based loss.
47+
frame_steps: [120, 240, 50] # List of hop size for STFT-based loss
48+
frame_lengths: [600, 1200, 240] # List of window length for STFT-based loss.
49+
50+
subband_stft_loss_params:
51+
fft_lengths: [384, 683, 171] # List of FFT size for STFT-based loss.
52+
frame_steps: [30, 60, 10] # List of hop size for STFT-based loss
53+
frame_lengths: [150, 300, 60] # List of window length for STFT-based loss.
54+
55+
###########################################################
56+
# ADVERSARIAL LOSS SETTING #
57+
###########################################################
58+
lambda_feat_match: 10.0 # Loss balancing coefficient for feature matching loss
59+
lambda_adv: 2.5 # Loss balancing coefficient for adversarial loss.
60+
61+
###########################################################
62+
# DATA LOADER SETTING #
63+
###########################################################
64+
batch_size: 64 # Batch size.
65+
batch_max_steps: 8192 # Length of each audio in batch for training. Make sure dividable by hop_size.
66+
batch_max_steps_valid: 81920 # Length of each audio for validation. Make sure dividable by hope_size.
67+
remove_short_samples: true # Whether to remove samples the length of which are less than batch_max_steps.
68+
allow_cache: true # Whether to allow cache in dataset. If true, it requires cpu memory.
69+
is_shuffle: true # shuffle dataset after each epoch.
70+
71+
###########################################################
72+
# OPTIMIZER & SCHEDULER SETTING #
73+
###########################################################
74+
generator_optimizer_params:
75+
lr_fn: "PiecewiseConstantDecay"
76+
lr_params:
77+
boundaries: [100000, 200000, 300000, 400000, 500000, 600000, 700000]
78+
values: [0.0005, 0.0005, 0.00025, 0.000125, 0.0000625, 0.00003125, 0.000015625, 0.000001]
79+
amsgrad: false
80+
81+
discriminator_optimizer_params:
82+
lr_fn: "PiecewiseConstantDecay"
83+
lr_params:
84+
boundaries: [100000, 200000, 300000, 400000, 500000]
85+
values: [0.00025, 0.000125, 0.0000625, 0.00003125, 0.000015625, 0.000001]
86+
amsgrad: false
87+
88+
###########################################################
89+
# INTERVAL SETTING #
90+
###########################################################
91+
discriminator_train_start_steps: 200000 # steps begin training discriminator
92+
train_max_steps: 4000000 # Number of training steps.
93+
save_interval_steps: 20000 # Interval steps to save checkpoint.
94+
eval_interval_steps: 5000 # Interval steps to evaluate the network.
95+
log_interval_steps: 200 # Interval steps to record the training log.
96+
97+
###########################################################
98+
# OTHER SETTING #
99+
###########################################################
100+
num_save_intermediate_results: 1 # Number of batch to be saved as intermediate results.

‎examples/multiband_pwgan/conf/multiband_pwgan.v1ft.yaml‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
2+
# This is the hyperparameter configuration file for Multi-Band MelGAN with PWGAN discriminator.
3+
# This one is adjusted for finetuning
4+
5+
###########################################################
6+
# FEATURE EXTRACTION SETTING #
7+
###########################################################
8+
sampling_rate: 22050
9+
hop_size: 256 # Hop size.
10+
format: "npy"
11+
12+
13+
###########################################################
14+
# GENERATOR NETWORK ARCHITECTURE SETTING #
15+
###########################################################
16+
model_type: "multiband_melgan_generator"
17+
18+
multiband_melgan_generator_params:
19+
out_channels: 4 # Number of output channels (number of subbands).
20+
kernel_size: 7 # Kernel size of initial and final conv layers.
21+
filters: 384 # Initial number of channels for conv layers.
22+
upsample_scales: [8, 4, 2] # List of Upsampling scales.
23+
stack_kernel_size: 3 # Kernel size of dilated conv layers in residual stack.
24+
stacks: 4 # Number of stacks in a single residual stack module.
25+
is_weight_norm: false # Use weight-norm or not.
26+
27+
###########################################################
28+
# DISCRIMINATOR NETWORK ARCHITECTURE SETTING #
29+
###########################################################
30+
parallel_wavegan_discriminator_params:
31+
out_channels: 1 # Number of output channels.
32+
kernel_size: 3 # Number of output channels.
33+
n_layers: 10 # Number of conv layers.
34+
conv_channels: 64 # Number of chnn layers.
35+
use_bias: true # Whether to use bias parameter in conv.
36+
nonlinear_activation: "LeakyReLU" # Nonlinear function after each conv.
37+
nonlinear_activation_params: # Nonlinear function parameters
38+
alpha: 0.2 # Alpha in LeakyReLU.
39+
40+
###########################################################
41+
# STFT LOSS SETTING #
42+
###########################################################
43+
stft_loss_params:
44+
fft_lengths: [1024, 2048, 512] # List of FFT size for STFT-based loss.
45+
frame_steps: [120, 240, 50] # List of hop size for STFT-based loss
46+
frame_lengths: [600, 1200, 240] # List of window length for STFT-based loss.
47+
48+
subband_stft_loss_params:
49+
fft_lengths: [384, 683, 171] # List of FFT size for STFT-based loss.
50+
frame_steps: [30, 60, 10] # List of hop size for STFT-based loss
51+
frame_lengths: [150, 300, 60] # List of window length for STFT-based loss.
52+
53+
###########################################################
54+
# ADVERSARIAL LOSS SETTING #
55+
###########################################################
56+
lambda_feat_match: 10.0 # Loss balancing coefficient for feature matching loss
57+
lambda_adv: 2.5 # Loss balancing coefficient for adversarial loss.
58+
59+
###########################################################
60+
# DATA LOADER SETTING #
61+
###########################################################
62+
batch_size: 64 # Batch size.
63+
batch_max_steps: 8192 # Length of each audio in batch for training. Make sure dividable by hop_size.
64+
batch_max_steps_valid: 81920 # Length of each audio for validation. Make sure dividable by hope_size.
65+
remove_short_samples: true # Whether to remove samples the length of which are less than batch_max_steps.
66+
allow_cache: true # Whether to allow cache in dataset. If true, it requires cpu memory.
67+
is_shuffle: true # shuffle dataset after each epoch.
68+
69+
###########################################################
70+
# OPTIMIZER & SCHEDULER SETTING #
71+
###########################################################
72+
generator_optimizer_params:
73+
lr_fn: "PiecewiseConstantDecay"
74+
lr_params:
75+
boundaries: [100000, 200000, 300000, 400000, 500000, 600000, 700000]
76+
values: [0.0005, 0.0005, 0.00025, 0.000125, 0.0000625, 0.00003125, 0.000015625, 0.000001]
77+
amsgrad: false
78+
79+
discriminator_optimizer_params:
80+
lr_fn: "PiecewiseConstantDecay"
81+
lr_params:
82+
boundaries: [100000, 200000, 300000, 400000, 500000]
83+
values: [0.00025, 0.000125, 0.0000625, 0.00003125, 0.000015625, 0.000001]
84+
amsgrad: false
85+
86+
###########################################################
87+
# INTERVAL SETTING #
88+
###########################################################
89+
discriminator_train_start_steps: 0 # steps begin training discriminator
90+
train_max_steps: 200000 # Number of training steps.
91+
save_interval_steps: 5000 # Interval steps to save checkpoint.
92+
eval_interval_steps: 1000 # Interval steps to evaluate the network.
93+
log_interval_steps: 200 # Interval steps to record the training log.
94+
95+
###########################################################
96+
# OTHER SETTING #
97+
###########################################################
98+
num_save_intermediate_results: 1 # Number of batch to be saved as intermediate results.

0 commit comments

Comments
 (0)