Different sampling rates

[EmreOzkose]

Hi!

Did you observe trainings with different sampling rates such as 8K->16K, 8K-> 22K, 16K->22K, etc.. ?
(diferent from demo page)

and what changes should we do to train with these data? (maybe hop length, n_fft, noise_schedule, pos_emb_scale, etc..)

[junjun3518]

Hi! Thank you for the great question!
We only tested for x2 and x3, but I think this work also available to apply on non-integer ratio upsampling.
You only need to change linear interpolation ratio and downsample method.
STFT is only needed for our in-phase downsampling, thus in this case, I recommend you to apply librosa.resample function instead of our downsampling functions in dataloader.

[EmreOzkose]

Thank you for quick reply :). I will try this experiment as soon as possible and report here.

[junjun3518]

Thank you for trying the additional experiment! Plz let me know if you need some help!

[EmreOzkose]

I did some experiments with 16k samples. I used 4h 16k data, and the default model is being training for 9 days. Up to now, everything is okey. I am sharing tensorboard log.

It seems that the model is able to boost the quality of the sound well qualitatively. I also want to observe difference between normal upsampling and neural upsampling. I have a 16k test set. I down-sampled this set for testing. I also have an acoustic model (which is trained in 16k audio). The results are here:

data	word error rate
16k original sounds	4.8
re-sampled with sox	7.0
re-sampled with neural-upsampling	8.3

So, neural upsampling is worse than sox :(.
When I listen up-sampled sound, I find some extra noise. How can achieve removing these unwanted noises? Do you think that I should change somewhere in adding noise part?

[junjun3518]

Interesting works!
I expected that you need to train more. Since denoising score matching process is trained by random Gaussian noise and complicated, we also got similar problems. We trained our model for two A100 or V100 GPUs over 2 weeks.

[junjun3518]

In addition, if you have an STT model you could apply conditional score generation suggested by Yang Song (https://arxiv.org/pdf/2011.13456.pdf Section 5 and Appendix I).

[EmreOzkose]

Thank you for advices. I think the problem is less training data and computational power. Training is continuing :). I am also going deeper and adapting the model to my case. I will report if the problem is solved.

Yang Song's work is also interesting. I will check out if I can apply. Thank you :).

[EmreOzkose]

Hi :).

I did some experiments on same dataset with different noise level. In paper, a different noise level is mentioned (noise_schedule: "torch.linspace(1e-4, 0.005, hparams.ddpm.max_step)") and I used that levels. After 6.8 days of training, there is some extra noises again like previous experiment. I will also did same evaluation (with sox result) and report here.

Do you think if that noise level is too much for 8k->16k? Or it is okey for that setup?

[junjun3518]

Hmm, I think you need to modify the inference schedule instead of the training schedule. Since 8 iteration's value is fit to our setup, maybe it could not be optimal for 8k->16k.

[EmreOzkose]

I think you are right, I didn't change inference part. I will check and report here. Thank you so much :).

[EmreOzkose]

I wanna share some more observations. There are spectrograms of a 16k test sound and upsampled sound (after downsampled to 8k).

Predicted area is so good. Here is upsampled version of the same sound with sox (after downsampled to 8k).

However, as we observed previously, some noise is added to 0khz-4khz part. Sound is qualified certainly, but my acoustic model performance is not better with neural-upsampled tests compared to sox-upsampled test. I think the primary reason is 0khz-4khz part.

In below figure, there are 3 waveform which are

1. downsampled example (same with above)
2. neural-upsampled version

and zoomed version

[junjun3518]

(Now I can see it)
In my opinion, neural upsampling and accuracy of AST/STT could be irrelevant OR your dataset is small to train general model.
If problem is a amount of dataset, I recommend to apply open dataset such as downsampled VCTK or LibriSpeech with your dataset.
For us, we similarly suffered high noise existence over 10kHz, but no low noises.

[junjun3518]

In addition, I am curious about your hyperparameters. Please let me know your batch or audio length or any difference between our hparameter.yaml file.

[EmreOzkose]

My hparameter.yaml :

train:
  batch_size: 6
  lr: 0.00003
  weight_decay: 0.00
  num_workers: 32
  gpus: 1 #ddp
  opt_eps: 1e-9
  beta1: 0.5
  beta2: 0.999

data:
  dir: "../neural_upsampling/data"
  format: '*.pt'
  cv_ratio: (1./2., 1./2., 0.00) #train/val/test

audio:
  sr: 16000
  nfft: 1024
  hop: 256
  ratio: 2 #upscale_ratio
  length: 32768 #32*1024 ~ 1sec

arch:
  residual_layers: 30 #
  residual_channels: 64
  dilation_cycle_length: 10
  pos_emb_dim: 512 

ddpm:
  max_step: 1000
  noise_schedule: "torch.linspace(1e-6, 0.006, hparams.ddpm.max_step)"
  pos_emb_scale: 50000
  pos_emb_channels: 128 
  infer_step: 8
  infer_schedule: "torch.tensor([1e-6,2e-6,1e-5,1e-4,1e-3,1e-2,1e-1,9e-1])"

log:
  name: 'nuwave_x2'
  checkpoint_dir: 'checkpoint'
  tensorboard_dir: 'tensorboard'
  test_result_dir: 'test_sample/results'

My GPU is Tesla P100-PCIE-16GB

[EmreOzkose]

I am increasing data now. I will start a training and report here as soon as possible.