Numerical divergence for fbank feats at certain number of bins

[FredSRichardson]

I found that the kaldifeat filter bank data starts to diverge significantly from Kaldi above a certain number of bins. For 16KHz data, this appears to be at 85 bin while for 8KHz data, this appears to be at 53 bins. I'm afraid I can't help much in terms of figuring out why there is this divergence.

To replicate this issue, duplicate the test code test_fbank_40_bins() in python/tests/test_fbank.py and and the reference generation code for test-40.txt in python/tests/test_data/run.sh replacing 40 with 86 (for the file names, command line parameter --num-mel-bins and the runtime option opts.mel_opts.num_bins). When I did this, the test failed at the assert torch.allclose(features, gt, rtol=1e-1) step.

[csukuangfj]

Plrase show the actual computed features and show the.abs().max() of the difference.

[FredSRichardson]

Plrase show the actual computed features and show the.abs().max() of the difference.

I promise to get back to you on that soon!

[FredSRichardson]

So the good news is that there is nothing to fix in kaldifeat. What does need a very minor adjusting is the regression test scripting.

In kaldifeat/python/tests/test_data/run.sh, when you generate audio using:
sox -n -r 16000 -b 16 test.wav synth 1.2 sine 300-3300
The output is dithered each time leading to different numerical data. To fix this, add the -R (reproduceable) option:
sox -R -n -r 16000 -b 16 test.wav synth 1.2 sine 300-3300
This actually stumped me as the numerical comparisons were different each time I ran my script even with dither=0!

One other thing is that I think you want to use that atol (absolute) instead of the rtol (relative) tolerance parameter in your calls to torch.allclose (that is, use torch.allclose(features, gt, atol=1e-1) instead of torch.allclose(features, gt, rtol=1e-1)). I found all the tests pass with this change!

Now for a table of comparisons (that are reproduceable with the -R option). The differences are on average below 1e-3 which is great and the max difference is below 1e-1. Finally, they all pass using the atol param:

device: cpu, sample_rate: 16000.0, num_bins: 30, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000228, abs_max_diff: 0.02429
device: cpu, sample_rate: 16000.0, num_bins: 40, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000272, abs_max_diff: 0.02386
device: cpu, sample_rate: 16000.0, num_bins: 50, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000314, abs_max_diff: 0.03506
device: cpu, sample_rate: 16000.0, num_bins: 60, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000340, abs_max_diff: 0.05302
device: cpu, sample_rate: 16000.0, num_bins: 70, is_close(rtol): False, is_close(atol): True, abs_mean_diff: 0.000383, abs_max_diff: 0.05794
device: cpu, sample_rate: 16000.0, num_bins: 80, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000428, abs_max_diff: 0.05892
device: cpu, sample_rate: 16000.0, num_bins: 40, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000272, abs_max_diff: 0.02386
device: cpu, sample_rate: 16000.0, num_bins: 41, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000277, abs_max_diff: 0.02980
device: cpu, sample_rate: 16000.0, num_bins: 42, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000281, abs_max_diff: 0.02964
device: cpu, sample_rate: 16000.0, num_bins: 43, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000285, abs_max_diff: 0.02903
device: cpu, sample_rate: 16000.0, num_bins: 44, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000287, abs_max_diff: 0.03408
device: cpu, sample_rate: 16000.0, num_bins: 45, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000291, abs_max_diff: 0.03670
device: cpu, sample_rate: 16000.0, num_bins: 46, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000296, abs_max_diff: 0.03666
device: cpu, sample_rate: 16000.0, num_bins: 47, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000298, abs_max_diff: 0.03437
device: cpu, sample_rate: 16000.0, num_bins: 48, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000302, abs_max_diff: 0.03468
device: cpu, sample_rate: 16000.0, num_bins: 49, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000304, abs_max_diff: 0.03035
device: cpu, sample_rate: 16000.0, num_bins: 50, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000314, abs_max_diff: 0.03506
device: cpu, sample_rate: 16000.0, num_bins: 80, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000428, abs_max_diff: 0.05892
device: cpu, sample_rate: 16000.0, num_bins: 81, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000428, abs_max_diff: 0.07075
device: cpu, sample_rate: 16000.0, num_bins: 82, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000429, abs_max_diff: 0.08063
device: cpu, sample_rate: 16000.0, num_bins: 83, is_close(rtol): False, is_close(atol): True, abs_mean_diff: 0.000418, abs_max_diff: 0.08446
device: cpu, sample_rate: 16000.0, num_bins: 84, is_close(rtol): False, is_close(atol): True, abs_mean_diff: 0.000417, abs_max_diff: 0.07716
device: cpu, sample_rate: 16000.0, num_bins: 85, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000426, abs_max_diff: 0.07631
device: cpu, sample_rate: 16000.0, num_bins: 86, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000441, abs_max_diff: 0.06969
device: cpu, sample_rate: 16000.0, num_bins: 87, is_close(rtol): False, is_close(atol): True, abs_mean_diff: 0.000431, abs_max_diff: 0.06127
device: cpu, sample_rate: 16000.0, num_bins: 88, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000428, abs_max_diff: 0.07038
device: cpu, sample_rate: 16000.0, num_bins: 89, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000439, abs_max_diff: 0.08352
device: cpu, sample_rate: 16000.0, num_bins: 90, is_close(rtol): True, is_close(atol): True, abs_mean_diff: 0.000438, abs_max_diff: 0.09251

[csukuangfj]

Thank you for your detailed tests.

Can you make a PR fo.fix that?

[FredSRichardson]

I'm afraid I can't make direct contributions due to my current company policies. I can only give you detailed notes on what would need to change.