M-AILABS Speech Dataset

[est31]

They have 237 hours of labeled German data from the Librivox project: http://www.m-ailabs.bayern/en/the-mailabs-speech-dataset/

[ynop]

Cool, thanks. I'll integrate that when I have time.

[aikekeaishenghuo]

They have 237 hours of labeled German data from the Librivox project: http://www.m-ailabs.bayern/en/the-mailabs-speech-dataset/

Why can't I visit?

[est31]

@aikekeaishenghuo for some reason, their enitre website is down. I guess that it'll be up again soon.

[est31]

https://www.caito.de/2019/01/the-m-ailabs-speech-dataset/

[AASHISHAG]

@aikekeaishenghuo @est31 : Did you create any script to use M-AILABS dataset for Mozilla DeepSpeech? If so, could you please share?

[ynop]

I haven't used it for deepspeech yet.
But I have created a library to read/writer audio datasets (https://github.com/ynop/audiomate).
There is a reader to load the dataset (https://audiomate.readthedocs.io/en/latest/reference/io.html#m-ailabs-speech-dataset).
You can just add it in the same way as the others

deepspeech-german/prepare_data.py

Lines 47 to 50 in 7d252e7

	if tuda_path is not None:
	tuda_corpus = audiomate.Corpus.load(tuda_path, reader='tuda')
	corpora.append(tuda_corpus)

Just change to reader = 'mailabs'.

[AASHISHAG]

@ynop : Thanks! It worked. I note that the library also supports Common Voice dataset. Could you please point out how to use it for Common Voice?

[ynop]

It basically works the same way. Just with reader='common-voice'.
Let me know which part is not clear.
Then I might can improve the documentation of the audiomate library.

[AASHISHAG]

@ynop : I tried using the library replicating the same steps for common-voice, but I am getting the below error.

Traceback (most recent call last):
  File "./prepare_data.py", line 68, in <module>
    cv_corpus = audiomate.Corpus.load(cv_path, reader='common-voice')
  File "/home/agarwal/python-environments/german-asr/lib/python3.5/site-packages/audiomate/corpus/corpus.py", line 122, in load
    return reader.load(path)
  File "/home/agarwal/python-environments/german-asr/lib/python3.5/site-packages/audiomate/corpus/io/base.py", line 81, in load
    return self._load(path)
  File "/home/agarwal/python-environments/german-asr/lib/python3.5/site-packages/audiomate/corpus/io/common_voice.py", line 69, in _load
    CommonVoiceReader.load_subset(corpus, path, subset_idx)
  File "/home/agarwal/python-environments/german-asr/lib/python3.5/site-packages/audiomate/corpus/io/common_voice.py", line 97, in load_subset
    age = CommonVoiceReader.map_age(entry[4])
IndexError: list index out of range

[ynop]

You used the most recent version of common voice?
And only the german part?

[AASHISHAG]

Yes, I used the most recent German Common Voice Corpus.

[AASHISHAG]

@ynop : I realized the corpus was not properly un-tarred. Now the code is running fine but the Mozilla Corpus is not getting appened in the test, train and dev files, while the other corpus are there. I have attached prepare_data.py. prepare_data.zip

Here is my Common Voice directory structure.

(env) agarwal:~/german-speech-corpus/mozilla/tmp$ ls
clips  de.tar.gz  dev.tsv  invalidated.tsv  other.tsv  test.tsv  train.tsv  validated.tsv

Command:
(env) agarwal:~/deepspeech-german$ ./prepare_data.py --cv ../german-speech-corpus/mozilla/tmp/ --voxforge ../german-speech-corpus/voxforge/ ../german-speech-corpus/testing

Also, when I just try to process Common Voice dataset, I get the below error.

(env) agarwal:~/deepspeech-german$ ./prepare_data.py --cv ../german-speech-corpus/mozilla/tmp/ ../german-speech-corpus/testing
Traceback (most recent call last):
  File "./prepare_data.py", line 80, in <module>
    merged_corpus.import_subview('train', splits['train'])
KeyError: 'train'

Could you please guide, if I missed anything or it's the issue with the library?

[ynop]

Found the problem. The latest release of audiomate does not contain the updated common-voice reader. You could just install audiomate from master branch for now. I will just check if everything is fine for a new release.

[ynop]

I created a new release (https://pypi.org/project/audiomate/4.0.0/).
Let me know if it works.