Datasets used for the French pretrained pipelines

[michelemarzollo]

Hello,

I have some questions regarding the French pretrained pipelines:

• Is there a reason for which you used the UD-Sequoia treebank instead of the UD-GSD treebank, which is a larger dataset and is offered with a less restrictive license?
• Is there a reason for which the transformer pipeline does not have a NER component?
• Did the tok2vec and the transformer component also use the UD-Sequoia treebank for training?

Thank you for your attention!

[adrianeboyd]

I haven't looked at the most recent version of the French GSD treebank in detail, but the quality of the UD GSD treebanks is often poor and until recently they were licensed under CC NC, which we generally try to avoid for pipelines that we distribute. There are a few exceptions (Greek and Italian, for historical reasons), but we strongly prefer datasets that can be used for commercial purposes.

We only have NER components in transformer models when there is NER annotation for the exact same data as the treebank data, since training a single transformer from partial annotation does not work well in practice.

The tok2vec and transformer components are used by the morphologizer and parser, so they are also trained with the UD French Sequoia data.

If you want to train models on UD French GSD instead, this project should be a good starting point: https://github.com/explosion/projects/tree/v3/pipelines/tagger_parser_ud. If you want to use vectors, update spacy and generate an updated config with spacy init config to include some bug fixes for vectors in the tok2vec config.

[michelemarzollo]

Thank you very much! That's exactly the information I was looking for!