Russian model proposal

[kuk]

cc @buriy

We trained high quality Russian model for SpaCy and would like to propose it for official SpaCy models registry.

• General description, links to evaluation tables https://github.com/natasha/natasha-spacy#natasha-spacy
• Links to training data, commands to reproduce our results https://github.com/natasha/natasha-spacy#training
• Trained model, usage example https://github.com/natasha/natasha-spacy#usage

All training sources are available under MIT license. We adapt datasets and embeddings table to fit SpaCy utilities. Training procedure uses standart spacy convert, spacy init-model, spacy train.

Could you please take a look at https://github.com/natasha/natasha-spacy? Is it possible to add such model to official SpaCy models registry? Is there anything we can do help you with this process? For example, we could contribute tests to spacy-models repo or main spacy repo.

[buriy]

@honnibal @adrianeboyd
Could you please take a look at this and suggest our further steps.

[honnibal]

This looks really cool! The results are certainly impressive. I do think we'll be able to integrate it as an official model, especially since it's MIT licensed.

The main requirement we have for distributing the models is that we need to add them to our automation systems, so that we can train the model ourselves. It looks like it'll be pretty easy to do that, the main annoyance being how bad the json format is in v2, which makes the training corpus hard to work with.

For spaCy v3, the model training process is now much simpler: we use the spacy project command to manage the various bits of preprocessing required, and use a config file to encapsulate all the settings. This will allow us to work from the source data as well, which is cool. We'll also be able to finally open-source the model training stuff, so that people can make direct pull-requests to add models.

The thing that would be most helpful from your side is to make sure it still works with spaCy v3, using the current spacy-nightly. The conversion now uses the DocBin binary format, so you might even be able to train on more text, without hitting the conversion problems.

Quick question about your evaluation: is the evaluation also on silver-standard data, or is the development and test data gold-standard?

[kuk]

The thing that would be most helpful from your side is to make sure it still works with spaCy v3

Sure thing, I will check it. How to reproduce spacy train --base-model ... from v2? We use pruned embeddings for initialization spacy init vectors ru data/navec.12B.300d.txt.gz train/vectors --prune 125000, then I do spacy train config.cfg --paths.vectors train/vectors, but final model uses embeddings trained from scratch, ignoring train/vectors.

The main requirement we have for distributing the models is that we need to add them to our automation systems, so that we can train the model ourselves. It looks like it'll be pretty easy to do that, the main annoyance being how bad the json format is in v2, which makes the training corpus hard to work with.

We use only standard SpaCy commands spacy convert, spacy init-model, spacy train so that it is easier for you to integrate our work into your pipeline. I can think of 2 things that might be problematic, both due to training corpus size:

1. spacy convert requires 32 GB of RAM
2. spacy train take 2 hours per iteration on CPU

Quick question about your evaluation: is the evaluation also on silver-standard data, or is the development and test data gold-standard?

Yep, evaluation is also silver-standard. But we also check the final model on a number of Russian gold-standart datasets, see morphology, syntax, and NER sections of our repo dedicated to public models evaluations.

[kuk]

The thing that would be most helpful from your side is to make sure it still works with spaCy v3

Added SpaCy v3 support using SpaCy Projects.

• config.cfg + project.yml https://github.com/natasha/natasha-spacy/tree/master/project
• v3 model + usage example https://github.com/natasha/natasha-spacy/blob/master/README.md#usage
• project run commands https://github.com/natasha/natasha-spacy/blob/master/README.md#v3
• package https://github.com/natasha/natasha-spacy/blob/master/README.md#v3-1

How to reproduce spacy train --base-model ... from v2?

I think the problem was in config. By default initialize.vectors is set to null, changed it to vectors = ${paths.vectors}.

[adrianeboyd]

The model size on disk is also a factor and for now, I'd like to keep the md models uniform.

Here's a preview of the sm model if you'd like to have a look: https://drive.google.com/file/d/1neef5lH_jmHJ_-s2q4wm4ww1ceEJsZt3/view?usp=sharing . Due to config changes, you'll need to install develop from source to load it. [removed after official release]

The configs are the same for the md and lg models except for the vectors. We train the components separately as much as possible, so we have separate tasks for: tok2vec+tagger+morphologizer+parser, senter, and ner. The senter models took ~1 hour, the ner models ~2 hours, and the longest one (md morphologizer+parser) took ~8 hours, which is all fine.

[buriy]

@adrianeboyd Thanks a lot. I'm mostly interested in comparing to a configuration with all components set to work properly and then copying or finding the best hyper-params.

[kuk]

I am planning
Split train into 10 chunks, each 10k docs. Replace train.conllu with train.conllu.tag.gz
Add SpaceAfter=No

Do I still need to add those features or training works fine on your side?

The model size on disk is also a factor and for now, I'd like to keep the md models uniform

Okey

Here's a preview of the sm model if you'd like to have a look

Looks good! Eval section in meta.json is even better then ours. How to access token morphology? Installed Spacy from development branch

>>> token.pos_, token.tag_

# spacy, spacy-nightly
NOUN, NOUN__Animacy=Anim|Case=Nom|Gender=Masc|Number=Sing
 
# development, ru_core_news_sm-3.0.0a1
NOUN, NOUN

[adrianeboyd]

We have everything set up on our end, so you don't need to modify anything for us. You might want to add SpaceAfter=No if you expect others to want to train their own spacy models from this data, though?

The morph information is under token.morph, see: https://nightly.spacy.io/api/morphology#morphanalysis

[kuk]

Ok, will keep dataset without changes for now. Currently you are the only consumer. Archived the repo https://github.com/natasha/natasha-spacy

[sdspieg]

This is GREAT news! Thanks to all involved indeed, and we can't wait to see it up there with the other languages...

[kuk]

Proposal got accepted in v3.0.0rc3 https://github.com/explosion/spaCy/releases/tag/v3.0.0rc3. See https://nightly.spacy.io/models/ru for official Russian model pretrained by Spacy team.