Training a lemmatizer on Universal Dependencies

[thiippal]

Hi!

I have trained a Transformer-based model for parsing and tagging Finnish using the excellent UD tagger/parser template.

Is there a way to train a lemmatizer on UD as well?

Having access to lemmas is crucial for our use case, namely geocoding.

[adrianeboyd]

No, there is not currently a trainable lemmatizer component. You could consider using stanza through spacy-stanza (will be updated for v3 shortly) or implement a custom lemmatizer that wraps another module like voikko? This should be relatively easy to do now with the --code arg to spacy train.

See the Russian pymorphy2 lemmatizer under spacy/lang/ru/lemmatizer.py for an example of a using a third-party module for lemmatization.

The new v3 docs still need to be expanded with more information about to extend the Lemmatizer, but the basics are to 1) load any modules as needed in __init__ and/or specify the required lookups tables in get_lookups_config and 2) implement a method that lemmatizes one token as the method {mode}_lemmatize, e.g. lookup_lemmatize or rule_lemmatize or pymorphy2_lemmatize.

You can obviously implement a separate custom lemmatizer class instead of extending Lemmatizer, but the idea was to make straightforward cases easier, so you don't need to worry about implementing __call__ or the serialization methods, just how to load resources and a method that lemmatizes one token.

[thiippal]

Thanks @adrianeboyd – I think I will look into implementing a custom lemmatizer that wraps a Universal Lemmatizer for Finnish (and perhaps for other languages as well).

@jmnybl would you be interested in collaborating on this?

[jmnybl]

@thiippal I will contact you by email.

[cayorodriguez]

Hi, I would be interested in incorporating this code into the catalan Spacy model. Can you share details on your implementation for Spacy?