Where are the lemmatizer rules defined?

[joakimwar]

I found this file in spacy-lookups-data, which defines the rules for nouns in Norwegian as:

    "noun": [
        ["en", "e"],
        ["a", "e"],
        ["et", ""],
        ["er", "e"],
        ["ene", "e"]
    ],

But when I test the lemmatizer, it does not seem to follow these rules:

import spacy

nlp = spacy.load('nb_core_news_lg')
lemmatizer = nlp.get_pipe('lemmatizer')
ex = 'læret på sofaen er ødelagt'
doc = nlp(ex)
token = doc[0]
rule_lemma = lemmatizer.rule_lemmatize(token)
print(rule_lemma)
print(token.text, token.lemma_, token.pos, token.pos_)

Outputs:

['lære']
læret lære 92 NOUN

It recognises the word as a noun, but still it does not lemmatize it to "lær," which I would expect based on these rules. So the rules above don't seem to define how the lemmatizer works. So where are the rules for the lemmatizer defined? Or is this a bug, and the above rules are supposed to be applied?

And a slightly separate question: if I find mistakes in the lemmatizer, is there a way for me to fix them? E.g. make exceptions for certain words, or have separate lemmatizations for certain words when they are used as a verb vs when they are used as a noun. E.g. "lærer" in Norwegian can mean both "teacher" or "learns," and should be lemmatized to either "lærer" or "lære," depending on whether it was used as a noun or a verb. Even if the rules defined above did work, "lærer" would be lemmatized to "lære," even if it was used as a noun, which is incorrect.

[adrianeboyd]

The lemmatizer also includes a table of exceptions that has precedence over the rules:

print(lemmatizer.lookups.get_table("lemma_exc")["noun"]["læret"])
# ['lære']

You can customize any of the tables in lemmatizer.lookups.tables to change the lemmatizer behavior. If you save the model with nlp.to_disk(), your changes will be preserved.

lemmatizer.lookups.get_table("lemma_exc")["noun"]["læret"] = ["whatever"]

Be aware that there's a lemmatizer cache, so you'd might not see the changes until you save and reload the model, or manually wipe out the cache:

lemmatizer.cache = {}

If you want these changes in a new model you're training from scratch, you'd want to have a custom install of spacy-lookups-data with the edits or an alternate source of the lemmatizer tables specified in the lemmatizer initialization.

[joakimwar]

Thank you very much!