Incorrect lemmas for Italian language

[acazzaro]

Hello,

I think there might be a bug in the lemmatizer for Italian language. When I try to access the lemma_ of a NOUN, the result is the lemma of a VERB. Below my code:

title = "Gli studi sul Covid: creato un 'super-anticorpo' che blocca anche la variante indiana"
description = "Lo studio europeo sul nuovo monoclonale pubblicato su Nature"

nlp = spacy.load("it_core_news_md")
bag_of_words = []
doc1 = nlp(title.lower())
doc2 = nlp(description.lower())
docs = [doc1, doc2]

for doc in docs:
    for token in doc:
        if (token.pos_ == 'NOUN') or (token.pos_ == 'PROPN'):
            if token.is_stop == False:
                bag_of_words.append(token.lemma_)
                print(token.text)
bag_of_words

Result:
studi
anticorpo
variante
studio
monoclonale
nature
['studio', 'anticorpo', 'variare', 'studiare', 'monoclonale', 'natura']

As you can see both NOUNS 'variante' and 'studio' have their lemma transformed into a VERB 'variare' and 'studiare'. The plural of 'studio', 'studi' is lemmatized correctly into 'studio'. It seems the issue is only for singular NOUNS.

Thanks

My Environment:

• Operating System: macOS
• Python Version Used: 3.8.2
• spaCy Version Used: 3.0.5

[adrianeboyd]

Hi, the current Italian lemmatizer is very simple lookup lemmatizer with one big table for all words with no information about parts-of-speech, so in cases that are ambiguous, it just has one fixed lemma for each form, which may not correspond to the usage in the text.

A user has been working on an improved lemmatizer that uses better POS-based lookup tables, see: #7824