Italian lemmatizer low performance on agglitinated verbs

[ferrixio]

Hi,

I recently used Spacy 3.4.4 to classify Italian verbs, but ran into the following problem using the pretrained model it_core_news_lg:

`
sentence = "aprimi la porta"
--- output ---

text	lemma	pos	tag
aprimi	aprimo	ADJ	A
la	il	DET	RD
porta	porta	NOUN	S

`
Sadly, the lemmatizer recognizes the verb "aprimi" as an adjective and in other cases it fails to recognize the right conjugation (I used "leggimi un libro" as sentence and Spacy said that "leggimi" comes from the verb "leggimare").
In general it seems that spacy has difficulty recognizing agglutinated verbs that involve pronouns. I tried to update Spacy to version 3.6.1, but the problem persists.

Is there any reason that explains it?

Many thanks!

Your Environment

• Operating System: Windows 10 and Windows 11
• Python Version Used: 3.9.6 and 3.11.1
• spaCy Version Used: 3.4.4 and 3.6.1

[adrianeboyd]

The provided it_core_news_* pipelines for v3.3+ use a statistical lemmatizer (https://spacy.io/api/edittreelemmatizer) that may generate unexpected forms, especially for the kinds of sentences that aren't in the training data (imperatives and questions are rare in most of our training data, text that looks like newspaper text will generally see better performance overall).

You can try switching from the trainable lemmatizer to the default lemmatizer as described here: https://spacy.io/models/#design-modify . This is a POS-lookup-based lemmatizer, so in cases where the POS is wrong the lemma will frequently also be wrong, but you can always easily see exactly which lemmas are provided by the lookup tables: https://github.com/explosion/spacy-lookups-data/tree/master/spacy_lookups_data/data

[adrianeboyd]

Let me convert this to a discussion...