Various wrong lemmas

[yolpsoftware]

Describe the bug
Evaluating "Ich wasche meine Hände." in Stanza 1.11 leads to "Hände" being treated as a verb with lemma=hinden. There is no verb "hinden" in German, and Hände is a noun meaning "hands".

To Reproduce
Steps to reproduce the behavior:
Install python 3.10 and Stanza 1.11.0

import stanza
stanza.download('de')
nlp = stanza.Pipeline('de')
nlp('Ich wasche meine Hände.')

Expected behavior
"Hände" should be a noun with lemma=hand.

Environment (please complete the following information):

• OS: Ubuntu 22.04 running on Windows 10 WSL.
• Python version: 3.10 preinstalled with Ubuntu 22.04
• Stanza version: 1.11.0

[AngledLuffa]

That would be surprising, given that Hände shows up many times in the training data.

However, I do not get this result:

>>> import stanza
>>> nlp = stanza.Pipeline("de")
>>> print("{:C}".format(nlp('Ich wasche meine Hände.')))
# text = Ich wasche meine Hände.
# sent_id = 0
# constituency = (ROOT (VROOT (S (PRON Ich) (VERB wasche) (NP (DET meine) (NOUN Hände))) (PUNCT .)))
# sentiment = 1
1       Ich     ich     PRON    PPER    Case=Nom|Number=Sing|Person=1|PronType=Prs      2       nsubj   _       start_char=0|end_char=3|ner=O
2       wasche  waschen VERB    VVFIN   Mood=Ind|Number=Sing|Person=1|Tense=Pres|VerbForm=Fin   0       root    _       start_char=4|end_char=10|ner=O
3       meine   mein    DET     PPOSAT  Case=Acc|Gender=Fem|Number=Plur|Number[psor]=Sing|Person=1|Poss=Yes|PronType=Prs        4       det:poss        _      start_char=11|end_char=16|ner=O
4       Hände   Hand    NOUN    NN      Case=Acc|Gender=Fem|Number=Plur 2       obj     _       SpaceAfter=No|start_char=17|end_char=22|ner=O
5       .       .       PUNCT   $.      _       2       punct   _       SpaceAfter=No|start_char=22|end_char=23|ner=O

[yolpsoftware]

You are of course right. My result was a cached one from Stanza 1.10. Great that this was fixed with 1.11 🥳

[AngledLuffa]

If you encounter any others, we can add individual errors to the training data and hopefully improve the overall quality.

Also, the transformer based models (default_accurate) is significantly better overall.

Lastly, if you don't need or want constituency results, you may find it faster without any downsides to eliminate it from the list of processors used.

[yolpsoftware]

@AngledLuffa Here are a few errors (wrong lemmas, mostly words that do not even exist):

# lemma of "galline" should be "gallina", not "galline"
stanza.Pipeline('it')('Le galline beccano i chicchi.')

# lemma of "birilli" should be "birillo", not "birille"
stanza.Pipeline('it')('Lancia la palla verso i birilli.')

# lemma of "termiti" should be "termite", not "termito"
stanza.Pipeline('it')('Le termiti possono distruggere il legno.')

# lemma of "Nägel" should be "Nagel", not "Nägeln"
stanza.Pipeline('de')('Sie schneidet sich die Nägel.')

If you want more, I can find more.

The fact that the three last lemmas do not even exist as a word could maybe be used for training: feed the network lots of text from some text bank and penalize it for producing non-existing lemmas. That way, any text source in a language together with a list of existing words can be turned into a training source.

Of course, such training would not be useful on its own, but maybe in a balanced combination with the other, more specialized training data, it could be an improvement.

[yolpsoftware]

Another training option to fix stuff like that: LLMs like GPT-5 or Claude are maybe not as good as Stanza in part-of-speech tagging, but they can detect a wrong lemma or other problems quite well. You could feed thousands of sentences to Stanza and let an LLM filter the output to get the wrong ones (and tag them, e.g. "lemma should be termite instead of termito". Those would then need to be checked and fixed manually and can then go to the training data.

I know, this takes time, but it's most likely the faster method to increase the quality than me reporting those cases (there are still quite a few).

[yolpsoftware]

# lemma of "sardine" should be "sardina", not "sardine"
stanza.Pipeline('it')('Ho comprato una lattina di sardine per pranzo.')
# lemma of "vibre" should be "vibrer", not "vibre"
stanza.Pipeline('fr')('Le portable vibre dans la poche.')

# different case: here, "schaue nach" is a conjugation of the verb "nachschauen", so I think this should be a MWT with lemma "nachschauen" - however, Stanza does not detect that those two words belong together
stanza.Pipeline('de')('Ich schaue in meinem Kalender nach.')
# another case: "fühlt sich ... an" is a conjugation of "sich anfühlen", but Stanza does not treat them as a MWT
stanza.Pipeline('de')('Das Material fühlt sich weich an.')
# another case: "hält hin" is a conjugation of "hinhalten"
stanza.Pipeline('de')('Er hält uns schon seit Stunden hin.')

[AngledLuffa]

I definitely like the idea of adding lemmas it is getting wrong.

For both German and Italian we already did some bulk downloads. For German we used some Wiktionary data, and for Italian we had a long list of verbs and their conjugations from an Italian professor using our toolkit. I am not at all surprised there are gaps in these datasets, though, it's hard to cover every word.

another case: "fühlt sich ... an" is a conjugation of "sich anfühlen", but Stanza does not treat them as a MWT

MWT for German is an interesting idea, but the dataset (UD German_GSD) we build the models from only treats im, am, zum, zur, and other short words like that as MWT. In fact, this particular example is attested in the GSD treebank, so you can see the standard used in the treebank:

# sent_id = train-s259
# text = Ein Nachmittag in Strassenhof fühlt sich an einem warmen Frühlingssonntag im April schon fast wie Urlaub an.
1       Ein     ein     DET     ART     Case=Nom|Definite=Ind|Gender=Masc|Number=Sing|NumType=Card|PronType=Art 2       det     _       _
2       Nachmittag      Nachmittag      NOUN    NN      Case=Nom|Gender=Masc|Number=Sing        5       nsubj   _       _
3       in      in      ADP     APPR    _       4       case    _       _
4       Strassenhof     Strassenhof     PROPN   NE      Case=Dat|Gender=Masc|Number=Sing        2       nmod    _       NamedEntity=Yes
5       fühlt   fühlen  VERB    VVFIN   Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin   0       root    _       _
6       sich    sich    PRON    PRF     Case=Acc|Person=3|PronType=Prs|Reflex=Yes       5       obj     _       _
7       an      an      ADP     APPR    _       10      case    _       _
8       einem   ein     DET     ART     Case=Dat|Definite=Ind|Gender=Masc|Number=Sing|NumType=Card|PronType=Art 10      det     _       _
9       warmen  warm    ADJ     ADJA    Case=Dat|Degree=Pos|Gender=Masc|Number=Sing     10      amod    _       _
10      Frühlingssonntag        Frühlingssonntag        NOUN    NN      Case=Dat|Gender=Masc|Number=Sing        5       obl     _       _
11-12   im      _       _       _       _       _       _       _       _
11      in      in      ADP     APPR    _       13      case    _       _
12      dem     der     DET     ART     Case=Dat|Definite=Def|Gender=Masc|Number=Sing|PronType=Art      13      det     _       _
13      April   April   PROPN   NN      Case=Dat|Gender=Masc|Number=Sing        10      nmod    _       NamedEntity=Yes
14      schon   schon   ADV     ADV     _       15      advmod  _       _
15      fast    fast    ADV     ADV     _       5       advmod  _       _
16      wie     wie     CCONJ   KOKOM   _       17      case    _       _
17      Urlaub  Urlaub  NOUN    NN      Case=Acc|Gender=Masc|Number=Sing        5       obl     _       _
18      an      an      ADP     PTKVZ   _       5       compound:prt    _       SpaceAfter=No
19      .       .       PUNCT   $.      _       5       punct   _       _

I intend to get these - it's just been a busy couple days. Hopefully today or tomorrow.

[yolpsoftware]

Another one with a wrong lemma:

# lemma of "s'" should be "se", not "soi"
stanza.Pipeline('fr')("Les frontières s'éstompent.")

I just realized that the "German MWT cases" above are kind of the opposite of an MWT: the MWT im is a contraction, so the POS tagger should show its components in and dem.

However with "fühlt sich ... an", it's the other way round. The POS tagger should (IMO) show that there is a connection between fühlt and an, because they are both part of the verb in this case. So treating it as an MWT seems wrong.

It does correctly classify an as xpos=PTKVZ (separable verbal particle), so that seems fine. Also, the head is set to the verb. So I'm not sure if there's an easy solution to the German cases above. Ideally Stanza should set lemma of fühlt to anfühlen or sich anfühlen, but that might have its own problems. Perhaps some kind of compound_lemma property that is only set in such cases?

[AngledLuffa]

I don't envision compound_lemma happening any time soon. That'd be a whole new project, with limited training resources, and as you have observed, the information you want is already encoded in the dependencies.

I put a few of the examples you gave above into the Pipeline with the default_accurate models, which uses the pretrained charlm to improve the lemmatizer.

>>> doc = pipe('Le galline beccano i chicchi.')
>>> print("{:C}".format(doc))
# text = Le galline beccano i chicchi.
# sent_id = 0
1       Le      il      DET     RD      Definite=Def|Gender=Fem|Number=Plur|PronType=Art        2       det     _       start_char=0|end_char=2
2       galline gallina NOUN    S       Gender=Fem|Number=Plur  3       nsubj   _       start_char=3|end_char=10
3       beccano beccare VERB    V       Mood=Ind|Number=Plur|Person=3|Tense=Pres|VerbForm=Fin   0       root    _       start_char=11|end_char=18
4       i       il      DET     RD      Definite=Def|Gender=Masc|Number=Plur|PronType=Art       5       det     _       start_char=19|end_char=20
5       chicchi chicco  NOUN    S       Gender=Masc|Number=Plur 3       obj     _       SpaceAfter=No|start_char=21|end_char=28
6       .       .       PUNCT   FS      _       3       punct   _       SpaceAfter=No|start_char=28|end_char=29

>>> doc = pipe('Lancia la palla verso i birilli.')
>>> print("{:C}".format(doc))
# text = Lancia la palla verso i birilli.
# sent_id = 0
1       Lancia  Lancia  VERB    V       Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin   0       root    _       start_char=0|end_char=6
2       la      il      DET     RD      Definite=Def|Gender=Fem|Number=Sing|PronType=Art        3       det     _       start_char=7|end_char=9
3       palla   palla   NOUN    S       Gender=Fem|Number=Sing  1       obj     _       start_char=10|end_char=15
4       verso   verso   ADP     E       _       6       case    _       start_char=16|end_char=21
5       i       il      DET     RD      Definite=Def|Gender=Masc|Number=Plur|PronType=Art       6       det     _       start_char=22|end_char=23
6       birilli birillo NOUN    S       Gender=Masc|Number=Plur 1       obl     _       SpaceAfter=No|start_char=24|end_char=31
7       .       .       PUNCT   FS      _       1       punct   _       SpaceAfter=No|start_char=31|end_char=32

>>> doc = pipe('Le termiti possono distruggere il legno.')
>>> print("{:C}".format(doc))
# text = Le termiti possono distruggere il legno.
# sent_id = 0
1       Le      il      DET     RD      Definite=Def|Gender=Fem|Number=Plur|PronType=Art        2       det     _       start_char=0|end_char=2
2       termiti termite NOUN    S       Gender=Fem|Number=Plur  4       nsubj   _       start_char=3|end_char=10
3       possono potere  AUX     VM      Mood=Ind|Number=Plur|Person=3|Tense=Pres|VerbForm=Fin   4       aux     _       start_char=11|end_char=18
4       distruggere     distruggere     VERB    V       VerbForm=Inf    0       root    _       start_char=19|end_char=30
5       il      il      DET     RD      Definite=Def|Gender=Masc|Number=Sing|PronType=Art       6       det     _       start_char=31|end_char=33
6       legno   legno   NOUN    S       Gender=Masc|Number=Sing 4       obj     _       SpaceAfter=No|start_char=34|end_char=39
7       .       .       PUNCT   FS      _       4       punct   _       SpaceAfter=No|start_char=39|end_char=40

>>> doc = pipe('Ho comprato una lattina di sardine per pranzo.')
>>> print("{:C}".format(doc))
# text = Ho comprato una lattina di sardine per pranzo.
# sent_id = 0
1       Ho      avere   AUX     VA      Mood=Ind|Number=Sing|Person=1|Tense=Pres|VerbForm=Fin   2       aux     _       start_char=0|end_char=2
2       comprato        comprare        VERB    V       Gender=Masc|Number=Sing|Tense=Past|VerbForm=Part        0       root    _       start_char=3|end_char=11
3       una     uno     DET     RI      Definite=Ind|Gender=Fem|Number=Sing|PronType=Art        4       det     _       start_char=12|end_char=15
4       lattina lattina NOUN    S       Gender=Fem|Number=Sing  2       obj     _       start_char=16|end_char=23
5       di      di      ADP     E       _       6       case    _       start_char=24|end_char=26
6       sardine sardina NOUN    S       Gender=Fem|Number=Plur  4       nmod    _       start_char=27|end_char=34
7       per     per     ADP     E       _       8       case    _       start_char=35|end_char=38
8       pranzo  pranzo  NOUN    S       Gender=Masc|Number=Sing 2       obl     _       SpaceAfter=No|start_char=39|end_char=45
9       .       .       PUNCT   FS      _       2       punct   _       SpaceAfter=No|start_char=45|end_char=46

It got all four of the problem cases you identified right. Now, of course we can still add them to the training data, and then the faster default package will hopefully get them right as well. In fact, if you'd check those sentences for other errors and don't find any, I can do exactly that. But my overall conclusion is that it's probably better just to use default_accurate, if you want the best results possible

[AngledLuffa]

German:

>>> pipe = stanza.Pipeline("de", package="default_accurate", processors="tokenize,pos,lemma,depparse")
>>> doc = pipe('Sie schneidet sich die Nägel.')
>>> print("{:C}".format(doc))
# text = Sie schneidet sich die Nägel.
# sent_id = 0
1       Sie     sie     PRON    PPER    Case=Nom|Gender=Fem|Number=Sing|Person=3|PronType=Prs   2       nsubj   _       start_char=0|end_char=3
2       schneidet       schneiden       VERB    VVFIN   Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin   0       root    _       start_char=4|end_char=13
3       sich    sich    PRON    PRF     Case=Dat|Person=3|PronType=Prs|Reflex=Yes       2       obl:arg _       start_char=14|end_char=18
4       die     der     DET     ART     Case=Acc|Definite=Def|Number=Plur|PronType=Art  5       det     _       start_char=19|end_char=22
5       Nägel   Nagel   NOUN    NN      Case=Acc|Gender=Masc|Number=Plur        2       obj     _       SpaceAfter=No|start_char=23|end_char=28
6       .       .       PUNCT   $.      _       2       punct   _       SpaceAfter=No|start_char=28|end_char=29

French:

>>> pipe = stanza.Pipeline("fr", package="default_accurate", processors="tokenize,pos,lemma,depparse")
>>> doc = pipe('Le portable vibre dans la poche.')
>>> print("{:C}".format(doc))
# text = Le portable vibre dans la poche.
# sent_id = 0
1       Le      le      DET     _       Definite=Def|Gender=Masc|Number=Sing|PronType=Art       2       det     _       start_char=0|end_char=2
2       portable        portable        NOUN    _       Gender=Masc|Number=Sing 3       nsubj   _       start_char=3|end_char=11
3       vibre   vibrer  VERB    _       Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin   0       root    _       start_char=12|end_char=17
4       dans    dans    ADP     _       _       6       case    _       start_char=18|end_char=22
5       la      le      DET     _       Definite=Def|Gender=Fem|Number=Sing|PronType=Art        6       det     _       start_char=23|end_char=25
6       poche   poche   NOUN    _       Gender=Fem|Number=Sing  3       obl:mod _       SpaceAfter=No|start_char=26|end_char=31
7       .       .       PUNCT   _       _       3       punct   _       SpaceAfter=No|start_char=31|end_char=32

>>> doc = pipe("Les frontières s'éstompent.")
>>> print("{:C}".format(doc))
# text = Les frontières s'éstompent.
# sent_id = 0
1       Les     le      DET     _       Definite=Def|Number=Plur|PronType=Art   2       det     _       start_char=0|end_char=3
2       frontières      frontière       NOUN    _       Gender=Fem|Number=Plur  4       nsubj:pass      _       start_char=4|end_char=14
3       s'      soi     PRON    _       Person=3|PronType=Prs|Reflex=Yes        4       expl:pass       _       SpaceAfter=No|start_char=15|end_char=17
4       éstompent       éstomper        VERB    _       Mood=Ind|Number=Plur|Person=3|Tense=Pres|VerbForm=Fin   0       root    _       SpaceAfter=No|start_char=17|end_char=26
5       .       .       PUNCT   _       _       4       punct   _       SpaceAfter=No|start_char=26|end_char=27

So the last one is not as you expected. but the issue here is that there are no examples of s' lemmatized as se in the training data. All examples of s' as SCONJ are lemmatized as si, and all examples as PRON are lemmatized as soi. Unless you can convince the treebank maintainers to change their standard, this is pretty much what you're stuck with when using our models

[yolpsoftware]

Thank you for trying them out. So the issue seems to be that my Stanza installation produces consistently different results than yours. All of the examples above were run at least 3 times, with persistent wrong lemmas.

My Stanza runs on Ubuntu 22.04, Python 3.10 on a WSL instance inside of Windows 10 on an AMD CPU. And I just ran it on my Macbook Pro M4 multiple times: all examples above except the German one with "Nägel" were giving the same wrong lemma as above ("Nägel" was correct on the Macbook however). And two of them I ran at least 20 times, to see if its an issue of neural network probabilities - not it isn't, I got 20 times the same wrong lemma.

So this is really strange. Does Stanza somehow cache results? However, all evaluations took roughly the same amount of time, so I don't expect caching to be the reason.

Do you have an explanation? Why do the lemmas work correctly on your machine but repeatedly not on mine? Do you have the same standard installation as I do, like pip install stanza ==> import stanza ==> nlp = stanza.Pipeline('it'), or do you use something different?