English and Polish lemmatization not working

[TheCodingCoconut]

Hi all,

I've tried lemmatization using both English and Polish pipelines, but it's not working. All lemmas return '0'. Any idea why it's not working?

[polm]

Are you using the pretrained pipelines or custom ones? What version of spaCy etc are you using / what does spacy info show?

Also note token.lemma gives you a numeric hash of the string, you want token.lemma_ for the string value.

[TheCodingCoconut]

Default pipelines from spacy, pl_core_media_sm and eng_core_media_sm.

Sure thing, but the value returned for token.lemma is 0, and token.lemma_ is an empty string. I've checked that again in PyCharm and in bottom right hand corner where you have the list of your variables I saw that lemma and lemma_ attributes were respectively 0 and " ".

Token's lang attribute was "pl" and "eng", depending on the language.

[polm]

Just to check, are those actual your model names and language codes? We do not distribute models with those names. We distribute pl_core_news_sm and en_core_web_sm. We also do not use the language code eng, it would be en.

This code works for me.

import spacy

nlp = spacy.load("en_core_web_sm")

for tok in nlp("We have many fine cheeses"):
    print(tok, tok.lemma_, sep="\t")

Output:

We	we
have	have
many	many
fine	fine
cheeses	cheese

Please include the output of spacy info and the actual code you are running.