Can't make experimental_edit_tree_lemmatizer work

[ricardojosehlima]

(Alert: Beginner user of Spacy here)

I have followed the instructions at https://explosion.ai/blog/edit-tree-lemmatizer#

pip install -U pip setuptools wheel
pip install spacy-experimental==0.2.0

and the spacy_experimental folder is at /home/ricardo/.local/lib/python3.8/site-packages/spacy_experimental/edit_tree_lemmatizer

The next instruction is:
You can then use the experimental_edit_tree_lemmatizer component factory:

[components.experimental_edit_tree_lemmatizer]
factory = "experimental_edit_tree_lemmatizer"

That’s all! Of course, we encourage you to experiment with more with the model.

For me, that wasn't all for I couldn't figure out what to do, so I went to https://github.com/explosion/spacy-experimental
There I found:

Then you can add the experimental components to your config or import from spacy_experimental:

[components.experimental_edit_tree_lemmatizer]
factory = "experimental_edit_tree_lemmatizer"

So I went to config.cfg for the model I am using (pt_news_core_lg) and changed this:

pipeline = ["tok2vec","morphologizer","parser","senter","attribute_ruler","lemmatizer","ner", "experimental_edit_tree_lemmatizer"]

and added this right after [components.lemmatizer]:
[components.experimental_edit_tree_lemmatizer]
factory = "experimental_edit_tree_lemmatizer"
backoff = "orth"
min_tree_freq = 3
overwrite = false
top_k = 1
scorer = {"@scorers":"spacy.lemmatizer_scorer.v1"}

Then I ran the code

import spacy
from spacy_experimental import edit_tree_lemmatizer
nlp = spacy.load("pt_core_news_lg")
texto = "Uma empresa tem sua ajuda."
doc = nlp(texto)
for token in doc:
print(token.text, token.lemma_)

I found out that the experimental edit_tree_lemmatizer folder should be inside the pt_news_core_lg-3.2.0 folder.
I then copied the experimental_edit_tree_lemmatizer to inside pt_news_core_lg-3.2.0
However, there is still something missing for the message I get is:

ValueError: Can't read file: /home/ricardo/.local/lib/python3.8/site-packages/pt_core_news_lg/pt_core_news_lg-3.2.0/experimental_edit_tree_lemmatizer/cfg

I tried copying the config.cfg file that exists inside pt_news_core_lg-3.2.0 to the folder pt_core_news_lg-3.2.0/experimental_edit_tree_lemmatizer and rename it as cfg and the message received was

ValueError: Expected object or value

[adrianeboyd]

Try starting with the demo project here, which trains the lemmatizer separately:

https://github.com/explosion/spacy-experimental/tree/master/projects/edit_tree_lemmatizer

You would want to modify this to the right lang/treebank and also the config to use the vectors from pt_core_news_lg.

Then you can use nlp.add_pipe(source=) to add the separately trained component to pt_core_news_lg.

[ricardojosehlima]

Thanks for the answer, but it opened new ways of difficulty to me. I will wait for some version of Spacy that will include the experimental lemmatizer or something similar to https://huggingface.co/spaces/spacy/pipeline-visualizer

[adrianeboyd]

Yes, after we've done more testing and we're happy with the performance, we plan to add the edit tree lemmatizer to some of the trained pipelines like pt_core_news_lg.

[ricardojosehlima]

Ok, thanks again! I got so excited with the announcement of a (for me long waited) well performed lemmatizer for Portuguese and to provide feedback that I forgot my limited skills with some of the Spacy's parts, namely projects which I have never tried.

[svlandeg]

We're happy to see the excitement at least! :-)