Configuration parameters for portuguese edit_tree_lemmatizer

[abcp4]

Hello, I would like to know which (or some) configuration to reach the reported performance for lemmatization in Portuguese show here:

Also in the link.
I have used default parameters for training the lemmatizer component, with or without other components, reaching only 88% lemma acc. Also, I am training without transformers

[danieldk]

What do the scores for other components look like? It is common to get such low scores when no static vectors are loaded. You could for instance use the vectors of the pt_core_news_lg model.

If you start with the standard configuration of the pt_core_news_lg model, you can do so by setting the vectors setting in the paths section to pt_core_news_lg:

[paths]
# ...
vectors = "pt_core_news_lg"

The standard configuration should already initialize the vectors from this path:

[initialize]
# ...
vectors = ${paths.vectors}

Also make sure that static vectors are used by the tok2vec embedder:

[components.tok2vec.model.embed]
# ...
include_static_vectors = true

If this isn't the issue, could you post your configuration, so that we can have a look?

[abcp4]

Hello, thanks for answering and helping me on this problem, I really want to train and reproduce the related performance. I set the dataset as UD_Portuguese-Bosque in project.yml. I annexed 2 sample config I have tried with different tokenizers and tried to load the embedding vectors in the tokenizer with them, having the same results. I can later send the performances from others components trained together.
lemma1.txt
lemma2.txt
.

[danieldk]

That's odd, I took your second configuration (lemma2.txt) and trained it on the UD_Portuguese-Bosque treebanks and get 97% accuracy on the dev set and 96% on the test set. Maybe something went awry in the conversion of the treebank?

Edit: I can confirm the low scores when using a checkout from https://github.com/UniversalDependencies/UD_Portuguese-Bosque, I'll investigate this a bit more...

[danieldk]

Could you try converting the train/dev/test sets with the -T option of spaCy convert? E.g.:

python -m spacy convert -C UD_Portuguese-Bosque/pt_bosque-ud-dev.conllu ./ --n-sents 10 -T

The issue is that otherwise the UD subword tokens and the corresponding tokens in the raw text cannot be aligned. For example, consider the following fragment:

# text = Muitas das prioridades do novo governo coincidem com as prioridades do PT.
# sent_id = CF1-8
# source = CETENFolha n=1 cad=Opinião sec=opi sem=94a
1       Muitas  muito   PRON    _       Gender=Fem|Number=Plur|PronType=Ind     9       nsubj   _       _
2-3     das     _       _       _       _       _       _       _       _
2       de      de      ADP     _       _       4       case    _       _
3       as      o       DET     _       Definite=Def|Gender=Fem|Number=Plur|PronType=Art        4       det     _       _

The raw text has the token das, but the gold-standard data will have the subword tokens de and as. Since the tokenizer doesn't tokenize das as de followed by as, the tokenized sentence and the gold standard data cannot be properly aligned. Consequently, the predicted lemma will be considered as an incorrect. The -T option merges CoNLL-U subtokens in the conversion, so that das will be used in the gold-standard data, making proper alignment between the gold-standard data and the predictions possible.

[abcp4]

Hello, it worked like a charm! It was really the subwords tokens, now the performance is on point, thank you very much!!