How can I reproduce the models accuracy evaluations?

[vmatter]

I would like to reproduce the accuracy evaluations of the Portuguese models .

I tried to search for the code used in the accuracy evaluation, however, I could not find it...

Is there a way to reproduce them?

Many thanks.

[polm]

Evaluation metrics for a pipline are saved in meta.json under the performance key. Evaluation (on the dev set) is performed automatically during training, but you can also do it manually (on any data, such as test data) with spacy evaluate. The metrics on the model pages are for the dev set.

[vmatter]

Where can I find this performance key?

Just to make sure that we are on the same page, I want to know how to reproduce the same results as the one found in the Portuguese models:

My custom model uses the spacy evaluate.

However, I would really like to know how can I get the general accuracy of the UD Portuguese Bosque v2.8 and WikiNER corpus that achieved the F-Score of 0.88.

Did spaCy use these corpus' training and test datasets in the spacy evaluate function? If so do you know how to get them?

Thank you.

[polm]

Where can I find this performance key?

In the root directory of your pipeline, there is a meta.json file. That is a JSON dict, and it has a performance key. It looks like this:

...
  "performance":{
    "token_acc":0.9993092439,
    "token_p":0.9956819193,
    "token_r":0.9957659295,
    "token_f":0.9957239226,
...

Thoe are the numbers on the models page, and are based on an evaluation of the dev set used in training.

In this case the evaluation is based on two datasets, the UD dataset and WikiNER. I believe in this case the pipelines are trained separately (WikiNER for NER, UD for the rest) and then combined, so you should be able to do the same for evaluation.

If you have the same annotations on your data I think spacy evaluate should provide you with the same kinds of scores.

Did spaCy use these corpus' training and test datasets in the spacy evaluate function? If so do you know how to get them?

For UD corpora, the data is split into train/dev/test, with each in a separate file. So pt_bosque-ud-dev.conllu is the dev set that the evaluation numbers are against.

[vmatter]

Many thanks, now I got the essence of how it was evaluated.

I will replicate the tests and evaluate the results.

Once again, thank you!