How to assess Spacy Dependency Parsing Accuracy

[fatihbozdag]

Greetings all,

For my research paper, I used Spacy to extract dative sentences through Spacy en_core_model_trf. Despite detailed explanation and referring to Spacy's page for accuracy scores, a reviewer insisted me to publish accuracy score of the model over my dataset.

Is it possible to evaluate dependency parsing results and present any score?

Regards.

[thomashacker]

Hello,
for evaluating spaCy models we have the spacy evaluate command, however, one of the problems with the English models is that you need to have evaluation data with the same annotation scheme but we can't publish the data directly.

[fatihbozdag]

so, there is no possibility to publish such a score then?

[thomashacker]

If you have an evaluation dataset for the DependencyParser, you can use spacy evaluate to get the scores. If the command doesn't produce the specific scores you're looking for, you can also write your own evaluation script.

[fatihbozdag]

Can please provide sample code on how to apply spacy evaluate on pandas df? I have indeed a dataset which is pretagged manually. I assume I need to parse the evaluation dataset first with Spacy, and then?

[adrianeboyd]

To use spacy evaluate, you want to prepare a .spacy file in the same way you would for training: https://spacy.io/usage/training#training-data

But you can also run en_core_web_trf on your texts, export the annotation from the spacy docs into the format you normally use, and evaluate with another script/tool.

When reporting scores directly from spacy in a publication, be sure you're aware of the internal scorer details like the fact that punctuation labels are ignored by default. (Some details here: https://spacy.io/api/dependencyparser#init)

[fatihbozdag]

is there any other script/tool suggested?