Evaluation scores for spancat model

[ludovicmoncla]

Hi,

I am using the spaCy train function for training a spancat model using data labeled with Prodigy.
It seems to work well but I wonder how I can get a more detailed evaluation?
I mean, in addition to SPANS_SC_F, SPANS_SC_P , SPANS_SC_R , SCORE. I am interested in precision and recall for each label individually.
Is there a built-in solution that I missed?

I've tried the Scorer.score_spans() function with examples with a snippet of code found in another discussion:

import spacy
from spacy.scorer import Scorer
from spacy.training import Example

scorer = Scorer()

nlp = spacy.load('fr_core_news_sm')
TEST_DATA = [
    ("Lyon, ville de France", {"entities": [(0, 4, "LOC")]}),
    ("Mon nom est Martin", {"entities": [(12, 18, "PER")]}),
]

examples = []
for text, annotations in TEST_DATA:
    doc_pred=nlp(text)
    example=Example.from_dict(doc_pred, annotations)
    examples.append(example)

scores = Scorer.score_spans(examples, 'ents')
print(scores['ents_per_type'])

This is working when used with the default fr_core_news_sm model. But it is not working with my custom spancat model (all scores are equal to zero which I know is not normal).
Is there something wrong with the use of the spancat model or with the test data format?
I understand from https://spacy.io/api/data-formats#dict-input that I should use spans instead of entities in the TEST_DATA dictionary but it is not working. I've also tried do change ents with spans in the score_spans() function without success. I imagine that there is something misleading between NER and spancat. But in my case, the model is trained only with tok2vec and spancat in the nlp pipeline.

Moreover, how can I use the Scorer.score_spans() function with a binary spacy dataset (.spacy file)?

Thanks,

[adrianeboyd]

If you use spacy evaluate, you can provide your custom spans key (a new option as of spacy v3.7.0) and save the detailed evaluation output in a file in order to see the per-label scores from Scorer.score_spans:

python -m spacy evaluate MODEL DATA_PATH --spans-key custom_key -o metrics.json

(Earlier versions of spacy include the detailed output for the spans key sc, but you couldn't pass a custom key to the evaluate command.)

[ludovicmoncla]

Thanks this is perfect!