Scoring of textcat output

[albin02t]

Can someone show some example of using Scorer.score_cats() and the format of the Example objects to be passed for a single label Text classification model

[adrianeboyd]

import spacy
from spacy.scorer import Scorer
from spacy.training import Example

nlp = spacy.blank("en")
# load/add your textcat pipe instead

text = "This is a positive text."

# create a reference doc
reference = nlp.make_doc(text)
reference.cats = {"POSITIVE": 1.0}

# create a predicted doc
predicted = nlp(text)
# fake a prediction for this demo since there's no textcat
predicted.cats = {"POSITIVE": 0.95}

example = Example(predicted, reference)

print(Scorer.score_cats([example], attr="cats", labels=["POSITIVE"], multi_label=True, threshold=0.75))