How to get confidence score of custom trained NER model using Spacy?

[karndeepsingh]

Hi,
I have trained NER model and want to get a confidence score for each entity prediction. How that can achieved?
Please provide guidelines to achieve it.

Thanks

[polm]

The NER model doesn't expose confidence scores, the way it works the scores wouldn't be meaningful for that (see #831).

What you can do instead is train a spancat model, which can report confidence scores. Take a look at the spancat example project.

[polm]

I am using Transformer Model to train NER doesn't it provide a Confidence score?

The NER component does not expose confidence scores, whether you're using Transformers or not doesn't affect that.

Do this SpanCat works similarly to NER and does my NER annotated data works here as well or do I have to prepare another data for it to train?

Spancat mostly works like NER, you can use the same annotation data but you'll have to set it on Doc.spans rather than Doc.ents. Please see the example project and spancat docs.

[hiteshpara]

If i write the follwing code to predict the new sample for NER and type doc.spans then it gives empty dictionary. How can i get the confidence score of that prediction
import spacy
model = spacy.load(R"NERoutput/model-last") #load the best model
sent="hitesh is a good boy"
doc=model(sent)
for ent in doc.ents:
print(ent.text,ent.start_char,ent.end_char,ent.label_)

[adrianeboyd]

Please read the top message again. The ner model cannot give you confidence scores. One alternative option is to train a spancat model instead.

[giuppo]

The spancat blogpost makes it clear that spancat was introduced to

handle arbitrary and overlapping spans... providing more flexibility

However, it seems that performance significantly decreases when it is possible to use/compare with ner (EntityRecognizer).

So, for the case of training a model for a couple of easily defined entities (no overlap, etc.), what would you recommend doing to mitigate the problem of not having an explicit score to 'post-weight' predictions?

[thomashacker]

Unfortunately, there isn't much you can do with an existing NER model. I would recommend experimenting with spancat for your task and go from there.