How to get confidence score for each entity for custom NER model?

[koyelseba]

• spaCy Version Used: 3.1

How to get confidence score for each entity for custom NER model?

[shadeMe]

This is unfortunately not possible - please refer to this comment for more context and information about an alternative method to calculate confidence scores.

[shadeMe]

The post that I linked above has a link to an example spancat project that should help you get started. The SpanCat blog post should give you an overview of the component, and you can find the API documentation here.

[koyelseba]

Thanks.
I have trained the SpanCategorizer model and got the good accuracy. Few cases not able to detect correct span or entity but the confidence score is to high( like 0.995) . If model not able to detect correct span or entity thn it should return low confidence. but its not happening. Can you tell me why it's happening? How to solve this issue?

[shadeMe]

The accuracy of the model directly depends on the data on which it is trained. Since the model does not have any concept of "correctness" beyond what is represented therein, a span that is mislabeled with high confidence is likely an indication that the training data doesn't sufficiently cover (or isn't fully representative of) the distribution of the classes being predicted.

You can try to augment the data by providing more examples of the correct and incorrect classes. This can help the model suss out the differences between them, especially if they are similar/ambiguous.

[koyelseba]

Thanks for the guidance. I will try to add more examples.
I have compared the result between Spacy NER and Spacy Spancat. I got 0.98 Score in Spacy NER but only 0.88 Score in Spacy Spancat. My data dont have any overlap Span, still I am getting less score in Spancat. Can you tell me how to get the 0.98 Score in Spacy Spancat? or Is it not possible to achieve that score? I need confidence score thats why I have started to use SpanCat but score is low. Please guide me

[shadeMe]

The NER and SpanCat components built on different model architectures and as such have different predictive powers. Beyond augmenting and expanding your training data, you could experiment with the different hyperparameters of the SpanCat model (refer to the blog post and API docs linked in the previous comment) to see if you can improve the results.