NER transformer confidence

[fracav]

Hi,
i am using the latest version of spacy to extract key-value information from european documents of many different types. The approach followed is to obtain the text contained in the document with ocr techniques, apply a strong augmentation of the textual data, such as replacing the entities with similar ones or introducing artificial ocr errors in the template string and finally apply a NER algorithm. The technique works quite well and I am managing to get very interesting results (compared to more complex techniques that also analyze layout or characteristics of the image such as LayoutLM, or CUTIE). I am using a distilbert transformer-based architecture and it would be very useful for me to be able to obtain the confidence of the various predicted entities in terms of probability. I have found various issues and information about SpanCategorizer but it is not very clear to me if it is possible to use it with transformer-based models.

Anyone have any suggestions or an example to share on how to get a confidence score in this type of model?

[polm]

You would need to train a spancat model of your own to get the scores. The scores are not yet exposed, but they will be after #8855 goes out, which should be in the next release (>3.1.1); the docs will explain how to access them. If you're feeling adventurous you can try installing from master.

The spancat component will work the same whether the input is a Transformer or tok2vec.

[fracav]

Thank you for the answer I will try to follow your advice and install from the master 👍