Understanding SpanCat's confidence scores

[nsrinidhibhat]

Hi,
I wish to understand how the confidence scores in SpanCat work in these two following scenarios.

• When the trained model does or doesn't predict a label: The sum of the confidence scores (both cases) doesn't add up to 1.0. Does this mean that 1-(sum of confidence score) is assigned to "not a label"?
• Multiple labels assigned to a word during prediction: Why is the confidence score above 1.0 when summed up? Does this mean that a model takes iterations to predict label(s)(one by one) and calculates a confidence score based on that? If so, on what criteria does a model know that there is another label it can assign after making its first label prediction on the word?

I need to perform uncertainty sampling on the confidence scores received from the model and hence the query. Kindly help me out with this. Thank you!

[ljvmiranda921]

Hi @nsrinidhibhat ,

• Spancat's default scorer currently uses the LinearLogistic layer. This means that it's treating each class independently, hence their confidences do not sum up to 1. This works in theory because we want spancat to work with overlapping entities. But if you have exclusive classes, you might need to change the scorer to Softmax.
• The classification part of spancat works as a multilabel classification problem, so no, it doesn't try to predict label(s) one by one. Instead, what happens is that you have an array of confidence scores (per label) and you try to obtain the argmax of that array. Right now, there's no concept of "no label" in spancat. In addition, the LinearLogistic layer outputs a relevance probability (a value between 0 to 1) for each class independently. So this means that if non of the classes seem relevant then the sum of the class scores will be close to 0. If all classes seem relevant then the sum is close to the number of classes.

[ljvmiranda921]

Hi @efratophir , you can pass thinc's Softmax layer (thinc.api.Softmax) in the components.spancat.model.scorer section of your configuration file. You can check the layer swapping documentation for more info.

[lanchiang]

Hello @ljvmiranda921, could you elaborate on how to set the parameter for the softmax layer in the config file?
Without setting init_W and init_B, the model uses the default zero_init, which is complained by spacy that "it's not JSON serializable".

[ljvmiranda921]

Hi @lanchiang , you can update this part of the config into something like this:

...
[model]
@architectures = "spacy.SpanCategorizer.v1"
- scorer = {"@layers": "spacy.LinearLogistic.v1"}
+ scorer = {"@layers": "Softmax.v2"}
...

[lanchiang]

Hi @ljvmiranda921, thanks for the reply. In my case, I needed to add the following code as well, otherwise spacy complains that zero_init cannot be serialized.

[model.scorer.init_W]
@initializers = "zero_init.v1"

[model.scorer.init_b]
@initializers = "zero_init.v1"`

[jamnicki]

@ljvmiranda921 How to change scorer to Softmax in default spancat config?
In my case its Polish with only spancat component, CPU, efficiency.

...
[components.spancat.model.scorer]
- @layers = "spacy.LinearLogistic.v1"
- nO = null
- nI = null
+ @layers = "Softmax.v2"
...

Above change return following error message.

SystemError: <method '__getitem__' of 'dict' objects> returned a result with an error set

After a moment I realized I should open new discussion, sorry

How to change scorer to Softmax in spancat?