Alignment between ParserStepModel output scores dim and EntityRecognizer.move_names off by 1

[nikita-galileo]

Hi there! I'm trying to match the output scores from the ParserStepModel's forward pass with the available actions, but I'm finding that the output scores produced in line 329 of parser_model.pyx do not match in length to the returned values of the EntityRecognizer.move_names, which is the only place I've been able to find an ordered list of possible actions. For context, I'm training an "ner" component from scratch.

If I have a single label in my dataset then the EntityRecognizer.move_names corresponds with a list of 5 actions: ['O', 'B-my_class', 'I-my_class', 'L-my_class', 'U-my_class'], but EntityRecognizer.moves.n_moves equals 6 and the scores outputted by the ParserStepModel are of length 6 as well. What do the 6 scores correspond with?

Spacy version 3.2.1

[nikita-galileo]

I see that the discrepancy might be because the first element corresponds to a "U-" token which is then removed when fetching the move_names. Is there any explanation for what the purpose of the "U-" token is?

[adrianeboyd]

U- is used internally for blocked entities, which are tokens that are marked as not being allowed to be part of any entity, so ner won't predict any entity spans that include them. This feature isn't used that frequently, but it's supported by Doc.set_ents and ner.

The list from move_names is only for display to the user (not for anything internal in the parser), so my guess is that it's removed here to avoid confusion related to an "invalid-looking" tag that users wouldn't be expecting.

[nikita-galileo]

Got it thank you very much for the answer! Do you have any sense as to what the scores outputted from the model might look like? For example for samples which the model might be confused between labels on, would the scores reflect that confusion by giving similar values to the two labels' tag scores?

[adrianeboyd]

You can get semi-usable scores by using beam search (we do this in prodigy), but the scores do not correspond directly to a confidence or probability. Related threads (with some out-of-date details, but the general issues are still the same):

• 💫 Train parser and NER with regression objective, to make scores express expected parse quality #881
• Probabilistic cutoff for NER #2601

If you want confidence scores for entity spans, you can try training a spancat instead: https://spacy.io/api/spancategorizer