How to setup [training.score_weights] for ents_per_type in the config file?

[pauljouet]

Hello,

I have trouble finding the answer because it is difficult asking the question to Google. I am trying to train a simple NER model, and I saw that the config file section for setting up the importance of the scoring metrics [training.score_weights] has an ents_per_type arg set to null.

I guess that this argument is useful for telling the model which labels it should get right (ex: ORG is more important than LOC). This would be very useful for me because my model's detections are very uneven currently.

The question is simple: how to write the config file? Should I pass a dict with the name of each label, each with their weight per metric (kind of similarly as returned by the scorer)? If so what are the names of the keys? Is there a difference if I have custom entities?

Also, if the question is too trivial, please redirect me to a good reference for learning what all config options do, as I could not find one yet, it would be very appreciated!

[adrianeboyd]

The score weights settings can't be used to influence the model training like this. The main thing they're used for is to pick the best final model and for early stopping.

The way the current ner model works, there's no way to prioritize particular entity types. You could try training a separate ner model for each type to see if that improves the results or lets you prioritize particular types. See also the "double NER" example project: https://github.com/explosion/projects/tree/v3/tutorials/ner_double. Alternatively, it's still a bit experimental, but you could see if spancat might be a better fit. It has a global threshold and not per-type thresholds, but you could use a low threshold and then run post-filtering to prioritize annotations based on your task.

If you want to use scores for individual types for best model / early stopping, you'd need to write a custom scorer that saves the relevant scores as top-level keys, since the score weights only support top-level keys and not values that are further down in the dict. A related thread: #9282 (comment)