Understanding the metrics displayed while training spacy components

[shrinidhin]

Hi!I have been training my custom components and trying to understand the different output metrics for each component. I am finding the documentation a bit confusing to understand the outputs. I am trying to understand the output metrics displayed for NER, Entity Linker and TextCat components/
For Eg: Which metric should I consider to understand how well trained my TextCat model is : the cats_score or score metric?

[polm]

SCORE is always the total score for your pipeline. It's calculated based on scores of other components as configured in the training.score_weights part of the config.

If you are training just a textcat (and tok2vec or whatever) then normally the CATS_SCORE will just be a value from 0-100 and SCORE will be the same value from 0 to 1, so it doesn't matter which one you look at.

If you were training other components at the same time, they might also influence the SCORE, so CATS_SCORE would be the value for just the textcat.

[polm]

If you are training NER, EntityLinker, and textcat components at the same time then SCORE will be a combination of those values. Can you share the training.score_weights section of your config? Also, just to be clear, is the textcat multilabel or not?

[shrinidhin]

I have already trained my NER and Entity Linker components and added them separately to my pipeline. Note the training section of the config:

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
patience = 3000
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = ["tok2vec","ner","entity_linker"]
annotating_components = []
before_to_disk = null

Here is the training.score_weights section:

[training.score_weights]
cats_score = 1.0
cats_score_desc = null
cats_micro_p = null
cats_micro_r = null
cats_micro_f = null
cats_macro_p = null
cats_macro_r = null
cats_macro_f = null
cats_macro_auc = null
cats_f_per_type = null
cats_macro_auc_per_type = null
sents_p = null
sents_r = null
sents_f = null
ents_f = null
ents_p = null
ents_r = null
ents_per_type = null

[adrianeboyd]

Can you check [training.score_weights] in the config saved in the model-best output directory? It's possible a component is automatically filling in another score weight before training. (In effect, spacy train runs init fill-config before it starts training.)

[shrinidhin]

Hi!This is the [training.score_weights] section in the config saved in the model_best directory:

[training.score_weights]
sents_f = null
sents_p = null
sents_r = null
ents_f = null
ents_p = null
ents_r = null
ents_per_type = null
nel_micro_f = 0.5
nel_micro_r = null
nel_micro_p = null
cats_score = 0.5
cats_score_desc = null
cats_micro_p = null
cats_micro_r = null
cats_micro_f = null
cats_macro_p = null
cats_macro_r = null
cats_macro_f = null
cats_macro_auc = null
cats_f_per_type = null
cats_macro_auc_per_type = null

[polm]

You'll that there are two non-null lines there:

nel_micro_f = 0.5
cats_score = 0.5

So the SCORE value should be an average of those values.

It looks like you didn't have the NEL score info in your scoring section, so the defaults were filled in to create your final scoring config. (It'd be better if this were more obvious...)