Can I access the quality metrics of my SpaCy model in code?

[AnnemarieWittig]

Hello!

I am using SpaCy's python library, allowing run-time training of NER models. I'd now like to model the quality of the model against the given testdata. Since there is a console logging of it, I assumed I could access it in code, too. Could someone point me to the needed calls for this?

Figuratively:

How do I access this output:

automation_server  | ============================= Training pipeline =============================
automation_server  | ℹ Pipeline: ['transformer', 'tagger', 'parser', 'ner']
automation_server  | ℹ Initial learn rate: 0.0
automation_server  | E    #       LOSS TRANS...  LOSS TAGGER  LOSS PARSER  LOSS NER  TAG_ACC  DEP_UAS  DEP_LAS  SENTS_F  ENTS_F  ENTS_P  ENTS_R  SCORE 
automation_server  | ---  ------  -------------  -----------  -----------  --------  -------  -------  -------  -------  ------  ------  ------  ------
automation_server  |   0       0         121.26         0.00         0.00    139.68     0.00     0.00     0.00     0.00    0.00    0.00    0.00    0.00
automation_server  | 100     200       17324.04         0.00         0.00  35147.60     0.00     0.00     0.00     0.00   36.47   36.47   36.47    0.12
automation_server  | 200     400         606.09         0.00         0.00  12031.12     0.00     0.00     0.00     0.00   95.91   95.35   96.47    0.32
automation_server  | 300     600          90.85         0.00         0.00    445.00     0.00     0.00     0.00     0.00   94.80   93.18   96.47    0.31
automation_server  | 400     800           0.55         0.00         0.00      5.07     0.00     0.00     0.00     0.00   96.51   95.40   97.65    0.32
automation_server  | 500    1000           1.80         0.00         0.00      7.77     0.00     0.00     0.00     0.00   96.51   95.40   97.65    0.32
automation_server  | 600    1200           0.34         0.00         0.00      2.12     0.00     0.00     0.00     0.00   95.95   94.32   97.65    0.32
automation_server  | 700    1400           6.75         0.00         0.00      8.90     0.00     0.00     0.00     0.00   95.95   94.32   97.65    0.32
automation_server  | 800    1600           0.00         0.00         0.00      0.00     0.00     0.00     0.00     0.00   95.95   94.32   97.65    0.32
automation_server  | 900    1800           2.87         0.00         0.00      8.51     0.00     0.00     0.00     0.00   95.95   94.32   97.65    0.32
automation_server  | 1000    2000           0.00         0.00         0.00      0.00     0.00     0.00     0.00     0.00   95.95   94.32   97.65    0.32

in my python code.

Can I access these values for each of my questions separately?

[ljvmiranda921]

Hi @AnnemarieWittig , do you only need the final values for training? If yes, then it should be available in the meta.json of your training folder.

However, if you need the values per epoch, you have to write a custom logger (you can check the linked example for more information). Alternatively, you can also use ConsoleLogger.v2, set a value for the output_file in your config, and then parse the resulting JSONL file.

[AnnemarieWittig]

This already helped, thanks a lot!