HOW: custom logger that shows learning rate during training + Option for saving training results?

[PeachDew]

Hi all, I am using spacy's CLI train for training a custom NER model. I would LOVE to view the learning rate during training, ie an additional column here -->

So I can verify that my learning rate schedulers are working as intended. Has anyone implemented a custom logger before and can point me in a direction?

Also, is there an option for saving training score/loss results history? Right now I have to resort to copying and pasting from the command line which is not ideal...

Thank you all for reading! Hope you have a great day! :-)

[shadeMe]

Hi, you can take a look at the spacy-loggers repository for examples on how to implement a custom logger.

There is no built-in mechanism to save the training score/loss history, but this info is passed to the logger. So, one could write a custom logger to save this information in a more structured manner.

[PeachDew]

thank you for the reply! to confirm im looking in the right direction, should i be adding code in my functions.py that includes functions like setup_logger, log_step and finalize shown in eg. https://github.com/explosion/spacy-loggers/blob/main/spacy_loggers/clearml.py ?

[shadeMe]

No, you don't need to add any (new) code at all. Just set up the experiment/run in the Mlflow dashboard/console and use the correct experiment/run IDs in the config file (as shown in the example in the README). For more info on how loggers work, please refer to the docs.

The code in the file you linked contains the implementation of the MlflowLogger.

[PeachDew]

tysm for the reply! i see that i can implement my own logger here, and i see that the logger accepts a dictionary from spacy. however it does not seem like spacy record/pass learning rate to the logger function, only epoch, step, scores, and losses.

Can custom loggers access other variables eg. batch size if using batch scheduler/ learning rate when using a lr scheduler?

[shadeMe]

The setup callback of the custom logger will receive the Language instance during the initialization stage. You can store this instance and query data from it inside the log_step callback. Current optimizer/batch scheduler state cannot be accessed this way, though.

One way of accessing the current learning rate would be use a custom LR scheduler that caches the yielded learning rate somewhere that the (custom) logger can access (like a global variable).

[PeachDew]

ahh i see, thank you for your helpful responses :-), I'll look into caching the lr as you suggested in a custom scheduler