Minimum/maximum loss value configuration for projects?

[Lolologist]

Hi there!
I've got a model trained on a couple thousand examples that, depending on the random split of training/testing docs, occasionally does "very well" early on but with a fairly enormous loss, and I'd like to be able to configure (in the project's config.cfg file and/or the project.yml presumably) a maximum loss allowable for something to be considered model-best. Is this something that already exists, or I can implement easily enough?

[polm]

The model-best is saved based on performance on the dev set. To be clear, the issue is that you have models that perform well on the dev set despite having high loss (= performing poorly on the training set)? That sounds like a really strange thing to happen.

There isn't a parameter for this, and the state of being best or not is held by the process rather than on disk, so I'm not sure it'd be easy to customize. You might want to look at hte spacy/training/loop.py code to see how that works - since it is Python you can modify it without recompiling if you need to.

[Lolologist]

Thanks for the speedy reply!

You got the situation correct there, more or less! F-score during training got its best value during a really high loss step. It's probably due to my data being noisy, frankly; I know it isn't 100% accurate.