Which score does the patience parameter use to decide whether there is improvement or not?

[czcindy426]

Hi,

I have a question about the patience parameter. I know that patience = 1600 means that if training goes for 1600 batches without improvement it will stop. In the output of the training, I see there are measurement scores in ENTS_F, ENTS_P, ENTS_R, and SCORE. Which score of these does the patience parameter use to determine whether there is improvement or not?

[mbrunecky]

In your configuration file, you will see a section looking like:"

[training.score_weights]
ents_per_type = null
ents_f = 0.5
ents_p = 0.25
ents_r = 0.25

This tells Spacy how to combine p/r/f into one score used to measure 'improvement'. Somewehere it says that fractions need not to summ up as 1.00, but I always make sure they do. And I often emphasize the r-score ... just my project.
Note that patience is not in batches but in 'iterations' (and I have yet to completely understand what an iteration is in case of transformers), but in 'normal' Spacy pipelines it corresponst to number of docs.
And of course, what patience you want to use depends on your data, if you have a data set of 10,000 documents then patience of 1000 may be too low. IMHO, there should be 'minEpoch' parameter, because I definitely do not want to stop training in the middle of the first epoch.

[czcindy426]

Thank you for your reply! Is the combination of p/r/f the score shown in the last column of the output when we train a model? I attach a picture here.

If so, I don't understand the accepted answer to this question on stack overflow. https://stackoverflow.com/questions/68471586/training-epochs-interpretation-during-spacy-ner-training It seems that the person who answers the question only uses ENTS_R as the measurement of the improvement.

I hope I have made myself clear and thank you for your patience!

[mbrunecky]

Yes

[czcindy426]

Did you have a chance to take a look at the example from stack overflow whose url link is given above? You can see the output shared by the person who asked the question. In that example, the score in the last column does not make any improvement from batch 1200 (because it is already a perfect score), but the training stops at batch 4600 even if the patience parameter is set to 1600. Therefore, this suggests that it is not the score in the last column that is used to determine whether there is improvement or not.

I hope I have made myself clear. Thanks!

[mbrunecky]

In that example, the score at
3000 52.67 1462.61 99.96 99.93 99.99 1.00
may well have been the best, and hence it stopped at 4600 (1600 iterations later). The log display rounded to two decimal places is not precise enough, especially when the p/r/f are so high.
@polm knows far more about this, and I am not sure what he means by 'not evaluation iterations but batches'. It feels like he tries to convey that patience counts training iterations, not how many times was the 'dev' sample evaluated or how big the 'dev' sample is.

The whole 'stopping' issue also depends on how fast you wan to train= how big your 'dev' sample needs to be to represent your 'training' sample' (5%? 20%?), and how often you evaluate (eval_frequency = 200). When my 'epoch' has 8k docs, and I run up to 40k iterations, I use eval_frequency = 800 so that I 'only' evaluate the 'dev' docs 50 times (times 1000 docs in 'dev' = 50,000 doc evaluated during the training run). A smaller 'dev' sample speeds up the training, but it may lead to a premature stop.

[polm]

@polm knows far more about this, and I am not sure what he means by 'not evaluation iterations but batches'. It feels like he tries to convey that patience counts training iterations, not how many times was the 'dev' sample evaluated or how big the 'dev' sample is.

Yes, you're correct about what I was trying to convey here. The "evaluation iterations" or evaluations of the dev set are what's actually logged as output, so some people are confused that the steps/iterations used for patience aren't the same.