Typical learning metrics behavior? new training parameters

[cleterrier]

Hi,

For my latest training I changed a couple of parameters:

• Camera is sCMOS with a 97-nm pixel, compared to previous trainings for an EMCCD with a 160-nm pixel
• As camera has less sensitivity and no amplification, acquisition images have a lot less S/N
• Size of the image patch up from 40x40 to 64x64 pixels (because PSF were very elongated with ~20-pixel width due to high astigmatism)
• In order to fit training in the 8 GB of my graphics card, I had to drop batch size from 64 to 32
• Learning rate was dropped from 0.004 (that I usually use rather than de default 0.006 for the 40x40 EMCCD training, because I had frequent occurrences of bad learning with 0.006) to 0.002.

This is what the currently running training looks like for the sCMOS (red) compared to an older training with the default parameters on the EMCCD (orange):

I understand the efficiency/fi/jac/mad/rec/rmse metrics for the red training are not as good as the orange one due to lower S/N. However I'm not sure about two things:

• is it usual/ok that metrics such as mad/rec/rmse decrease sharply in the first epochs, but then rise toward a plateau on the red training rather than monotonically decrease toward a plateau like the orange training? I've never seen such a behavior before.
• Are the differences in learning metrics (test_ep, train_batch, train_ep) expected/normal given the difference in batch size and learning rate? I'm not sure what they refer to from their names (https://decode.readthedocs.io/en/master/logging.html?highlight=metrics#metrics) but I assume they depend on those settings.
• Should I stop the training after the same number of epochs than the default parameters (1000) or go larger? To me both of these training look like they don't change much after 500 epochs, but I'm no specialist...

Sorry for the pestering, it's just that the current training is running for 36 hours now and it would be quite time-consuming to find out what is a good training or not by trial and error! Thanks

[ASpeiser]

Hey,

1. Yes I would not worry about this. Basically what happens is that at the very beginning the network only detects the very brightest emitters (see the recall) and therefore the achievable rmse is much lower. The overall performance is still steadily improving (see efficiency)
2. These are the values of our loss function evaluated directly during training after each batch that was processed (train_batch and the mean of that train_ep) or after one epoch on the test data (test_ep). It's a bit hard to interpret because we use Log probabilities so it can go below zero. As a guideline, we do take the mean over the batch dimension but each ground truth emitter adds a loss term that we sum over. So the batch size and learning rate should have no influence on the value the loss converges to. But having more emitters (due to larger window size) and lower SNR explains the differences you see. There's also a background fitting loss term that can move these values around.
3. Yes this agrees with our experience. 500 epochs should suffice in this case (and in most cases).

And no worries, keep the questions coming!