Elastic resets

[t0278611]

Using model ema (at least for small-ish models) seems to drastically improve the validation results. I wonder why the ema weights are never used in training (a bit like the lookahead optimizer). It should be fairly forward to implement "elastic resets", where the online weights periodically get overwritten by the ema weights. I did not find the feature in the train args, if it exist please do point it out

[rwightman]

@t0278611 no, I never implemented this, there was also a simplified SAM variant that used EMA, not sure if it was of the model weights or past gradients to approx the original SAM algorithm.. I tried that one at one point but couldn't improve past training runs.

Open to adding this if there's some evidence of success with hacked timm scripts and models in the image space ... I've run across a lot of paper ideas that I couldn't replicate over the years...

[t0278611]

Thanks for the reply! I'll hack a quick version on a fork. The implementation should be quite straightforward. I'll test it on the CIFAR sets and ImageWoof with tiny vit.

I am wondering, why in train.py the ema_eval_metrics is used to overwrite eval_metrics and as such is the one logged in the summary. It would seem to me that eval_metrics is more informative for model progress. ema_eval_metrics seems to be mainly useful at the end to select the best checkpoint for inference

[t0278611]

I tried a few experiments and resetting to ema didn't seem to produce any clear benefits