Handling extreme cases in validation sanity check

[ptoews]

I'm using the default settings for the validation sanity check and think it is a useful feature.

However, because the validation sanity check only processes a tiny amount of data, there might be some extreme cases occuring. Specifially, I'm using torchmetric's ConfusionMatrix, and most of the time not all of the existing classes are represented in the first few batches. Therefore the computation of the confusion matrix results in a warning:

UserWarning: 2 NaN values found in confusion matrix have been replaced with zeros.

I get this for every training run and it's a bit annoying. What's the best way to prevent this warning from occuring in such cases? I don't want to filter this warning globally, because if this happens in the actual validation it is definitely an issue. So I wondered how such extreme cases in the validation sanity check are supposed to be handled. Is it somehow possible to run some code before and after this sanity check, to disable this warning temporarily?

[akihironitta]

Hi @ptoews on_train_start() runs right after sanity-checking, so you can filter out the warning until sanity check finishes and remove the warning filter in on_train_start() hook so that you get the warning during the actual validations. FYI, here's the execution order of hooks:
https://pytorch-lightning.readthedocs.io/en/1.7.7/common/lightning_module.html#hooks

[ptoews]

Thanks, it's not as elegant as having something that only surrounds the sanity check, but it's good enough.