Search and Replace NaN in Model parameters

[a-parida12]

There is so many time that a model's learnable parameter may become NaN even though gradient clipping or lower lr may be implemented by the user. It maybe useful to add a couple of functions that can check if a model parameter becomes a NaN to replace it with a number. This may help address issues when some random stochastic parameter causes this trouble.

Ideally all the models should have the 2 functions- one to search for NaN and other to replace the NaN. Here is some example implementation.

def is_param_nan(self):
    for param in self.parameters():
        if torch.isnan(param).any():
            return True
    return False

def replace_nans(self):
    for param in self.parameters():
        if torch.isnan(param).any():
            param.data = torch.nan_to_num(param.data)

[KumoLiu]

Hi @a-parida12, thank you for your proposal and interest here.
I'm wondering whether it's a common way to replace the model parameter. Have you come across any popular references? Would you mind sharing them and perhaps we can refer to it?
Thanks!