The training Chinese crnn model appears to have a loss of nan

[htaoruan]

Hi, I am training a model based on crnn_mini-vgg_5e_mj.py configuration with Chinese text recognition dataset, but the model log output loss is incorrect.

My modification is to replace the dictionary with a dictionary in English and Chinese, modified as follows:
dict_file='{{ fileDirname }}/... /... /... /dicts/chinese_english_digits.txt'.

The following error occurs:

2023/04/27 14:33:53 - mmengine - INFO - Epoch(train) [1][100/389] lr: 1.0000e+00 eta: 1:00:21 time: 0.6586 data_time: 0.5103 memory: 5773 loss: inf loss_ctc: inf

[htaoruan]

I tried to reduce the learning rate to prevent this problem, but still no solution

[xReniar]

@htaoruan did you managed to solve your problem?

[xReniar]

In your case you're having loss:inf, but in my case i get this:
09/08 11:40:34 - mmengine - INFO - Epoch(train) [1][ 50/6778] lr: 1.0000e+00 eta: 12:26:54 time: 0.0214 data_time: 0.0010 memory: 191 loss: nan loss_ctc: nan