-
Notifications
You must be signed in to change notification settings - Fork 378
Open
Description
NaN or Inf found in input tensor.
{'loss': 1.4681, 'grad_norm': nan, 'learning_rate': 6.224324660774306e-10, 'epoch': 0.0}
0%| | 20/160654 [01:15<111:16:43, 2.49s/it]NaN or Inf found in input tensor.
{'loss': 0.0, 'grad_norm': nan, 'learning_rate': 1.2448649321548611e-09, 'epoch': 0.0}
0%| | 30/160654 [01:49<148:57:38, 3.34s/it]NaN or Inf found in input tensor.
{'loss': 0.0, 'grad_norm': nan, 'learning_rate': 1.867297398232292e-09, 'epoch': 0.0}
0%| | 40/160654 [02:17<137:47:56, 3.09s/it]NaN or Inf found in input tensor.
{'loss': 0.0, 'grad_norm': nan, 'learning_rate': 2.4897298643097223e-09, 'epoch': 0.0}
Has anyone else encountered this issue? I did not change any setting in the qwen2_5_vl_full_sft.yaml
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels