Skip to content

Commit 6bd8afc

Browse files
committed
fix: add eps to denominator
1 parent 11c7fc7 commit 6bd8afc

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

pytorch_optimizer/optimizer/nero.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -91,7 +91,7 @@ def step(self, closure: CLOSURE = None) -> LOSS:
9191

9292
bias_correction: float = 1.0 - self.beta ** state['step']
9393

94-
grad_normed = grad / (exp_avg_sq / bias_correction).sqrt() + self.eps
94+
grad_normed = grad / ((exp_avg_sq / bias_correction).sqrt() + self.eps)
9595
grad_normed[torch.isnan(grad_normed)] = 0.0
9696

9797
p.sub_(group['lr'] * state['scale'] * grad_normed)

0 commit comments

Comments
 (0)