The confused the concepts of fixed decay and decoupled decay

[zzp1012]

lion-pytorch/lion_pytorch/lion_pytorch.py

Line 79 in 6a74fdc

wd /= init_lr

Decoupled decay refers to isolate the weight decay from the "gradient". The usual way to apply weight decay is to add a L2 regularization in the loss function. For SGD, it is equivalent to directly do direct weight decay on parameters, i.e., $\theta_{t+1} = \theta_t - \eta (\nabla L + \lambda \theta_t)$, however, for adam or other more complex optimizers, the weight decay is hidden in the 'gradients'. See details in the paper https://arxiv.org/abs/1711.05101 .

Here, what you implemented is the fixed decay, i.e., $\theta (1 - \lambda)$.

[lucidrains]

@zzp1012 yes i see, it is confusing

i could add a l2_reg and add back the gradient modification? i had read this blog post and there he removed the l2 reg, so i thought it was ok