1.7.6

another fix for training under amp

1.7.5

another fix for training under amp

1.7.4

tweak linear attention

1.7.3

remove weight standardization, as somehow it does not play well with amp

1.7.2

remove weight standardization, as somehow it does not play well with amp

1.7.1

just move towards using rmsnorm, given success of llama. and to avoid…

1.7.0

just move towards using rmsnorm, given success of llama. and to avoid…

1.6.4

move offset noise to only during training, and make the strength over…

1.6.3

adopt offset noise from Nicholas, adopt rescaled cfg from Bytedance

1.6.2

revert for now, something is wrong