Struggling to find the optimizer code

[nicolas-dufour]

Hi can you point out to where we can find the optimizer code?
Would be nice to have a self contained code we can port elsewhere as well!

Thanks!

[1BIMU]

It is available on SSO Pretrain, which only support for Megatron.
I also suggest the author releases SSO in pytorch in the future, which would raise his work's contribution a lot.