Adding "spectral clipped" weight decay to be used together with matrix-based preconditioning optimizers#38

skyw merged 4 commits intoNVIDIA-NeMo:mainNVIDIA-NeMo/Emerging-Optimizers:mainfrom

mkhona-nvidia:mkhona/spectral_weight_decaymkhona-nvidia/Emerging-Optimizers:mkhona/spectral_weight_decayCopy head branch name to clipboard