Add support for the normalized and orthogonal optimizers with matrix sign #28
Description
Is your feature request related to a problem? Please describe.
The optimizer for normalized layers is introduced by Franz Cesista in https://leloykun.github.io/ponder/steepest-descent-stiefel/#6-bonus-a-muon-like-optimizer-for-the-embedding-and-unembedding-layers
and the optimizer for stiefel is introduced by Jianlin Su in https://kexue.fm/archives/11221
Further, Tilde has introduced a set of optimizers relaxing the strong constraints: https://www.tilderesearch.com/vignettes/gram-space
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.