pytorch-optimizer v2.6.0

Change Log

Feature

• Implement SM3 optimizer, #130
  • Memory-Efficient Adaptive Optimization
• Tweak Scalable Shampoo optimizer, #128, #129
  • implement a new preconditioner type, OUTPUT.
  • optimize speed/memory usage of coupled Newton iteration and power iteration methods.
  • use in-place operation (SQRT-N Grafting).
  • clean-up shampoo_utils more readable.
  • support skip_preconditioning_rank_lt parameter to skip preconditioning in case of the low-rank gradient.
  • set default value for preconditioning_compute_steps to 1000.
  • set default value for start_preconditioning_step to 25.