Skip to content

pytorch-optimizer v2.6.0

Choose a tag to compare

@kozistr kozistr released this 22 Apr 07:56
19dcf2b

Change Log

Feature

  • Implement SM3 optimizer, #130
  • Tweak Scalable Shampoo optimizer, #128, #129
    • implement a new preconditioner type, OUTPUT.
    • optimize speed/memory usage of coupled Newton iteration and power iteration methods.
    • use in-place operation (SQRT-N Grafting).
    • clean-up shampoo_utils more readable.
    • support skip_preconditioning_rank_lt parameter to skip preconditioning in case of the low-rank gradient.
    • set default value for preconditioning_compute_steps to 1000.
    • set default value for start_preconditioning_step to 25.