Independent wd for orthogonalized optimizer #66
Conversation
Signed-off-by: mikail <[email protected]>
| use_nesterov: bool = False, | ||
| weight_decay: float = 0.01, | ||
| use_decoupled_weight_decay: bool = True, | ||
| use_independent_weight_decay: bool = False, |
There was a problem hiding this comment.
Change: The variable name is getting too long. let's make them "use_decoupled_we", "use_independent_wd".
I don't think there are ambiguities around wd being short for weight decay.
There was a problem hiding this comment.
changed name inside optim to use_decoupled_wd and use_independent_wd. The init call still uses the longer name, to not break megatron wrappers
There was a problem hiding this comment.
Megatron can update, there will be a lot of update on dev anyway.
Nothing will break as dependency was made on commit not head of branch
emerging_optimizers/orthogonalized_optimizers/orthogonalized_optimizer.py
Outdated
Show resolved
Hide resolved
emerging_optimizers/orthogonalized_optimizers/orthogonalized_optimizer.py
Outdated
Show resolved
Hide resolved
Signed-off-by: mikail <[email protected]>
| use_nesterov: bool = False, | ||
| weight_decay: float = 0.01, | ||
| use_decoupled_weight_decay: bool = True, | ||
| use_independent_weight_decay: bool = False, |
There was a problem hiding this comment.
Megatron can update, there will be a lot of update on dev anyway.
Nothing will break as dependency was made on commit not head of branch
Signed-off-by: mikail <[email protected]>
Signed-off-by: mikail <[email protected]>
|
/ok to test 50f613b |
Signed-off-by: mikail <[email protected]>
|
/ok to test 417bb0f |
2 participants
Independent weight decay: https://arxiv.org/abs/2510.19093
Enabled by
use_independent_wdwhich is defaulted toFalseto not break prior runs