Fix LR not being correctly set after using LearningRateFinder callback
#21068
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…en used as callback Previously, LearningRateFinder applied the suggested LR before restoring the checkpoint, so the optimizer LR was reverted by the restore step. This caused the callback to print “Learning rate set to …” without persisting the change. Change: - Move LR application to after checkpoint restore and update both the LM attr and active optimizer param groups so the LR persists for training. Tests: - Add unit test [test_lr_finder_callback_applies_lr_after_restore] to assert the optimizer LR matches the LR Finder suggestion after the search completes. Fixes Lightning-AI#21030
littlebullGit
requested review from
Borda,
ethanwharris,
justusschock,
lantiga and
tchaton
as code owners
SkafteNicki
reviewed
Aug 14, 2025
Borda
changed the title
LearningRateFinder callback
Co-authored-by: Nicki Skafte Detlefsen <[email protected]>
for more information, see https://pre-commit.ci
SkafteNicki
approved these changes
Aug 14, 2025
|
Added #16787 to issue description as it will be fixed by this PR |
|
Error: The timeout of 60 minutes has triggered but not all required jobs were passing. This job will need to be re-run to merge your PR. If you do not have write access to the repository you can ask Lightning-AI/lai-frameworks to re-run it for you. If you have any other questions, you can reach out to carmocca for help. The failed check of "Probot/required-jobs" seems a timeout and has nothing to do with the fix. Can you retrigger the check ? |
yes this is an aggregation check which, expects that all particular checks finish within an hour so in case something takes longer, it needs to be restarted |
Borda
approved these changes
Aug 15, 2025
Borda
pushed a commit
that referenced
this pull request
Aug 28, 2025
…ack (#21068) * fix(tuner/lr_finder): apply LR suggestion after checkpoint restore when used as callback Previously, LearningRateFinder applied the suggested LR before restoring the checkpoint, so the optimizer LR was reverted by the restore step. This caused the callback to print “Learning rate set to …” without persisting the change. Change: - Move LR application to after checkpoint restore and update both the LM attr and active optimizer param groups so the LR persists for training. Tests: - Add unit test [test_lr_finder_callback_applies_lr_after_restore] to assert the optimizer LR matches the LR Finder suggestion after the search completes. * changelog * Apply suggestions from code review --------- Co-authored-by: Nicki Skafte Detlefsen <[email protected]> Co-authored-by: Jirka Borovec <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit 3ed9d4e)
lantiga
pushed a commit
that referenced
this pull request
Aug 29, 2025
…ack (#21068) * fix(tuner/lr_finder): apply LR suggestion after checkpoint restore when used as callback Previously, LearningRateFinder applied the suggested LR before restoring the checkpoint, so the optimizer LR was reverted by the restore step. This caused the callback to print “Learning rate set to …” without persisting the change. Change: - Move LR application to after checkpoint restore and update both the LM attr and active optimizer param groups so the LR persists for training. Tests: - Add unit test [test_lr_finder_callback_applies_lr_after_restore] to assert the optimizer LR matches the LR Finder suggestion after the search completes. * changelog * Apply suggestions from code review --------- Co-authored-by: Nicki Skafte Detlefsen <[email protected]> Co-authored-by: Jirka Borovec <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit 3ed9d4e)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Title
fix(tuner/lr_finder): apply LR suggestion after checkpoint restore when used as callback [Fixes #21030]
Fixes #16787
Summary
Tests
Files changed
Breaking changes
Docs/Changelog
Fixes
📚 Documentation preview 📚: https://pytorch-lightning--21068.org.readthedocs.build/en/21068/