feat: support moe hf chkpt #486
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for making a pull request! 😃 |
kmehant
force-pushed
the
moe-hf-chkpt-wf
branch
2 times, most recently
from
12f2345 to
9bd210f
Compare
kmehant
requested review from
Ssukriti,
aluu317,
anhuong and
fabianlim
as code owners
Collaborator
|
@kmehant can you fix the lint error |
dushyantbehl
reviewed
Mar 6, 2025
kmehant
force-pushed
the
moe-hf-chkpt-wf
branch
4 times, most recently
from
f049c36 to
e2ad0f0
Compare
willmj
reviewed
Mar 6, 2025
Collaborator
willmj
left a comment
willmj
left a comment
There was a problem hiding this comment.
Thanks Mehant! This looks good to me, but could we update the notes section on Fast MoE to include that this conversion now happens automatically?
Collaborator
|
Also if we could add a small unit test here (maybe similar to the e2e tests we have in test_sft_trainer) to make sure this conversion is happening correctly and inference can be run on the model, that would be good - but it may be hard to find a small moe model to test with. |
kmehant
force-pushed
the
moe-hf-chkpt-wf
branch
2 times, most recently
from
d658990 to
33bbc92
Compare
Collaborator
Author
|
@willmj addressed your comments, thanks |
kmehant
force-pushed
the
moe-hf-chkpt-wf
branch
2 times, most recently
from
a55356e to
573f7a4
Compare
Signed-off-by: Mehant Kammakomati <[email protected]>
Signed-off-by: Mehant Kammakomati <[email protected]>
Signed-off-by: Mehant Kammakomati <[email protected]>
Signed-off-by: Mehant Kammakomati <[email protected]>
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of the change
This is to support automatic HF checkpointing for moe EP training runs using fms-acceleration.
Depends on foundation-model-stack/fms-acceleration#133
Related issue number
#480