Skip to content

ESM2 PEFT#766

Merged
polinabinder1 merged 37 commits intomainfrom
pbinder/auto_resume
Apr 25, 2025
Merged

ESM2 PEFT#766
polinabinder1 merged 37 commits intomainfrom
pbinder/auto_resume

Conversation

@polinabinder1
Copy link
Collaborator

@polinabinder1 polinabinder1 commented Mar 19, 2025

PEFT checkpointing and inference for esm2.

@polinabinder1
changes and files for peft
5543a7c 
Signed-off-by: Polina Binder <pbinder@nvidia.com>
@polinabinder1
trainer changes
ce5e138 
Signed-off-by: Polina Binder <pbinder@nvidia.com>
@polinabinder1
scripts for fine-tuning and inference with PEFT
a9549e3 
Signed-off-by: Polina Binder <pbinder@nvidia.com>
@polinabinder1
test cases for esm2
18999ba 
Signed-off-by: Polina Binder <pbinder@nvidia.com>
@polinabinder1 polinabinder1 marked this pull request as draft March 19, 2025 03:01
@polinabinder1
Merge branch 'main' into pbinder/auto_resume
a3e4011 
Signed-off-by: polinabinder1 <pbinder@nvidia.com>
@polinabinder1
correct pre-commit
fd2d7c3 
Signed-off-by: Polina Binder <pbinder@nvidia.com>
@polinabinder1
reverse nemo changes
bb60ef2 
Signed-off-by: Polina Binder <pbinder@nvidia.com>
@codecov-commenter
Copy link

codecov-commenter commented Mar 19, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 92.30769% with 4 lines in your changes missing coverage. Please review.

Project coverage is 84.39%. Comparing base (6568d2b) to head (4fe7621).

✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...emo-esm2/src/bionemo/esm2/scripts/finetune_esm2.py 88.88% 2 Missing ⚠️
...onemo-esm2/src/bionemo/esm2/model/finetune/peft.py 94.44% 1 Missing ⚠️
...ionemo-esm2/src/bionemo/esm2/scripts/infer_esm2.py 92.30% 1 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #766      +/-   ##
==========================================
+ Coverage   84.33%   84.39%   +0.06%     
==========================================
  Files         137      138       +1     
  Lines        8626     8685      +59     
==========================================
+ Hits         7275     7330      +55     
- Misses       1351     1355       +4
Files with missing lines Coverage Δ
...kages/bionemo-esm2/src/bionemo/esm2/model/model.py 92.00% <ø> (ø)
...-packages/bionemo-llm/src/bionemo/llm/lightning.py 91.97% <100.00%> (+0.15%) ⬆️
...bionemo-llm/src/bionemo/llm/model/biobert/model.py 91.66% <ø> (ø)
...onemo-esm2/src/bionemo/esm2/model/finetune/peft.py 96.66% <94.44%> (ø)
...ionemo-esm2/src/bionemo/esm2/scripts/infer_esm2.py 59.42% <92.30%> (+4.42%) ⬆️
...emo-esm2/src/bionemo/esm2/scripts/finetune_esm2.py 92.45% <88.88%> (-0.51%) ⬇️

trvachov
trvachov reviewed Mar 20, 2025
View reviewed changes
@trvachov
Copy link
Collaborator

trvachov commented Mar 20, 2025

The resulting model finetuned with LoRA has this file...do you know what it's supposed to have in it?

cat /results/lora_model/checkpoint-step\=999-consumed_samples\=64000.0-last/weights/adapter_metadata.json
{"model_ckpt_path": "None"}

@copy-pr-bot
Copy link

copy-pr-bot bot commented Mar 31, 2025

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@polinabinder1
seed as an argument
68e62fc 
Signed-off-by: Polina Binder <pbinder@nvidia.com>
@polinabinder1
Merge remote-tracking branch 'origin/main' into pbinder/auto_resume
903035b
@polinabinder1
test file changes
01b02ef 
Signed-off-by: Polina Binder <pbinder@nvidia.com>
@polinabinder1
resumption test case
034eaf7 
Signed-off-by: Polina Binder <pbinder@nvidia.com>
@polinabinder1
debugging inference
eb9db7b 
Signed-off-by: Polina Binder <pbinder@nvidia.com>
@polinabinder1
experimetning with inference auto resume
7c2b2b4 
Signed-off-by: Polina Binder <pbinder@nvidia.com>
@polinabinder1
Update conftest.py
15aa844 
Signed-off-by: polinabinder1 <pbinder@nvidia.com>
@polinabinder1
Merge branch 'main' into pbinder/auto_resume
48c91e9
@polinabinder1
Copy link
Collaborator Author

polinabinder1 commented Apr 23, 2025

/ok to test 48c91e9

@polinabinder1 polinabinder1 enabled auto-merge April 23, 2025 16:41
@polinabinder1
adding correct ngc location formatting
b9db5a2 
Signed-off-by: Polina Binder <pbinder@nvidia.com>
@polinabinder1
updating inference notebook
68bda48 
Signed-off-by: Polina Binder <pbinder@nvidia.com>
@polinabinder1
Copy link
Collaborator Author

polinabinder1 commented Apr 24, 2025

/ok to test 68bda48

@polinabinder1
correct ngc info
0b82299 
Signed-off-by: Polina Binder <pbinder@nvidia.com>
@polinabinder1
Copy link
Collaborator Author

polinabinder1 commented Apr 24, 2025

/ok to test 0b82299

@polinabinder1
removing a test case that does not run well with megatron environemen…
c6bc517 
…ts. This path is tested elsewhere.

Signed-off-by: Polina Binder <pbinder@nvidia.com>
@polinabinder1
Copy link
Collaborator Author

polinabinder1 commented Apr 24, 2025

/ok to test c6bc517

@polinabinder1 polinabinder1 added this pull request to the merge queue Apr 24, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 25, 2025
@polinabinder1
adding correct notebooks
6f9712a 
Signed-off-by: Polina Binder <pbinder@nvidia.com>
@polinabinder1
Copy link
Collaborator Author

polinabinder1 commented Apr 25, 2025

/ok to test 6f9712a

@polinabinder1 polinabinder1 enabled auto-merge April 25, 2025 01:46
@polinabinder1 polinabinder1 added this pull request to the merge queue Apr 25, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 25, 2025
@polinabinder1
adding notebook
8708714 
Signed-off-by: Polina Binder <pbinder@nvidia.com>
@polinabinder1
Merge branch 'main' into pbinder/auto_resume
4fe7621
@polinabinder1
Copy link
Collaborator Author

polinabinder1 commented Apr 25, 2025

/ok to test 4fe7621

@polinabinder1 polinabinder1 enabled auto-merge April 25, 2025 04:53
@polinabinder1 polinabinder1 added this pull request to the merge queue Apr 25, 2025
Merged via the queue into main with commit 192e537 Apr 25, 2025
10 checks passed
@polinabinder1 polinabinder1 deleted the pbinder/auto_resume branch April 25, 2025 07:21
@trvachov trvachov changed the title Pbinder/auto resume ESM2 PEFT Apr 26, 2025
cspades pushed a commit that referenced this pull request May 4, 2025
@polinabinder1 @cspades
Pbinder/auto resume (#766)
a6fb55b 
PEFT checkpointing and inference for esm2.

---------

Signed-off-by: Polina Binder <pbinder@nvidia.com>
Signed-off-by: polinabinder1 <pbinder@nvidia.com>
Signed-off-by: Cory Ye <cye@nvidia.com>
farhadrgh pushed a commit that referenced this pull request May 5, 2025
@polinabinder1 @farhadrgh
Pbinder/auto resume (#766)
26684c5 
PEFT checkpointing and inference for esm2.

---------

Signed-off-by: Polina Binder <pbinder@nvidia.com>
Signed-off-by: polinabinder1 <pbinder@nvidia.com>
Signed-off-by: Farhad Ramezanghorbani <farhadr@nvidia.com>
camirr-nv pushed a commit that referenced this pull request Jun 26, 2025
@polinabinder1 @camirr-nv
Pbinder/auto resume (#766)
f5fd06e 
PEFT checkpointing and inference for esm2.

---------

Signed-off-by: Polina Binder <pbinder@nvidia.com>
Signed-off-by: polinabinder1 <pbinder@nvidia.com>
Signed-off-by: Ubuntu <camirr@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@farhadrgh farhadrgh farhadrgh approved these changes

@trvachov trvachov trvachov approved these changes

@jstjohn jstjohn Awaiting requested review from jstjohn jstjohn is a code owner

@malcolmgreaves malcolmgreaves Awaiting requested review from malcolmgreaves

@skothenhill-nv skothenhill-nv Awaiting requested review from skothenhill-nv skothenhill-nv is a code owner

@pstjohn pstjohn Awaiting requested review from pstjohn pstjohn is a code owner

@jomitchellnv jomitchellnv Awaiting requested review from jomitchellnv jomitchellnv is a code owner

@sichu2023 sichu2023 Awaiting requested review from sichu2023

@dorotat-nv dorotat-nv Awaiting requested review from dorotat-nv dorotat-nv is a code owner

@jwilber jwilber Awaiting requested review from jwilber jwilber is a code owner

@cspades cspades Awaiting requested review from cspades cspades is a code owner

@DejunL DejunL Awaiting requested review from DejunL DejunL is a code owner

@guoqing-zhou guoqing-zhou Awaiting requested review from guoqing-zhou

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@polinabinder1 @codecov-commenter @trvachov @farhadrgh