Set EXPERIMENTAL_1b_CHECKPOINT to True by default

[jwilber]

Description

Sets EXPERIMENTAL_1b_CHECKPOINT to True by default, so that AUC scores are good.

Result for the new checkpoint are:

Data Fraction	FP8 On	FP8 Off
0.05	0.73	0.77
1	0.73	0.74

@jstjohn this still keeps our other checkpoint. We could also optionally change the cell to something like below if we want to hide the original 1b checkpoint, or even update our nemo2_evo2_1b_8k checkpoint to use the new one bf16 ckpt.

%%capture
MODEL_SIZE = "1b"  # also try 7b if you have a GPU with more than 32GB of memory

# Define checkpoint path
if MODEL_SIZE == "1b":
    from bionemo.core.data.load import load

    # Always use the fine-tuned BF16 checkpoint for 1b model
    checkpoint_path = load("evo2/1b-8k-bf16:1.0")
else:
    checkpoint_path = Path(f"nemo2_evo2_{MODEL_SIZE}_8k")

    # Check if the directory does not exist or is empty
    if not checkpoint_path.exists() or not any(checkpoint_path.iterdir()):
        !evo2_convert_to_nemo2 --model-path hf://arcinstitute/savanna_evo2_{MODEL_SIZE}_base --model-size {MODEL_SIZE} --output-dir nemo2_evo2_{MODEL_SIZE}_8k
    else:
        print("Checkpoint directory is not empty. Skipping command.")

Type of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Refactor
• Documentation update
• Other (please describe):

CI Pipeline Configuration

Configure CI behavior by applying the relevant labels:

• SKIP_CI - Skip all continuous integration tests
• INCLUDE_NOTEBOOKS_TESTS - Execute notebook validation tests in pytest
• INCLUDE_SLOW_TESTS - Execute tests labelled as slow in pytest for extensive testing

Note

By default, the notebooks validation tests are skipped unless explicitly enabled.

Authorizing CI Runs

We use copy-pr-bot to manage authorization of CI
runs on NVIDIA's compute resources.

• If a pull request is opened by a trusted user and contains only trusted changes, the pull request's code will
  automatically be copied to a pull-request/ prefixed branch in the source repository (e.g. pull-request/123)
• If a pull request is opened by an untrusted user or contains untrusted changes, an NVIDIA org member must leave an
  /ok to test comment on the pull request to trigger CI. This will need to be done for each new commit.

Usage

TODO: Add code snippet

Pre-submit Checklist

• I have tested these changes locally
• I have updated the documentation accordingly
• I have added/updated tests as needed
• All existing tests pass successfully

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[jstjohn]

This is fine, if you have time maybe remove the if/else logic entirely and just show how to do it with this one fine-tuned checkpoint? Like it's not that experimental at this point. Seems to work pretty well.

[farhadrgh]

LGTM

[jstjohn]

/build-tests

[jwilber]

/ok to test 2d2f873

[jwilber]

/ok to test 3bfebac

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 84.32%. Comparing base (dce2cb2) to head (3bfebac).

✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #840      +/-   ##
==========================================
- Coverage   84.35%   84.32%   -0.04%     
==========================================
  Files         139      139              
  Lines        8719     8719              
==========================================
- Hits         7355     7352       -3     
- Misses       1364     1367       +3     

see 1 file with indirect coverage changes