Set EXPERIMENTAL_1b_CHECKPOINT to True by default (#840)

### Description
Sets `EXPERIMENTAL_1b_CHECKPOINT` to True by default, so that AUC scores
are good.

Result for the new checkpoint are:

| Data Fraction | FP8 On | FP8 Off |
|---------------|--------|---------|
| 0.05          | 0.73   | 0.77    |
| 1              | 0.73   | 0.74    |


@jstjohn this still keeps our other checkpoint. We could also optionally
change the cell to something like below if we want to hide the original
1b checkpoint, or even update our `nemo2_evo2_1b_8k` checkpoint to use
the new one bf16 ckpt.




```python
%%capture
MODEL_SIZE = "1b"  # also try 7b if you have a GPU with more than 32GB of memory

# Define checkpoint path
if MODEL_SIZE == "1b":
    from bionemo.core.data.load import load

    # Always use the fine-tuned BF16 checkpoint for 1b model
    checkpoint_path = load("evo2/1b-8k-bf16:1.0")
else:
    checkpoint_path = Path(f"nemo2_evo2_{MODEL_SIZE}_8k")

    # Check if the directory does not exist or is empty
    if not checkpoint_path.exists() or not any(checkpoint_path.iterdir()):
        !evo2_convert_to_nemo2 --model-path hf://arcinstitute/savanna_evo2_{MODEL_SIZE}_base --model-size {MODEL_SIZE} --output-dir nemo2_evo2_{MODEL_SIZE}_8k
    else:
        print("Checkpoint directory is not empty. Skipping command.")
```


### Type of changes
<!-- Mark the relevant option with an [x] -->

- [x]  Bug fix (non-breaking change which fixes an issue)
- [ ]  New feature (non-breaking change which adds functionality)
- [ ]  Refactor
- [ ]  Documentation update
- [ ]  Other (please describe):

### CI Pipeline Configuration
Configure CI behavior by applying the relevant labels:

-
[SKIP_CI](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#skip_ci)
- Skip all continuous integration tests
-
[INCLUDE_NOTEBOOKS_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_notebooks_tests)
- Execute notebook validation tests in pytest
-
[INCLUDE_SLOW_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_slow_tests)
- Execute tests labelled as slow in pytest for extensive testing

> [!NOTE]
> By default, the notebooks validation tests are skipped unless
explicitly enabled.


#### Authorizing CI Runs

We use
[copy-pr-bot](https://docs.gha-runners.nvidia.com/apps/copy-pr-bot/#automation)
to manage authorization of CI
runs on NVIDIA's compute resources.

* If a pull request is opened by a trusted user and contains only
trusted changes, the pull request's code will
automatically be copied to a pull-request/ prefixed branch in the source
repository (e.g. pull-request/123)
* If a pull request is opened by an untrusted user or contains untrusted
changes, an NVIDIA org member must leave an
`/ok to test` comment on the pull request to trigger CI. This will need
to be done for each new commit.

### Usage
<!--- How does a user interact with the changed code -->
```python
TODO: Add code snippet
```

### Pre-submit Checklist
<!--- Ensure all items are completed before submitting -->

 - [x] I have tested these changes locally
 - [x] I have updated the documentation accordingly
 - [x] I have added/updated tests as needed
 - [x] All existing tests pass successfully

---------

Signed-off-by: Jared Wilber <jwilber@nvidia.com>
Co-authored-by: John St. John <jstjohn@users.noreply.github.com>

Author: 2 people

sub-packages/bionemo-evo2/examples/zeroshot_brca1.ipynb

@@ -562,16 +562,14 @@
    "outputs": [],
    "source": [
     "%%capture\n",
-    "EXPERIMENTAL_1b_CHECKPOINT: bool = False\n",
     "MODEL_SIZE = \"1b\"  # also try 7b if you have a GPU with more than 32GB of memory\n",
+    "\n",
     "# Define checkpoint path\n",
-    "if EXPERIMENTAL_1b_CHECKPOINT and MODEL_SIZE == \"1b\":\n",
+    "if MODEL_SIZE == \"1b\":\n",
     "    from bionemo.core.data.load import load\n",
     "\n",
-    "    # This is a new 1b checkpoint that has been fine-tuned on BF16 hardware. It should be able to handle FP8 as well.\n",
-    "    #  this line will download the checkpoint from NGC to your $HOME/.cache/bionemo directory and return the path.\n",
-    "    #  alternatively you can use `CHECKPOINT_PATH=$(download_bionemo_data evo2/1b-8k-bf16:1.0)`  to do the same on the\n",
-    "    #  command line.\n",
+    "    #  This line will download the checkpoint from NGC to your $HOME/.cache/bionemo directory and return the path.\n",
+    "    #  To do the same from the command line, use `CHECKPOINT_PATH=$(download_bionemo_data evo2/1b-8k-bf16:1.0)`\n",
     "    checkpoint_path = load(\"evo2/1b-8k-bf16:1.0\")\n",
     "else:\n",
     "    checkpoint_path = Path(f\"nemo2_evo2_{MODEL_SIZE}_8k\")\n",
@@ -1186,7 +1184,7 @@
     "\n",
     "| Model Size  | Dataset Sample Fraction | AUROC |\n",
     "|------------|------------------------|-------|\n",
-    "| Evo 2 1B   | 100%                   | 0.76  |\n",
+    "| Evo 2 1B   | 100%                   | 0.74  |\n",
     "| Evo 2 7B   | 100%                   | 0.87  |\n"
    ]
   }