Skip to content

Commit db065c1

Browse files
nvmvlejwilber
andauthored
Add validation tests for ESM2 fine-tuning benchmark partial-conv (#1010)
### Description This PR adds a comprehensive test validation section to the ESM2 fine-tuning benchmark partial-conv configuration. The tests define expected baseline metrics and ranges for model performance validation during CI runs. The validation includes: - Exit code validation (expecting successful completion with code 0) - Performance baselines for key metrics: - `consumed_samples` - `val_loss` - `val_acc` - `TFLOPS per GPU` - `reduced_train_loss` ### Type of changes - [ ] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Refactor - [ ] Documentation update - [ ] Other (please describe): ### CI Pipeline Configuration Configure CI behavior by applying the relevant labels: - [SKIP_CI](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#skip_ci) - Skip all continuous integration tests - [INCLUDE_NOTEBOOKS_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_notebooks_tests) - Execute notebook validation tests in pytest - [INCLUDE_SLOW_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_slow_tests) - Execute tests labelled as slow in pytest for extensive testing > [!NOTE] > By default, the notebooks validation tests are skipped unless explicitly enabled. #### Authorizing CI Runs We use [copy-pr-bot](https://docs.gha-runners.nvidia.com/apps/copy-pr-bot/#automation) to manage authorization of CI runs on NVIDIA's compute resources. * If a pull request is opened by a trusted user and contains only trusted changes, the pull request's code will automatically be copied to a pull-request/ prefixed branch in the source repository (e.g. pull-request/123) * If a pull request is opened by an untrusted user or contains untrusted changes, an NVIDIA org member must leave an `/ok to test` comment on the pull request to trigger CI. This will need to be done for each new commit. ### Pre-submit Checklist - [x] I have tested these changes locally - [ ] I have updated the documentation accordingly - [ ] I have added/updated tests as needed - [x] All existing tests pass successfully Signed-off-by: My Le <mvle@nvidia.com> --------- Signed-off-by: My Le <mvle@nvidia.com> Co-authored-by: Jared Wilber <jwilber@nvidia.com>
1 parent 21b1442 commit db065c1

File tree

1 file changed

+25
-0
lines changed

1 file changed

+25
-0
lines changed

ci/benchmarks/partial-conv/esm2_finetuning.yaml

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,3 +74,28 @@ script: |-
7474
--disable-checkpointing \
7575
--early-stop-on-step=${stop_steps} \
7676
--create-tflops-callback;
77+
tests:
78+
- logic_type: static
79+
logic_spec:
80+
exit_codes:
81+
- 0
82+
baselines:
83+
consumed_samples:
84+
operator: eq
85+
value: 3072000.0
86+
val_loss:
87+
operator: range
88+
max: 0.6678
89+
min: 0.5630
90+
reduced_train_loss:
91+
operator: range
92+
max: 0.1372
93+
min: 0.0993
94+
TFLOPS_per_GPU:
95+
operator: range
96+
max: 719.5984
97+
min: 694.8020
98+
val_acc:
99+
operator: range
100+
max: 0.8462
101+
min: 0.7833

0 commit comments

Comments
 (0)