Skip to content

Commit f96954d

Browse files
Adding baseline metrics for benchmarking ESM2 model (#831)
…ESM2 model ### Description <!-- Provide a detailed description of the changes in this PR --> The PR will update the baseline for some performance metrics - consumed_samples - val_loss - reduced_train_loss - tflops_per_sec_per_gpu These metrics can be used to validate the actual metrics from actual executions. ### Type of changes <!-- Mark the relevant option with an [x] --> - [ ] Bug fix (non-breaking change which fixes an issue) - [x ] New feature (non-breaking change which adds functionality) - [ ] Refactor - [ ] Documentation update - [ ] Other (please describe): ### CI Pipeline Configuration Configure CI behavior by applying the relevant labels: - [SKIP_CI](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#skip_ci) - Skip all continuous integration tests - [INCLUDE_NOTEBOOKS_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_notebooks_tests) - Execute notebook validation tests in pytest - [INCLUDE_SLOW_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_slow_tests) - Execute tests labelled as slow in pytest for extensive testing > [!NOTE] > By default, the notebooks validation tests are skipped unless explicitly enabled. #### Authorizing CI Runs We use [copy-pr-bot](https://docs.gha-runners.nvidia.com/apps/copy-pr-bot/#automation) to manage authorization of CI runs on NVIDIA's compute resources. * If a pull request is opened by a trusted user and contains only trusted changes, the pull request's code will automatically be copied to a pull-request/ prefixed branch in the source repository (e.g. pull-request/123) * If a pull request is opened by an untrusted user or contains untrusted changes, an NVIDIA org member must leave an `/ok to test` comment on the pull request to trigger CI. This will need to be done for each new commit. ### Usage <!--- How does a user interact with the changed code --> No interaction, just the baseline to check ### Pre-submit Checklist <!--- Ensure all items are completed before submitting --> - [x] I have tested these changes locally - [x] I have updated the documentation accordingly - [x] I have added/updated tests as needed - [x] All existing tests pass successfully --------- Signed-off-by: Truong Nguyen <tgnguyen@nvidia.com> Co-authored-by: Dorota Toczydlowska <115542912+dorotat-nv@users.noreply.github.com>
1 parent 0342382 commit f96954d

File tree

1 file changed

+21
-0
lines changed

1 file changed

+21
-0
lines changed

ci/benchmarks/partial-conv/esm2_pretrain.yaml

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,3 +46,24 @@ script: |-
4646
--wandb-job-type=${pipeline_label} \
4747
--log-every-n-steps=50 \
4848
--disable-checkpointing;
49+
tests:
50+
- logic_type: static
51+
logic_spec:
52+
exit_codes:
53+
- 0
54+
baselines:
55+
consumed_samples:
56+
operator: eq
57+
value: 13568000.0
58+
val_loss:
59+
operator: range
60+
max: 2.5906
61+
min: 2.4833
62+
reduced_train_loss:
63+
operator: range
64+
max: 2.5640
65+
min: 2.4578
66+
tflops_per_sec_per_gpu:
67+
operator: range
68+
max: 157.9352
69+
min: 141.2835

0 commit comments

Comments
 (0)