Skip to content

Commit 70acb85

Browse files
clefourrierclefourrier
authored
Fix 999: always provide parameters in the metric name to allow using several combinations (#1017)
* fix * added a warning message * fix unit tests * fix unit tests 2 * mini fix * minifix * test * update new metrics name * updated var names
1 parent 3af8925 commit 70acb85

File tree

65 files changed

+151
-148
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

65 files changed

+151
-148
lines changed

‎src/lighteval/metrics/utils/metric_utils.py‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -83,15 +83,18 @@ def __call__(self, sample_params: dict | None):
8383

8484
# Once the parameters are updated, we need to adjust the
8585
# metric name to what will be returned
86-
sample_params_name = "&".join(sample_params.keys())
86+
# CAREFUL: do not change the following logic!
87+
# It must always provide the values of all parameters, so that people can evaluate using a range of metrics
88+
# For example, pass@k=1&n=16, pass@k=10&n=16, etc
89+
sample_params_name = "&".join(f"{k}={v}" for k, v in sample_params.items())
8790
if isinstance(self, MetricGrouping):
8891
if hasattr(self.sample_level_fn, "metric_names"):
8992
# this is mostly for the gpass@k metrics
9093
self.metric_name = self.sample_level_fn.metric_names
9194
else:
92-
self.metric_name = [f"{metric}_with_{sample_params_name}" for metric in self.metric_name]
95+
self.metric_name = [f"{metric}:{sample_params_name}" for metric in self.metric_name]
9396
else:
94-
self.metric_name = f"{self.metric_name}_with_{sample_params_name}"
97+
self.metric_name = f"{self.metric_name}:{sample_params_name}"
9598
return self
9699

97100
@staticmethod

‎tests/reference_details/SmolLM2-1.7B-Instruct-transformers/details_leaderboard|arc:challenge|25_2025-09-19T14-21-59.670987.parquet‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
version https://git-lfs.github.com/spec/v1
2-
oid sha256:bf8c0061d55f76265ec3d88762e4d806ce2d932909384cd03637413fd5cb89be
3-
size 88248
2+
oid sha256:d2dce4416d022cb704a77d63dcbacc99e148cb598186f88f33e7b1c5c019335e
3+
size 87199

‎tests/reference_details/SmolLM2-1.7B-Instruct-transformers/details_leaderboard|hellaswag|10_2025-09-19T14-21-59.670987.parquet‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
version https://git-lfs.github.com/spec/v1
2-
oid sha256:f95752c0d186667610852bcaf34cdaa0aa0e1114bff50dc8a10244c31610eb3d
3-
size 107042
2+
oid sha256:8ac904dbbbd26b93de90df7400242713a359207985d5f4c4f75d31ee9bb3325f
3+
size 106015

‎tests/reference_details/SmolLM2-1.7B-Instruct-transformers/details_leaderboard|mmlu:college_chemistry|5_2025-09-19T14-21-59.670987.parquet‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
version https://git-lfs.github.com/spec/v1
2-
oid sha256:b53028865bed1ef182d8f797f0bf2ad189c512e2bb3042a2469ec202068ae22b
3-
size 37410
2+
oid sha256:e52b3dd01e79fa7028396bad84f6fba4d653fe6ede17a74cf1829115f809fdbe
3+
size 36114

‎tests/reference_details/SmolLM2-1.7B-Instruct-transformers/details_leaderboard|mmlu:us_foreign_policy|5_2025-09-19T14-21-59.670987.parquet‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
version https://git-lfs.github.com/spec/v1
2-
oid sha256:6776849a962942fe0484967b11a0dc229bd89a132c4dfdda93923ff53bebaff5
3-
size 37924
2+
oid sha256:73de608e18e75e21cd832c09aecd13f6e7a0dbb91f113cb4cb6f8984be474d77
3+
size 36635

‎tests/reference_details/SmolLM2-1.7B-Instruct-transformers/details_leaderboard|truthfulqa:mc|0_2025-09-19T14-21-59.670987.parquet‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
version https://git-lfs.github.com/spec/v1
2-
oid sha256:bacac56e0ed5c6c65642e1c610cbcdf4ea20d8f41160942198f25ab6bf04c99a
3-
size 26097
2+
oid sha256:dc795a85bcb77084b1275bfadfe2c613a3b44543a6184e3ffd32bc4588d8d64f
3+
size 25269

‎tests/reference_details/SmolLM2-1.7B-Instruct-transformers/details_lighteval|agieval:aqua-rat|0_2025-09-19T14-21-59.670987.parquet‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
version https://git-lfs.github.com/spec/v1
2-
oid sha256:eec7ced557be46cd1f3a54f8f51109512af7c0912b735d46adc63e1bdf0db21f
3-
size 22385
2+
oid sha256:2e75e6460dd0c3ba833b74c19b4943b1baa0f266e5207895454a54019dc9cbf6
3+
size 21944

‎tests/reference_details/SmolLM2-1.7B-Instruct-transformers/details_lighteval|agieval:logiqa-en|0_2025-09-19T14-21-59.670987.parquet‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
version https://git-lfs.github.com/spec/v1
2-
oid sha256:e7ffedbe7a6ae8de9c9a58e35e7510a547eec23aac3dbd7d478ad78f7d21d280
3-
size 34634
2+
oid sha256:6c96e81a70ef68946e7e83e30a9ef5dd5c04a4e8de215a021de33d4e841ec502
3+
size 34133

‎tests/reference_details/SmolLM2-1.7B-Instruct-transformers/details_lighteval|agieval:lsat-ar|0_2025-09-19T14-21-59.670987.parquet‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
version https://git-lfs.github.com/spec/v1
2-
oid sha256:f337a4e40a4bd6c4f2473f7bc70c1c604c068f582bebc1d6625c725cef74f2a3
3-
size 31273
2+
oid sha256:ebf20030a92a27e15144e4f2071c419edafd1ae9d0e8fe7b9bc38a3edf7a181e
3+
size 30775

‎tests/reference_details/SmolLM2-1.7B-Instruct-transformers/details_lighteval|agieval:lsat-lr|0_2025-09-19T14-21-59.670987.parquet‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
version https://git-lfs.github.com/spec/v1
2-
oid sha256:7fe3574ba3da54027a4a76049a166c5791820d3cbaef079d58f63ec6be61ead4
3-
size 39946
2+
oid sha256:01db21e17415bb49be149cf25da813faadfb6bac3b127ba246ae3dbcf96685d7
3+
size 39431

0 commit comments

Comments
 (0)