Skip to content

Commit bb14995

Browse files
authored
Align AIME pass@1 with literature (#666)
Recent papers like [SimpleRL-Zoo](https://arxiv.org/pdf/2503.18892) and [VAPO](https://arxiv.org/pdf/2504.05118) have adopted `n=32` as the default estimate for AIME24. This PR bumps our default to the same value so we align with what others report.
1 parent 632b69b commit bb14995

File tree

1 file changed

+2
-2
lines changed

1 file changed

+2
-2
lines changed

src/lighteval/tasks/default_tasks.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -325,7 +325,7 @@
325325
generation_size=32768,
326326
metric=[
327327
Metrics.expr_gold_metric,
328-
Metrics.math_pass_at_1_16n,
328+
Metrics.math_pass_at_1_32n,
329329
],
330330
version=1,
331331
)
@@ -342,7 +342,7 @@
342342
generation_size=10000,
343343
metric=[
344344
Metrics.expr_gold_metric,
345-
Metrics.math_pass_at_1_16n,
345+
Metrics.math_pass_at_1_32n,
346346
],
347347
version=1,
348348
)

0 commit comments

Comments
 (0)