Skip to content

Commit c776694

Browse files
authored
fix gsm8k metric (#688)
replaces the metric for gsm8k and the stop words to yield better results.
1 parent b07d418 commit c776694

File tree

1 file changed

+5
-3
lines changed

1 file changed

+5
-3
lines changed

src/lighteval/tasks/default_tasks.py

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7874,7 +7874,7 @@
78747874
few_shots_select="random_sampling_from_train",
78757875
generation_size=256,
78767876
metric=[Metrics.quasi_exact_match_gsm8k],
7877-
stop_sequence=["Question=", "Question", "="],
7877+
stop_sequence=["Question:"],
78787878
trust_dataset=True,
78797879
version=0,
78807880
)
@@ -7889,8 +7889,10 @@
78897889
few_shots_split=None,
78907890
few_shots_select="random_sampling_from_train",
78917891
generation_size=256,
7892-
metric=[Metrics.quasi_exact_match_gsm8k, Metrics.maj_at_8_gsm8k],
7893-
stop_sequence=["Question="],
7892+
metric=[
7893+
Metrics.expr_gold_metric,
7894+
],
7895+
stop_sequence=["Question:"],
78947896
trust_dataset=True,
78957897
version=0,
78967898
)

0 commit comments

Comments
 (0)