Skip to content

Commit 2507a89

Browse files
authored
add tasks.yml for Qwen2.5-7B-Instruct models (#20)
* add tasks.yml for Qwen2.5-7B-Instruct models * correct gsm8k metric name
1 parent ace8994 commit 2507a89

File tree

5 files changed

+155
-0
lines changed
  • Qwen/Qwen2.5-7B-Instruct/accuracy
  • RedHatAI
    • Qwen2.5-7B-Instruct-FP8-dynamic/accuracy
    • Qwen2.5-7B-Instruct-quantized.w4a16/accuracy
    • Qwen2.5-7B-Instruct-quantized.w8a8/accuracy
    • Qwen2.5-7B-quantized.w4a16/accuracy

5 files changed

+155
-0
lines changed
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
tasks:
2+
- name: arc_challenge
3+
metrics:
4+
- name: acc_norm,none
5+
value: 0.5939
6+
7+
- name: gsm8k
8+
metrics:
9+
- name: exact_match,strict-match
10+
value: 0.7976
11+
12+
- name: hellaswag
13+
metrics:
14+
- name: acc_norm,none
15+
value: 0.8017
16+
17+
- name: mmlu
18+
metrics:
19+
- name: acc,none
20+
value: 0.7415
21+
22+
- name: truthfulqa_mc2
23+
metrics:
24+
- name: acc,none
25+
value: 0.5637
26+
27+
- name: winogrande
28+
metrics:
29+
- name: acc,none
30+
value: 0.7569
31+
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
tasks:
2+
- name: arc_challenge
3+
metrics:
4+
- name: acc_norm,none
5+
value: 0.6314
6+
7+
- name: gsm8k
8+
metrics:
9+
- name: exact_match,strict-match
10+
value: 0.8006
11+
12+
- name: hellaswag
13+
metrics:
14+
- name: acc_norm,none
15+
value: 0.8111
16+
17+
- name: mmlu
18+
metrics:
19+
- name: acc,none
20+
value: 0.7404
21+
22+
- name: truthfulqa_mc2
23+
metrics:
24+
- name: acc,none
25+
value: 0.6487
26+
27+
- name: winogrande
28+
metrics:
29+
- name: acc,none
30+
value: 0.7443
31+
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
tasks:
2+
- name: arc_challenge
3+
metrics:
4+
- name: acc_norm,none
5+
value: 0.6323
6+
7+
- name: gsm8k
8+
metrics:
9+
- name: exact_match,strict-match
10+
value: 0.8059
11+
12+
- name: hellaswag
13+
metrics:
14+
- name: acc_norm,none
15+
value: 0.8065
16+
17+
- name: mmlu
18+
metrics:
19+
- name: acc,none
20+
value: 0.7319
21+
22+
- name: truthfulqa_mc2
23+
metrics:
24+
- name: acc,none
25+
value: 0.6427
26+
27+
- name: winogrande
28+
metrics:
29+
- name: acc,none
30+
value: 0.7419
31+
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
tasks:
2+
- name: arc_challenge
3+
metrics:
4+
- name: acc_norm,none
5+
value: 0.6323
6+
7+
- name: gsm8k
8+
metrics:
9+
- name: exact_match,strict-match
10+
value: 0.8074
11+
12+
- name: hellaswag
13+
metrics:
14+
- name: acc_norm,none
15+
value: 0.8106
16+
17+
- name: mmlu
18+
metrics:
19+
- name: acc,none
20+
value: 0.7387
21+
22+
- name: truthfulqa_mc2
23+
metrics:
24+
- name: acc,none
25+
value: 0.6458
26+
27+
- name: winogrande
28+
metrics:
29+
- name: acc,none
30+
value: 0.7482
31+
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
tasks:
2+
- name: arc_challenge
3+
metrics:
4+
- name: acc_norm,none
5+
value: 0.587
6+
7+
- name: gsm8k
8+
metrics:
9+
- name: exact_match,strict-match
10+
value: 0.7908
11+
12+
- name: hellaswag
13+
metrics:
14+
- name: acc_norm,none
15+
value: 0.7939
16+
17+
- name: mmlu
18+
metrics:
19+
- name: acc,none
20+
value: 0.7347
21+
22+
- name: truthfulqa_mc2
23+
metrics:
24+
- name: acc,none
25+
value: 0.5548
26+
27+
- name: winogrande
28+
metrics:
29+
- name: acc,none
30+
value: 0.7601
31+

0 commit comments

Comments
 (0)