Skip to content

Commit ace8994

Browse files
authored
add tasks.yml for mistralai/Mixtral-8x7B-Instruct-v0.1 (#21)
* add tasks.yml for mistralai/Mixtral-8x7B-Instruct-v0.1 * correct gsm8k metric name * try adjusting server options to avoid server crash
1 parent c956c95 commit ace8994

File tree

2 files changed

+32
-1
lines changed

2 files changed

+32
-1
lines changed

mistralai/Mixtral-8x7B-Instruct-v0.1/accuracy/server.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,4 @@
22
model: "mistralai/Mixtral-8x7B-Instruct-v0.1"
33
trust-remote-code: true
44
add-bos-token: false
5-
tokenizer-mode: "mistral"
5+
enable_chunked_prefill: true
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
tasks:
2+
- name: arc_challenge
3+
metrics:
4+
- name: acc_norm,none
5+
value: 0.7048
6+
7+
- name: gsm8k
8+
metrics:
9+
- name: exact_match,strict-match
10+
value: 0.655
11+
12+
- name: hellaswag
13+
metrics:
14+
- name: acc_norm,none
15+
value: 0.8733
16+
17+
- name: mmlu
18+
metrics:
19+
- name: acc,none
20+
value: 0.703
21+
22+
- name: truthfulqa_mc2
23+
metrics:
24+
- name: acc,none
25+
value: 0.6481
26+
27+
- name: winogrande
28+
metrics:
29+
- name: acc,none
30+
value: 0.8224
31+

0 commit comments

Comments
 (0)