@@ -131,26 +131,28 @@ Based on lm-eval-harness.
131
131
| ** Parameter** | ** Default Value** |
132
132
| :---: | :--- |
133
133
| ** model** | gemma |
134
- | ** model_name** | gemma_2b_en |
135
- | ** dtype** | bfloat16 |
136
- | ** num_beams** | 1 |
134
+ | ** model_name** | ** gemma_2b_en** , gemma_7b_en |
135
+ | ** dtype** | ** bfloat16** , float16, float32 |
136
+ | ** num_beams** | ** 1 ** |
137
137
```
138
138
git clone https://github.com/EleutherAI/lm-evaluation-harness.git lm_eval
139
139
cd lm_eval
140
140
git checkout b281b0921b636bc36ad05c0b0b0763bd6dd43463
141
141
git apply ../gemma.patch
142
142
pip install -r requirements.txt
143
+ pip install torch --index-url https://download.pytorch.org/whl/cpu --force-reinstall
144
+ export KERAS_BACKEND=jax
143
145
python main.py \
144
146
--model gemma \
145
- --model_args model_name=gemma_2b_en ,dtype=float32 ,num_beams=1 \
147
+ --model_args model_name=gemma_7b_en ,dtype=bfloat16 ,num_beams=4 \
146
148
--tasks openbookqa \
147
149
--no_cache
148
150
```
149
151
### Output
150
152
```
151
- gemma (model_name=gemma_2b_en ,dtype=float32 ,num_beams=1 ), limit: None, provide_description: False, num_fewshot: 0, batch_size: None
153
+ gemma (model_name=gemma_7b_en ,dtype=bfloat16 ,num_beams=4 ), limit: None, provide_description: False, num_fewshot: 0, batch_size: None
152
154
| Task |Version| Metric |Value| |Stderr|
153
155
|----------|------:|--------|----:|---|-----:|
154
- |openbookqa| 0|acc |0.302 |± |0.0206 |
155
- | | |acc_norm|0.398 |± |0.0219 |
156
+ |openbookqa| 0|acc |0.326 |± |0.0210 |
157
+ | | |acc_norm|0.454 |± |0.0223 |
156
158
```
0 commit comments