File tree Expand file tree Collapse file tree 1 file changed +4
-1
lines changed
Expand file tree Collapse file tree 1 file changed +4
-1
lines changed Original file line number Diff line number Diff line change @@ -72,7 +72,7 @@ This end-to-end example utilizes the common set of interfaces provided by `fms_m
7272
7373 ```bash
7474 lm_eval --model hf \
75- --model_args pretrained="Meta-Llama-3-8B-GPTQ,dtype=float16,autogptq =True,enforce_eager=True" \
75+ --model_args pretrained="Meta-Llama-3-8B-GPTQ,dtype=float16,gptqmodel=True =True,enforce_eager=True" \
7676 --tasks lambada_openai \
7777 --num_fewshot 5 \
7878 --device cuda:0 \
@@ -82,18 +82,21 @@ This end-to-end example utilizes the common set of interfaces provided by `fms_m
8282## Example Test Results
8383
8484- Unquantized Model
85+ -
8586|Model | Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr|
8687|------------|--------------|------:|------|-----:|----------|---|-----:|---|-----:|
8788| LLAMA3-8B |lambada_openai| 1|none | 5|acc |↑ |0.7103|± |0.0063|
8889| | | |none | 5|perplexity|↓ |3.7915|± |0.0727|
8990
9091- Quantized model with the settings showed above (`desc_act` default to False.)
92+ -
9193|Model | Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr|
9294|------------|--------------|------:|------|-----:|----------|---|------:|---|-----:|
9395| LLAMA3-8B |lambada_openai| 1|none | 5|acc |↑ |0.6365 |± |0.0067|
9496| | | |none | 5|perplexity|↓ |5.9307 |± |0.1830|
9597
9698- Quantized model with `desc_act` set to `True` (could improve the model quality, but at the cost of inference speed.)
99+ -
97100|Model | Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr|
98101|------------|--------------|------:|------|-----:|----------|---|------:|---|-----:|
99102| LLAMA3-8B |lambada_openai| 1|none | 5|acc |↑ |0.6193 |± |0.0068|
You can’t perform that action at this time.
0 commit comments