Skip to content

Commit 43c4432

Browse files
awaelchliawaelchli
authored
Add automated benchmark table (#1337)
1 parent 57f0a61 commit 43c4432

File tree

1 file changed

+61
-38
lines changed

1 file changed

+61
-38
lines changed

‎config_hub/finetune/README.md‎

Lines changed: 61 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -2,47 +2,70 @@
22

33
The table below lists the performances you can expect from the provided config files. Note that you can achieve lower memory consumption by lowering the micro batch size as needed. In addition, you can lower the rank (`lora_r`) in the LoRA configuration files and disable LoRA for certain layers (for example, setting `lora_projection` and other LoRA layer-specific parameters to `false`).
44
For more information, see the [Dealing with out-of-memory (OOM) errors](../../tutorials/oom.md) on lowering the memory requirements.
5+
The "Cost" column refers to the on-demand compute cost on [Lightning AI](https://lightning.ai) where these benchmarks were executed.
56

67
 
78

8-
| | Size | Dataset | Epochs | Val loss | Peak memory | Max seq length | Micro batch size | Precision | Training runtime |
9-
| --------------------------------- | ---- | --------- | ------ | -------- | ----------- | -------------- | ---------------- | --------- | -------------------|
10-
| | | | | | | | | | |
11-
| falcon-7b/lora.yaml | 7B | Alpaca 2k | 4 | 0.945 | 16.69 GB | 512 | 2 | bfloat16 | 24.88 min (1xA10G) |
12-
| falcon-7b/qlora.yaml | 7B | Alpaca 2k | 4 | 0.993 | 9.44 GB | 512 | 2 | bfloat16 | 50.76 min (1xA10G) |
13-
| | | | | | | | | | |
14-
| gemma-2b/lora.yaml | 2B | Alpaca 2k | 2 | 1.476 | 12.62 GB | 512 | 2 | bfloat16 | 9.29 min (1xA10G) |
15-
| gemma-2b/qlora.yaml | 2B | Alpaca 2k | 2 | 0.981 | 11.59 GB | 512 | 2 | bfloat16 | 12.90 min (1xA10G) |
16-
| gemma-2b/full.yaml | 2B | Alpaca 2k | 0.35 | 0.990 | 17.43 GB | 512 | 1 | bfloat16 | 13.61 min (4xA10G) |
17-
| | | | | | | | | | |
18-
| gemma-7b/lora.yaml | 7B | Alpaca 2k | 2 | 0.903 | 25.30 GB | 512 | 1 | bfloat16 | 11.47 min (1xA100) |
19-
| gemma-7b/qlora.yaml | 7B | Alpaca 2k | 2 | 0.951 | 17.31 GB | 512 | 1 | bfloat16 | 23.46 min (1xA100) |
20-
| | | | | | | | | | |
21-
| llama-2-7b/lora.yaml | 7B | Alpaca 2k | 4 | 0.802 | 19.77 GB | 512 | 2 | bfloat16 | 32.75 min (A10G) |
22-
| llama-2-7b/qlora.yaml | 7B | Alpaca 2k | 4 | 0.814 | 13.68 GB | 512 | 2 | bfloat16 | 45.68 min (A10G) |
23-
| llama-2-7b/full.yaml | 7B | Alpaca 2k | 1 | 0.941 | 26.81 GB | 512 | 4 | bfloat16 | 1.78 min (4xA100) |
24-
| | | | | | | | | | |
25-
| llama-3-8b/lora.yaml | 8B | Alpaca 2k | 2 | 0.890 | 19.73 GB | 512 | 1 | bfloat16 | 14.80 min (A10G) |
26-
| llama-3-8b/qlora.yaml | 8B | Alpaca 2k | 2 | 0.941 | 17.41 GB | 512 | 2 | bfloat16 | 22.34 min (A10G) |
27-
| llama-3-8b/full.yaml | 8B | Alpaca 2k | 1 | 1.451 | 35.48 GB | 512 | 4 | bfloat16 | 2.14 min (4xA100) |
28-
| | | | | | | | | | |
29-
| mistral-7b/lora.yaml (v0.1) | 7B | Alpaca 2k | 4 | 0.796 | 20.65 GB | 512 | 2 | bfloat16 | 31.04 min (1xA10G) |
30-
| mistral-7b/qlora.yaml (v0.1) | 7B | Alpaca 2k | 4 | 0.803 | 14.29 GB | 512 | 2 | bfloat16 | 44.69 min (1xA10G) |
31-
| | | | | | | | | | |
32-
| mistral-7b-v0.2/lora.yaml | 7B | Alpaca 2k | 4 | 0.801 | 20.65 GB | 512 | 2 | bfloat16 | 30.96 min (1xA10G) |
33-
| mistral-7b-v0.2/qlora.yaml | 7B | Alpaca 2k | 4 | 0.813 | 14.29 GB | 512 | 2 | bfloat16 | 44.68 min (1xA10G) |
34-
| | | | | | | | | | |
35-
| phi-2/lora.yaml | 2B | Alpaca 2k | 1 | 0.832 | 13.98 GB | 512 | 4 | bfloat16 | 3.82 min (1xA10G) |
36-
| phi-2/qlora.yaml | 2B | Alpaca 2k | 1 | 0.846 | 14.27 GB | 512 | 4 | bfloat16 | 4.55 min (1xA10G) |
37-
| phi-2/full.yaml | 2B | Alpaca 2k | 1 | 0.937 | 14.44 GB | 512 | 4 | bfloat16 | 13.00 min (1xA10G) |
38-
| | | | | | | | | | |
39-
| stablelm-base-alpha-3b/lora.yaml | 3B | Alpaca 2k | 4 | 1.367 | 8.58 GB | 512 | 2 | bfloat16 | 13.02 min (1xA10G) |
40-
| stablelm-base-alpha-3b/qlora.yaml | 3B | Alpaca 2k | 4 | 1.392 | 5.24 GB | 512 | 2 | bfloat16 | 25.71 min (1xA10G) |
41-
| stablelm-base-alpha-3b/full.yaml | 3B | Alpaca 2k | 1 | 1.494 | 21.23 GB | 512 | 1 | bfloat16 | 72.72 min (2xA10G) |
42-
| | | | | | | | | | |
43-
| tiny-llama/lora.yaml | 1.1B | Alpaca 2k | 3 | 1.038 | 13.50 GB | 512 | 8 | bfloat16 | 8.06 min (1xA10G) |
44-
| tiny-llama/qlora.yaml | 1.1B | Alpaca 2k | 3 | 1.056 | 16.24 GB | 512 | 8 | bfloat16 | 8.74 min (1xA10G) |
45-
| tiny-llama/full.yaml | 1.1B | Alpaca 2k | 1 | 1.105 | 14.10 GB | 512 | 4 | bfloat16 | 2.59 min (1xA10G) |
9+
| Config | Model | Dataset | Epochs | Max seq length | Micro batch size | Precision | Machine | Training runtime | Cost | Peak memory | Validation loss | Validation perplexity |
10+
| --------------------------------- | ---------------------- | -------- | ------ | -------------- | ---------------- | --------- | ------- | ---------------- | ---- | ----------- | --------------- | --------------------- |
11+
| falcon-7b/lora.yaml | falcon-7b | Alpaca2k | 4 | 512 | 1 | bf16-true | 1xA10G | 24.84 min | $0.7 | 16.69 GB | 0.945 | 2.573 |
12+
| falcon-7b/lora.yaml | falcon-7b | Alpaca2k | 4 | 512 | 1 | bf16-true | 4xA10G | 24.94 min | $2.0 | 16.69 GB | 0.945 | 2.573 |
13+
| falcon-7b/qlora.yaml | falcon-7b | Alpaca2k | 4 | 512 | 1 | bf16-true | 1xA10G | 50.85 min | $1.5 | 9.44 GB | 0.993 | 2.699 |
14+
| falcon-7b/qlora.yaml | falcon-7b | Alpaca2k | 4 | 512 | 1 | bf16-true | 4xA10G | 50.88 min | $4.1 | 9.44 GB | 0.993 | 2.699 |
15+
| | | | | | | | | | | | | |
16+
| gemma-2b/full.yaml | gemma-2b | Alpaca2k | 1 | 512 | 1 | bf16-true | 4xA10G | 14.06 min | $1.1 | 17.43 GB | 1.021 | 2.777 |
17+
| gemma-2b/lora.yaml | gemma-2b | Alpaca2k | 2 | 512 | 2 | bf16-true | 1xA10G | 9.41 min | $0.3 | 12.62 GB | 0.981 | 2.666 |
18+
| gemma-2b/lora.yaml | gemma-2b | Alpaca2k | 2 | 512 | 2 | bf16-true | 4xA10G | 9.41 min | $0.8 | 12.62 GB | 0.981 | 2.667 |
19+
| gemma-2b/qlora.yaml | gemma-2b | Alpaca2k | 2 | 512 | 2 | bf16-true | 1xA10G | 12.91 min | $0.4 | 11.58 GB | 1.085 | 2.959 |
20+
| gemma-2b/qlora.yaml | gemma-2b | Alpaca2k | 2 | 512 | 2 | bf16-true | 4xA10G | 12.91 min | $1.0 | 11.59 GB | 1.085 | 2.958 |
21+
| | | | | | | | | | | | | |
22+
| gemma-7b/lora.yaml | gemma-7b | Alpaca2k | 2 | 512 | 1 | bf16-true | 1xA10G | OOM | OOM | OOM | OOM | OOM |
23+
| gemma-7b/lora.yaml | gemma-7b | Alpaca2k | 2 | 512 | 1 | bf16-true | 4xA10G | OOM | OOM | OOM | OOM | OOM |
24+
| gemma-7b/qlora.yaml | gemma-7b | Alpaca2k | 2 | 512 | 1 | bf16-true | 1xA10G | 43.58 min | $1.3 | 17.18 GB | 0.973 | 2.646 |
25+
| gemma-7b/qlora.yaml | gemma-7b | Alpaca2k | 2 | 512 | 1 | bf16-true | 4xA10G | 43.58 min | $3.5 | 17.18 GB | 0.983 | 2.672 |
26+
| | | | | | | | | | | | | |
27+
| llama-2-7b/full.yaml | llama-2-7b | Alpaca2k | 1 | 512 | 4 | bf16-true | 4xA10G | OOM | OOM | OOM | OOM | OOM |
28+
| llama-2-7b/lora.yaml | llama-2-7b | Alpaca2k | 4 | 512 | 2 | bf16-true | 1xA10G | 32.82 min | $1.0 | 19.77 GB | 0.802 | 2.230 |
29+
| llama-2-7b/lora.yaml | llama-2-7b | Alpaca2k | 4 | 512 | 2 | bf16-true | 4xA10G | 32.83 min | $2.6 | 19.77 GB | 0.802 | 2.229 |
30+
| llama-2-7b/qlora.yaml | llama-2-7b | Alpaca2k | 4 | 512 | 2 | bf16-true | 1xA10G | 45.67 min | $1.4 | 13.68 GB | 0.814 | 2.258 |
31+
| llama-2-7b/qlora.yaml | llama-2-7b | Alpaca2k | 4 | 512 | 2 | bf16-true | 4xA10G | 45.69 min | $3.7 | 13.68 GB | 0.815 | 2.258 |
32+
| | | | | | | | | | | | | |
33+
| llama-3-8b/full.yaml | llama-3-8b | Alpaca2k | 1 | 512 | 4 | bf16-true | 4xA10G | OOM | OOM | OOM | OOM | OOM |
34+
| llama-3-8b/lora.yaml | llama-3-8b | Alpaca2k | 2 | 512 | 1 | bf16-true | 1xA10G | 14.79 min | $0.4 | 19.73 GB | 0.888 | 2.431 |
35+
| llama-3-8b/lora.yaml | llama-3-8b | Alpaca2k | 2 | 512 | 1 | bf16-true | 4xA10G | 14.88 min | $1.2 | 19.73 GB | 0.889 | 2.432 |
36+
| llama-3-8b/qlora.yaml | llama-3-8b | Alpaca2k | 2 | 512 | 2 | bf16-true | 1xA10G | 22.24 min | $0.7 | 17.41 GB | 0.939 | 2.558 |
37+
| llama-3-8b/qlora.yaml | llama-3-8b | Alpaca2k | 2 | 512 | 2 | bf16-true | 4xA10G | 22.20 min | $1.8 | 17.41 GB | 0.939 | 2.557 |
38+
| | | | | | | | | | | | | |
39+
| mistral-7b-v0.2/lora.yaml | mistral-7b-v0.2 | Alpaca2k | 4 | 512 | 2 | bf16-true | 1xA10G | 31.00 min | $0.9 | 20.66 GB | 0.801 | 2.228 |
40+
| mistral-7b-v0.2/lora.yaml | mistral-7b-v0.2 | Alpaca2k | 4 | 512 | 2 | bf16-true | 4xA10G | 31.00 min | $2.5 | 20.66 GB | 0.802 | 2.229 |
41+
| mistral-7b-v0.2/qlora.yaml | mistral-7b-v0.2 | Alpaca2k | 4 | 512 | 2 | bf16-true | 1xA10G | 44.75 min | $1.3 | 14.29 GB | 0.813 | 2.255 |
42+
| mistral-7b-v0.2/qlora.yaml | mistral-7b-v0.2 | Alpaca2k | 4 | 512 | 2 | bf16-true | 4xA10G | 44.75 min | $3.6 | 14.29 GB | 0.813 | 2.254 |
43+
| | | | | | | | | | | | | |
44+
| mistral-7b/lora.yaml | mistral-7b | Alpaca2k | 4 | 512 | 2 | bf16-true | 1xA10G | 31.01 min | $0.9 | 20.66 GB | 0.794 | 2.211 |
45+
| mistral-7b/lora.yaml | mistral-7b | Alpaca2k | 4 | 512 | 2 | bf16-true | 4xA10G | 31.03 min | $2.5 | 20.66 GB | 0.796 | 2.218 |
46+
| mistral-7b/qlora.yaml | mistral-7b | Alpaca2k | 4 | 512 | 2 | bf16-true | 1xA10G | 44.75 min | $1.3 | 14.29 GB | 0.803 | 2.231 |
47+
| mistral-7b/qlora.yaml | mistral-7b | Alpaca2k | 4 | 512 | 2 | bf16-true | 4xA10G | 44.81 min | $3.6 | 14.29 GB | 0.803 | 2.233 |
48+
| | | | | | | | | | | | | |
49+
| phi-2/full.yaml | phi-2 | Alpaca2k | 1 | 512 | 4 | bf16-true | 4xA10G | 11.87 min | $1.0 | 14.44 GB | 1.305 | 3.688 |
50+
| phi-2/lora.yaml | phi-2 | Alpaca2k | 1 | 512 | 4 | bf16-true | 1xA10G | 3.78 min | $0.1 | 13.98 GB | 0.819 | 2.269 |
51+
| phi-2/lora.yaml | phi-2 | Alpaca2k | 1 | 512 | 4 | bf16-true | 4xA10G | 3.78 min | $0.3 | 13.98 GB | 0.820 | 2.271 |
52+
| phi-2/qlora.yaml | phi-2 | Alpaca2k | 1 | 512 | 4 | bf16-true | 1xA10G | 4.51 min | $0.1 | 14.27 GB | 0.837 | 2.310 |
53+
| phi-2/qlora.yaml | phi-2 | Alpaca2k | 1 | 512 | 4 | bf16-true | 4xA10G | 4.52 min | $0.4 | 14.27 GB | 0.837 | 2.309 |
54+
| | | | | | | | | | | | | |
55+
| stablelm-base-alpha-3b/full.yaml | stablelm-base-alpha-3b | Alpaca2k | 1 | 512 | 1 | bf16-true | 4xA10G | 70.13 min | $5.6 | 21.23 GB | 1.513 | 4.540 |
56+
| stablelm-base-alpha-3b/lora.yaml | stablelm-base-alpha-3b | Alpaca2k | 4 | 512 | 1 | bf16-true | 1xA10G | 13.07 min | $0.4 | 8.58 GB | 1.361 | 3.900 |
57+
| stablelm-base-alpha-3b/lora.yaml | stablelm-base-alpha-3b | Alpaca2k | 4 | 512 | 1 | bf16-true | 4xA10G | 13.16 min | $1.1 | 8.58 GB | 1.362 | 3.906 |
58+
| stablelm-base-alpha-3b/qlora.yaml | stablelm-base-alpha-3b | Alpaca2k | 4 | 512 | 1 | bf16-true | 1xA10G | 25.86 min | $0.8 | 5.24 GB | 1.388 | 4.009 |
59+
| stablelm-base-alpha-3b/qlora.yaml | stablelm-base-alpha-3b | Alpaca2k | 4 | 512 | 1 | bf16-true | 4xA10G | 25.80 min | $2.1 | 5.24 GB | 1.391 | 4.020 |
60+
| | | | | | | | | | | | | |
61+
| tiny-llama/full.yaml | tiny-llama | Alpaca2k | 1 | 512 | 4 | bf16-true | 1xA10G | 2.58 min | $0.1 | 14.10 GB | 1.088 | 2.968 |
62+
| tiny-llama/full.yaml | tiny-llama | Alpaca2k | 1 | 512 | 4 | bf16-true | 4xA10G | 2.57 min | $0.2 | 14.10 GB | 1.088 | 2.968 |
63+
| tiny-llama/lora.yaml | tiny-llama | Alpaca2k | 3 | 512 | 8 | bf16-true | 1xA10G | 8.09 min | $0.2 | 13.50 GB | 1.039 | 2.826 |
64+
| tiny-llama/qlora.yaml | tiny-llama | Alpaca2k | 3 | 512 | 8 | bf16-true | 1xA10G | 8.70 min | $0.3 | 16.24 GB | 1.056 | 2.874 |
65+
| tiny-llama/qlora.yaml | tiny-llama | Alpaca2k | 3 | 512 | 8 | bf16-true | 4xA10G | 8.70 min | $0.7 | 16.24 GB | 1.056 | 2.874 |
66+
67+
*OOM = Out of memory
68+
4669

4770
 
4871
## Extending the context length

0 commit comments

Comments
 (0)