You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: program-data-separation/cpp/lora_example/README.md
+73-26Lines changed: 73 additions & 26 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,16 +4,15 @@ This directory contains the C++ code for the LoRA demo.
4
4
5
5
You'll learn how to:
6
6
1. Export LoRA PTE files that share a single foundation weight file.
7
-
2. Load and run the LoRA PTE files, and notice that the runtime memory is not doubled as the foundation weights are shared.
7
+
2. Load and run the LoRA PTE files, and notice that the runtime memory increases by the LoRA adapter size (small) and not the foundation weight size (large), because the foundation weights are shared.
8
8
9
9
Note:
10
10
- Weight-sharing is supported with the XNNPACK backend.
11
-
- Quantization (outside of embedding quantization) is not supported when weight-sharing.
11
+
- Quantization (outside of embedding quantization) is currently not supported when weight-sharing.
12
12
- There are many ways to fine-tune LoRA adapters. We will go through a few examples to create a demo.
13
13
14
14
## Size savings.
15
15
16
-
Size results will vary depending on the model and LoRA config. For this demo, we save ~5GB of disk space by storing weights in a separate, sharable file and ~5GB runtime memory by sharing weights at runtime through the XNNPACK weight cache. Detailed results are below.
17
16
Size results will vary depending on the model and LoRA config. For this demo, we save ~5GB of disk space by storing weights in a separate, sharable file and ~5GB runtime memory by sharing weights at runtime through the XNNPACK weight cache. Detailed results are below.
18
17
19
18
### XNNPACK weight sharing.
@@ -24,16 +23,15 @@ The XNNPACK backend is a singleton. Weight sharing is implemented via the XNNPAC
24
23
Download pre-trained dummy adapter to export and run along with a regular Llama-3-2-1B model.
25
24
26
25
## Fine-tune from scratch with Unsloth and Llama-3-2-1B.
27
-
We can use [Unsloth](https://unsloth.ai/), a popular tool to finetune and train LLMs, to create our LoRA adapters. Unsloth provides a [colab notebook](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/datasets-guide#synthetic-dataset-notebook) that showcases how to generate data using the Meta Synthetic Data Kit.
28
-
29
-
The training notebook takes a few shortcuts to reduce the latency/compute. You can change these settings for better results.
30
-
1. Play around with the chunk sizes and overlap to see what works best for your dataset.
31
-
2. The notebook trains on the last three data files generated; increase this for better coverage of your dataset.
32
-
3. At the training step, the notebook uses max_steps=60 to speed things up. Setting num_train_epochs=1 (or greater) for a full run and max_steps=None has better results.
26
+
[Unsloth](https://unsloth.ai/) provides a [colab notebook](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/datasets-guide#synthetic-dataset-notebook) that showcases how to generate data using the Meta Synthetic Data Kit, and then fine-tune it to create a LoRA adapter.
33
27
34
28
For this demo, we trained on two datasets:
35
-
1. executorch/docs/source: an adapter with domain knowledge of executorch. Using Meta Synthetic Data Kit, you can generate qa pairs based on the executorch documentation.
36
-
2. Recent Nobel prize winners (2024-2025): an adapter with knowledge beyond the cutoff date of Llama-3-2-1B. This data was taken from [Wikipedia](https://en.wikipedia.org/wiki/List_of_Nobel_laureates).
29
+
1. executorch/docs/source: an adapter with domain knowledge of executorch. This used Meta Synthetic Data Kit to generate qa pairs based on the documentation.
30
+
2. Recent Nobel prize winners (2024-2025): an adapter with knowledge beyond the cutoff date of Llama-3-2-1B. This data was taken from [Wikipedia](https://en.wikipedia.org/wiki/List_of_Nobel_laureates), and formatted into the chat template for training.
31
+
32
+
The training notebook takes a few shortcuts to reduce the latency/compute. You can change these settings for better results.
33
+
1. When generating data, play around with the chunk sizes and overlap to see what works best for your dataset.
34
+
2. At the training step, the notebook uses max_steps=60 to speed things up. Setting num_train_epochs=1 (or greater) for a full run and max_steps=None has better results.
37
35
38
36
Unsloth will output the adapter artifacts to the specified directory (in the colab notebook, 'lora_model/'). You will see a few files like such:
Or [install from source](https://docs.pytorch.org/executorch/stable/using-executorch-building-from-source.html#install-executorch-pip-package-from-source).
71
69
72
70
```
73
-
# Clone the ExecuTorch repo from GitHub.
74
-
git clone https://github.com/pytorch/executorch.git && cd executorch
71
+
# Move to the executorch subdirectory
72
+
cd ~/executorch-examples/program-data-separation/cpp/executorch
73
+
74
+
# Update to recent main.
75
+
git pull origin/main
75
76
76
77
# Install ExecuTorch pip package.
77
78
./install_executorch.sh --editable
78
79
```
79
80
80
-
NOTE: some features are not available in executorch==1.0.0, use main or a recent nightly.
81
+
NOTE: use main or a recent nightly, as some features are not available in executorch==1.0.0.
81
82
82
83
## Download base model
83
84
We're using https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct.
Expect to see two files: '<model_name>.pte' and 'foundation.ptd'. Run the command again to generate more adapter PTE files. For example:
119
+
Expect to see two files: '<model_name>.pte' and 'foundation.ptd'. Run the command again to generate more adapter PTE files. The generated `foundation.ptd` files should all be the same (we are using the same base model) and you only need to keep one of them.
119
120
121
+
Example files:
120
122
```
121
123
-rw-r--r-- 1 lfq users 45555712 Oct 17 18:05 et.pte
122
124
-rw-r--r-- 1 lfq users 5994013600 Oct 17 18:05 foundation.ptd
123
125
-rw-r--r-- 1 lfq users 45555712 Oct 17 18:00 nobel.pte
124
126
```
125
127
126
-
The `foundation.ptd` file should be the same regardless of the adapter.
127
-
Notice the adapter PTE files about the size of the adapter_model.safetensors file generated during training. The PTE contains the adapter weights (which are not shared) and the program.
128
+
Notice the adapter PTE files are about the same size as the `adapter_model.safetensors` file generated during training. The PTE contains the adapter weights (which are not shared) and the program.
128
129
129
130
## Install runtime dependencies.
130
131
The ExecuTorch repository is configured as a git submodule at `~/executorch-examples/program-data-separation/cpp/executorch`. To initialize it:
--prompt="Who were the winners of the Nobel Prize in Physics in 2025?" \
167
168
--apply_chat_template
168
169
```
169
-
Set `apply_chat_template` to true as this was trained as a chatbot.
170
+
Passing in the `DOWNLOADED_PATH` as the tokenizer directory will invoke the HFTokenizer, and parse the additional tokenizers files: `tokenizer_config.json` and `special_tokens_map.json`. `special_tokens_map.json` tells us which bos/eos token to use, especially if there are multiple.
170
171
171
-
Sample output:
172
+
`apply_chat_template` formats the prompt according to the LLAMA chat template, which is what the adapter was trained on.
172
173
174
+
Sample output:
175
+
```
176
+
I 00:00:00.538779 executorch:main.cpp:133] Generating with model et.pte..
177
+
...
178
+
I 00:00:06.999737 executorch:text_llm_runner.cpp:182] RSS after prompt prefill: 6941.296875 MiB (0 if unsupported)
179
+
I don't have information on the winners of the Nobel Prize in Physics in 2025.<|eot_id|>
180
+
...
181
+
I 00:00:11.635379 executorch:main.cpp:141] Generating with model nobel.pte...
182
+
...
183
+
I 00:00:14.109447 executorch:text_llm_runner.cpp:182] RSS after prompt prefill: 8041.632812 MiB (0 if unsupported)
184
+
John Clarke, Michel H. Devoret, John M. Martinis<|eot_id|>
185
+
```
186
+
We can see that the ExecuTorch-trained adapter model does not have knowledge of the recent Nobel Prize winners, as neither the base model or adapter was trained on it. Meanwhile, the Nobel-prize adapter model can answer well.
--prompt="Help me get started with ExecuTorch in 3 steps" \
200
+
--apply_chat_template
201
+
```
202
+
203
+
Sample output:
204
+
```
205
+
...
206
+
I 00:00:00.554048 executorch:main.cpp:133] Generating with model et.pte...
207
+
...
208
+
Here are 3 steps to get started with ExecuTorch:
209
+
210
+
Step 1: Install ExecuTorch dependencies. This includes installing Python 3.8+ library, PyTorch library, and the ExecuTorch runtime.
211
+
212
+
Step 2: Set up a Python environment with pip and a virtual environment (e.g., conda) to isolate ExecuTorch dependencies.
213
+
214
+
Step 3: Clone the Execu
215
+
I 00:00:27.243400 executorch:text_llm_runner.cpp:206] RSS after finishing text generation: 6940.410156 MiB (0 if unsupported)
216
+
...
217
+
I 00:00:27.243504 executorch:main.cpp:141] Generating with model nobel.pte...
218
+
...
219
+
Here are the 3 steps to get started with Excetorch:
220
+
221
+
**Step 1: Install Node.js and npm**
222
+
223
+
Excetorch is a JavaScript compiler, so you'll need Node.js and npm (the Node Package Manager) installed on your computer. You can download Node.js from the official website and npm from the npm website. Follow the installation instructions for your operating system.
224
+
225
+
**Step 2: Install Excetorch**
226
+
227
+
228
+
I 00:00:50.189743 executorch:text_llm_runner.cpp:206] RSS after finishing text generation: 8039.152344 MiB (0 if unsupported)
184
229
```
230
+
231
+
The ExecuTorch-trained adapter model has domain knowledge of ExecuTorch codebase, whereas the Nobel-prize trained adapter model does not.
0 commit comments