You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This directory contains the C++ code for the LoRA demo.
4
4
5
5
You'll learn how to:
6
-
1. Export two LoRA PTE files that share a single foundation weight file.
7
-
2. Load and run the LoRA PTE files, and notice that the runtime memory is not doubled as the foundation weights are shared.
6
+
1. Export LoRA PTE files that share a single foundation weight file.
7
+
2. Load and run multiple LoRA PTE files at the same, and notice that the runtime memory increases by the LoRA adapter size (small) and not the foundation weight size (large), because the foundation weights are shared.
8
8
9
9
Note:
10
10
- Weight-sharing is supported with the XNNPACK backend.
11
-
- Quantization (outside of embedding quantization) is not supported when weight-sharing.
11
+
- Quantization (outside of embedding quantization) is currently not supported when weight-sharing.
12
12
- There are many ways to fine-tune LoRA adapters. We will go through a few examples to create a demo.
Size results will vary depending on the model and LoRA config. For this demo, we save ~5GB of disk space by storing weights in a separate, sharable file and ~5GB runtime memory by sharing weights at runtime through the XNNPACK weight cache. Detailed results are below.
17
25
18
-
### XNNPACK weight sharing.
26
+
### XNNPACK weight sharing
19
27
20
28
The XNNPACK backend is a singleton. Weight sharing is implemented via the XNNPACK weight cache. At delegate init time, XNNPACK checks the weight cache for the weights it needs. If they don't exist, XNNPACK will fetch weights from the NamedDataMap (the API that exposes weights in a PTD file), pack them, store them in the weight cache and free the original. This means we won't keep around multiple copies of the same weights.
Or alternatively, [install conda on your machine](https://conda.io/projects/conda/en/latest/user-guide/install/index.html)
30
+
## Finetune from scratch with Unsloth and Llama
31
+
[Unsloth](https://unsloth.ai/) provides a [colab notebook](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/datasets-guide#synthetic-dataset-notebook) that showcases how to generate data using the Meta Synthetic Data Kit, and then fine-tune it to create a LoRA adapter.
32
+
33
+
For this demo, we trained on two datasets:
34
+
1. executorch/docs/source/: an adapter with domain knowledge of executorch. This used Meta Synthetic Data Kit to generate qa pairs based on the documentation.
35
+
2. Recent Nobel prize winners (2024-2025): an adapter with knowledge beyond the cutoff date of Llama-3-2-1B. This data was taken from [Wikipedia](https://en.wikipedia.org/wiki/List_of_Nobel_laureates), and formatted into the chat template for training.
36
+
37
+
The training notebook takes a few shortcuts to reduce the latency/compute. You can change these settings for better results.
38
+
1. When generating data, play around with the chunk sizes and overlap to see what works best for your dataset.
39
+
2. At the training step, the notebook uses max_steps=60 to speed things up. Setting num_train_epochs=1 (or greater) for a full run and max_steps=None has better results.
40
+
41
+
Unsloth will output the adapter artifacts to the specified directory (in the colab notebook, 'lora_model/'). You will see a few files like such:
-rw-r--r-- 1 lfq users 1092 Oct 15 11:01 adapter_config.json
44
+
-rw-r--r-- 1 lfq users 45118424 Oct 15 11:01 adapter_model.safetensors
45
+
-rw-r--r-- 1 lfq users 3827 Oct 15 11:01 chat_template.jinja
46
+
-rw-r--r-- 1 lfq users 5268 Oct 15 11:01 README.md
47
+
-rw-r--r-- 1 lfq users 454 Oct 15 11:01 special_tokens_map.json
48
+
-rw-r--r-- 1 lfq users 50642 Oct 15 11:01 tokenizer_config.json
49
+
-rw-r--r-- 1 lfq users 17209920 Oct 15 11:01 tokenizer.json
30
50
```
31
51
52
+
The files we want are:
53
+
- adapter_config.json
54
+
- adapter_model.safetensors
55
+
32
56
## Install executorch
33
-
Please install executorch. If you are using your own trained adapter (not the example one), please use a recent nightly build or installfromsource.
57
+
[Install from source](https://docs.pytorch.org/executorch/stable/using-executorch-building-from-source.html#install-executorch-pip-package-from-source).
34
58
35
59
```
36
-
pip install executorch==1.0.0
60
+
# Move to the executorch subdirectory
61
+
cd ~/executorch-examples/program-data-separation/cpp/executorch
Or [install from source](https://docs.pytorch.org/executorch/stable/using-executorch-building-from-source.html#install-executorch-pip-package-from-source).
78
+
Use main or a recent nightly, as some features are not available in executorch==1.0.0.
45
79
80
+
## Export models
46
81
47
-
## Export the model/s.
48
-
Change into the program-data-separation directory and create a directory to hold exported artifacts.
49
-
```bash
50
-
cd~/executorch-examples/program-data-separation
51
-
mkdir models
82
+
1. Download the base model. We're using https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct.
52
83
```
84
+
pip install huggingface_hub
53
85
54
-
Export models into the `models` directory.
55
-
- The first command generates a regular llama_3_2_1B model.
56
-
- The second command generates a llama_3_2_1B lora model.
- PTD: contains the constant tensors used by the PTE. This format is similar to safetensors. It relies on flatbuffers instead of json for serde.
115
+
Expect to see two files: '<model_name>.pte' and 'foundation.ptd'. Run the command again to generate more adapter PTE files. You only need to keep one `foundation.ptd` file.
74
116
75
-
Sample file sizes:
76
-
```
77
-
-rw-r--r-- 1 lfq users 5994013600 Oct 17 14:31 foundation.ptd
117
+
You can also run `~/executorch-examples/program-data-separation/export_lora.sh`. This will export the dummy lora model and the base Llama-3-2-1B model PTE files.
118
+
119
+
Example files, trained on executorch/docs/source/ and recent Nobel prize winners.
120
+
```bash
121
+
# executorch docs trained adapter model.
122
+
-rw-r--r-- 1 lfq users 45555712 Oct 17 18:05 et.pte
123
+
# foundation weight file
124
+
-rw-r--r-- 1 lfq users 5994013600 Oct 17 18:05 foundation.ptd
125
+
# dummy lora model.
78
126
-rw-r--r-- 1 lfq users 27628928 Oct 17 14:31 llama_3_2_1B_lora.pte
79
-
-rw-r--r-- 1 lfq users 317248 Oct 17 14:28 llama_3_2_1B.pte
127
+
# Nobel prize winners trained adapter model.
128
+
-rw-r--r-- 1 lfq users 45555712 Oct 17 18:00 nobel.pte
80
129
```
81
130
82
-
Notice the lora - llama file size difference is about 27.3MB. This is the size of the adapter weights, and changes depending on the LoRA config. This demo is using the config from https://huggingface.co/lucylq/llama3_1B_lora/blob/main/adapter_config.json.
Notice the adapter PTE files are about the same size as the `adapter_model.safetensors`/`adapter_model.pt` files generated during training. The PTE contains the adapter weights (which are not shared) and the program.
86
132
87
-
## Install runtime dependencies.
133
+
## Install runtime dependencies
88
134
The ExecuTorch repository is configured as a git submodule at `~/executorch-examples/program-data-separation/cpp/executorch`. To initialize it:
--prompt="Who were the winners of the Nobel Prize in Physics in 2025?" \
174
+
--apply_chat_template
123
175
```
176
+
Passing in the `DOWNLOADED_PATH` as the tokenizer directory will invoke the HFTokenizer, and parse additional tokenizers files: `tokenizer_config.json` and `special_tokens_map.json`. `special_tokens_map.json` tells us which bos/eos token to use, especially if there are multiple.
124
177
125
-
You should see some logs showing the Resident Set Size (RSS) at various points of the execution. Some sample logs may look like this:
178
+
`apply_chat_template` formats the prompt according to the LLAMA chat template, which is what the adapter was trained on.
126
179
180
+
Sample output:
127
181
```
128
-
Generating with model <model file path>
129
-
RSS after loading model: 6909.328125 MiB
130
-
RSS after prompt prefill: 6909.328125 MiB
131
-
RSS after finishing text generation: 6909.328125 MiB
132
-
133
-
Generating with lora...
134
-
RSS after loading model: 7941.667969 MiB
135
-
RSS after prompt prefill: 7941.667969 MiB
136
-
RSS after finishing text generation: 7941.667969 MiB
182
+
I 00:00:00.538779 executorch:main.cpp:133] Generating with model et.pte..
183
+
...
184
+
I 00:00:06.999737 executorch:text_llm_runner.cpp:182] RSS after prompt prefill: 6941.296875 MiB (0 if unsupported)
185
+
I don't have information on the winners of the Nobel Prize in Physics in 2025.<|eot_id|>
186
+
...
187
+
I 00:00:11.635379 executorch:main.cpp:141] Generating with model nobel.pte...
188
+
...
189
+
I 00:00:14.109447 executorch:text_llm_runner.cpp:182] RSS after prompt prefill: 8041.632812 MiB (0 if unsupported)
190
+
John Clarke, Michel H. Devoret, John M. Martinis<|eot_id|>
137
191
```
138
-
There is about ~1.4GB memory increase between running the two models.
139
-
~1GB comes from embeddings that are not lowered to XNNPACK (and currently are not shared). This can be alleviated by quantizing the embeddings by adding the config `quantization.embedding_quantize=\'4,32\'` to the export command.
140
-
~40MB comes from running the non-lora model, to running the lora model.
192
+
We can see that the ExecuTorch-trained adapter model does not have knowledge of the recent Nobel Prize winners, as neither the base model or adapter was trained on it. Meanwhile, the Nobel-prize adapter model can answer well.
141
193
142
-
You can see the difference without weight-sharing by removing the flag `-DEXECUTORCH_XNNPACK_ENABLE_WEIGHT_CACHE=True` from `build_example.sh`. Expect to see almost double the memory usage, ie. ~14-15GB instead of ~8GB.
194
+
There is about ~1.1GB memory increase between running the two models.
195
+
Most of that (about ~1GB) comes from embeddings that are not lowered to XNNPACK (and currently are not shared). This can be alleviated by quantizing the embeddings by adding the config `quantization.embedding_quantize=\'4,32\'` to the export command.
196
+
~50MB comes from the adapter model, which is not shared.
--prompt="Help me get started with ExecuTorch in 3 steps" \
209
+
--apply_chat_template
210
+
```
211
+
212
+
Sample output:
149
213
```
214
+
...
215
+
I 00:00:00.554048 executorch:main.cpp:133] Generating with model et.pte...
216
+
...
217
+
Here are 3 steps to get started with ExecuTorch:
218
+
219
+
Step 1: Install ExecuTorch dependencies. This includes installing Python 3.8+ library, PyTorch library, and the ExecuTorch runtime.
220
+
221
+
Step 2: Set up a Python environment with pip and a virtual environment (e.g., conda) to isolate ExecuTorch dependencies.
222
+
223
+
Step 3: Clone the Execu
224
+
I 00:00:27.243400 executorch:text_llm_runner.cpp:206] RSS after finishing text generation: 6940.410156 MiB (0 if unsupported)
225
+
...
226
+
I 00:00:27.243504 executorch:main.cpp:141] Generating with model nobel.pte...
227
+
...
228
+
Here are the 3 steps to get started with Excetorch:
229
+
230
+
**Step 1: Install Node.js and npm**
231
+
232
+
Excetorch is a JavaScript compiler, so you'll need Node.js and npm (the Node Package Manager) installed on your computer. You can download Node.js from the official website and npm from the npm website. Follow the installation instructions for your operating system.
233
+
234
+
**Step 2: Install Excetorch**
235
+
236
+
237
+
I 00:00:50.189743 executorch:text_llm_runner.cpp:206] RSS after finishing text generation: 8039.152344 MiB (0 if unsupported)
238
+
```
239
+
240
+
The ExecuTorch-trained adapter model has domain knowledge of ExecuTorch codebase, whereas the Nobel-prize trained adapter model does not.
0 commit comments