Add Model type toString() (#85)

kirklandsign · lucylq · commit 7fb6731d70fd · 2025-10-19T13:14:12.000-07:00
diff --git a/program-data-separation/cpp/lora_example/README.md b/program-data-separation/cpp/lora_example/README.md
@@ -3,7 +3,7 @@
 This directory contains the C++ code for the LoRA demo.
 
 You'll learn how to:
-1. Export two LoRA PTE files that share a single foundation weight file.
+1. Export LoRA PTE files that share a single foundation weight file.
 2. Load and run the LoRA PTE files, and notice that the runtime memory is not doubled as the foundation weights are shared.
 
 Note:
@@ -13,12 +13,43 @@ Note:
 
 ## Size savings.
 
+Size results will vary depending on the model and LoRA config. For this demo, we save ~5GB of disk space by storing weights in a separate, sharable file and ~5GB runtime memory by sharing weights at runtime through the XNNPACK weight cache. Detailed results are below.
 Size results will vary depending on the model and LoRA config. For this demo, we save ~5GB of disk space by storing weights in a separate, sharable file and ~5GB runtime memory by sharing weights at runtime through the XNNPACK weight cache. Detailed results are below.
 
 ### XNNPACK weight sharing.
 
 The XNNPACK backend is a singleton. Weight sharing is implemented via the XNNPACK weight cache. At delegate init time, XNNPACK checks the weight cache for the weights it needs. If they don't exist, XNNPACK will fetch weights from the NamedDataMap (the API that exposes weights in a PTD file), pack them, store them in the weight cache and free the original. This means we won't keep around multiple copies of the same weights.
 
+## [Quick Start](quick_start.md)
+Download pre-trained dummy adapter to export and run along with a regular Llama-3-2-1B model.
+
+## Fine-tune from scratch with Unsloth and Llama-3-2-1B.
+We can use [Unsloth](https://unsloth.ai/), a popular tool to finetune and train LLMs, to create our LoRA adapters. Unsloth provides a [colab notebook](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/datasets-guide#synthetic-dataset-notebook) that showcases how to generate data using the Meta Synthetic Data Kit.
+
+The training notebook takes a few shortcuts to reduce the latency/compute. You can change these settings for better results.
+1. Play around with the chunk sizes and overlap to see what works best for your dataset.
+2. The notebook trains on the last three data files generated; increase this for better coverage of your dataset.
+3. At the training step, the notebook uses max_steps=60 to speed things up. Setting num_train_epochs=1 (or greater) for a full run and max_steps=None has better results.
+
+For this demo, we trained on two datasets:
+1. executorch/docs/source: an adapter with domain knowledge of executorch. Using Meta Synthetic Data Kit, you can generate qa pairs based on the executorch documentation.
+2. Recent Nobel prize winners (2024-2025): an adapter with knowledge beyond the cutoff date of Llama-3-2-1B. This data was taken from [Wikipedia](https://en.wikipedia.org/wiki/List_of_Nobel_laureates).
+
+Unsloth will output the adapter artifacts to the specified directory (in the colab notebook, 'lora_model/'). You will see a few files like such:
+```
+-rw-r--r-- 1 lfq users     1092 Oct 15 11:01 adapter_config.json
+-rw-r--r-- 1 lfq users 45118424 Oct 15 11:01 adapter_model.safetensors
+-rw-r--r-- 1 lfq users     3827 Oct 15 11:01 chat_template.jinja
+-rw-r--r-- 1 lfq users     5268 Oct 15 11:01 README.md
+-rw-r--r-- 1 lfq users      454 Oct 15 11:01 special_tokens_map.json
+-rw-r--r-- 1 lfq users    50642 Oct 15 11:01 tokenizer_config.json
+-rw-r--r-- 1 lfq users 17209920 Oct 15 11:01 tokenizer.json
+```
+
+The files we want are:
+- adapter_config.json
+- adapter_model.safetensors
+
 ## Virtual environment setup.
 Create and activate a Python virtual environment:
 ```bash
@@ -30,11 +61,6 @@ conda create -yn executorch-ptd python=3.10.0 && conda activate executorch-ptd
 ```
 
 ## Install executorch
-Please install executorch. If you are using your own trained adapter (not the example one), please use a recent nightly build or install from source.
-
-```
-pip install executorch==1.0.0
-```
 
 You can also install from the nightly build.
 ```
@@ -43,47 +69,63 @@ pip install executorch==1.1.0.devYYYYMMDD --extra-index-url https://download.pyt
 
 Or [install from source](https://docs.pytorch.org/executorch/stable/using-executorch-building-from-source.html#install-executorch-pip-package-from-source).
 
+```
+# Clone the ExecuTorch repo from GitHub.
+git clone https://github.com/pytorch/executorch.git && cd executorch
 
-## Export the model/s.
-Change into the program-data-separation directory and create a directory to hold exported artifacts.
-```bash
-cd ~/executorch-examples/program-data-separation
-mkdir models
+# Install ExecuTorch pip package.
+./install_executorch.sh --editable
 ```
 
-Export models into the `models` directory.
-- The first command generates a regular llama_3_2_1B model.
-- The second command generates a llama_3_2_1B lora model.
+NOTE: some features are not available in executorch==1.0.0, use main or a recent nightly.
 
-```bash
-sh export_lora.sh
+## Download base model
+We're using https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct.
 ```
-Expect the files:
-- llama_3_2_1B.pte
-- llama_3_2_1B.ptd
-- llama_3_2_1B_lora.pte
-- foundation_weights.ptd
-- tokenizer.model
+pip install huggingface_hub
 
-llama_3_2_1B.ptd and foundation_weights.ptd contain the same contents, and you can remove llama_3_2_1B.ptd.
-tokenizer.model is copied from the temp directory where we downloaded the HF artifacts. It is used at runtime.
+# As this is a gated model, login.
+huggingface-cli login
+huggingface-cli download meta-llama/Llama-3.2-1B-Instruct --local-dir ./Llama-3.2-1B-Instruct
+```
 
-Note:
-- PTE: contains the program execution logic.
-- PTD: contains the constant tensors used by the PTE. This format is similar to safetensors. It relies on flatbuffers instead of json for serde.
+## Export the adapter models.
 
-Sample file sizes:
+Set your paths and the model name.
 ```
--rw-r--r-- 1 lfq users 5994013600 Oct 17 14:31 foundation.ptd
--rw-r--r-- 1 lfq users   27628928 Oct 17 14:31 llama_3_2_1B_lora.pte
--rw-r--r-- 1 lfq users     317248 Oct 17 14:28 llama_3_2_1B.pte
+DOWNLOADED_PATH=Llama-3.2-1B-Instruct
+ADAPTER_PATH=lora_model
+MODEL_NAME=<model_name>
 ```
 
-Notice the lora - llama file size difference is about 27.3MB. This is the size of the adapter weights, and changes depending on the LoRA config. This demo is using the config from https://huggingface.co/lucylq/llama3_1B_lora/blob/main/adapter_config.json.
+Export command. Run this with different MODEL_NAMEs for each adapter.
+```
+python -m executorch.extension.llm.export.export_llm \
+    base.checkpoint="${DOWNLOADED_PATH}/original/consolidated.00.pth" \
+    base.params="${DOWNLOADED_PATH}/original/params.json" \
+    base.tokenizer_path="${DOWNLOADED_PATH}/original/tokenizer.model" \
+    base.adapter_checkpoint="${ADAPTER_PATH}/adapter_model.safetensors" \
+    base.adapter_config="${ADAPTER_PATH}/adapter_config.json" \
+    model.use_kv_cache=true \
+    model.use_sdpa_with_kv_cache=true \
+    model.dtype_override="fp32" \
+    backend.xnnpack.enabled=true \
+    backend.xnnpack.extended_ops=true \
+    export.output_name="${MODEL_NAME}.pte" \
+    export.foundation_weights_file="foundation.ptd"
+```
+
+Expect to see two files: '<model_name>.pte' and 'foundation.ptd'. Run the command again to generate more adapter PTE files. For example:
+
 ```
-{"r": 64, "lora_alpha": 128, "target_modules": ["q_proj", "v_proj", "o_proj"], "peft_type": "LORA", "base_model_name_or_path": "meta-llama/Llama-3.2-1B-Instruct"}
+-rw-r--r-- 1 lfq users   45555712 Oct 17 18:05 et.pte
+-rw-r--r-- 1 lfq users 5994013600 Oct 17 18:05 foundation.ptd
+-rw-r--r-- 1 lfq users   45555712 Oct 17 18:00 nobel.pte
 ```
 
+The `foundation.ptd` file should be the same regardless of the adapter.
+Notice the adapter PTE files about the size of the adapter_model.safetensors file generated during training. The PTE contains the adapter weights (which are not shared) and the program.
+
 ## Install runtime dependencies.
 The ExecuTorch repository is configured as a git submodule at `~/executorch-examples/program-data-separation/cpp/executorch`.  To initialize it:
 ```bash
@@ -115,35 +157,28 @@ sh build_example.sh
 ```bash
 cd ~/executorch-examples/program-data-separation/cpp/lora_example
 
+DOWNLOADED_PATH=Llama-3.2-1B-Instruct
 ./build/bin/executorch_program_data_separation \
-    --tokenizer_path="../../tokenizer.model" \
-    --model1="../../models/llama_3_2_1B_lora.pte" \
-    --model2="../../models/llama_3_2_1B.pte"  \
-    --weights="../../models/foundation.ptd"
+    --tokenizer_path="${DOWNLOADED_PATH}" \
+    --model1="et.pte" \
+    --model2="nobel.pte"  \
+    --weights="foundation.ptd" \
+    --prompt="Who were the winners of the Nobel Prize in Physics in 2025?" \
+    --apply_chat_template
 ```
+Set `apply_chat_template` to true as this was trained as a chatbot.
 
-You should see some logs showing the Resident Set Size (RSS) at various points of the execution. Some sample logs may look like this:
-
-```
-Generating with model <model file path>
-RSS after loading model: 6909.328125 MiB
-RSS after prompt prefill: 6909.328125 MiB
-RSS after finishing text generation: 6909.328125 MiB
+Sample output:
 
-Generating with lora...
-RSS after loading model: 7941.667969 MiB
-RSS after prompt prefill: 7941.667969 MiB
-RSS after finishing text generation: 7941.667969 MiB
-```
-There is about ~1.4GB memory increase between running the two models.
-~1GB comes from embeddings that are not lowered to XNNPACK (and currently are not shared). This can be alleviated by quantizing the embeddings by adding the config `quantization.embedding_quantize=\'4,32\'` to the export command.
-~40MB comes from running the non-lora model, to running the lora model.
 
-You can see the difference without weight-sharing by removing the flag `-DEXECUTORCH_XNNPACK_ENABLE_WEIGHT_CACHE=True` from `build_example.sh`. Expect to see almost double the memory usage, ie. ~14-15GB instead of ~8GB.
 
-## Clean up.
-```bash
-rm -rf build
-cd ~/executorch-examples/program-data-separation
-rm -rf models/
+Let's try with an executorch-specific prompt.
+```
+DOWNLOADED_PATH=Llama-3.2-1B-Instruct
+./build/bin/executorch_program_data_separation \
+    --tokenizer_path="${DOWNLOADED_PATH}" \
+    --model1="adapter_model1.pte" \
+    --model2="adapter_model2.pte"  \
+    --weights="foundation.ptd" \
+    --prompt="Help me get started with ExecuTorch"
 ```
diff --git a/program-data-separation/cpp/lora_example/main.cpp b/program-data-separation/cpp/lora_example/main.cpp
@@ -130,7 +130,6 @@ int main(int argc, char *argv[]) {
   }
 
   ET_LOG(Info, "Generating with model %s...", model1);
-  ET_LOG(Info, "Formatted prompt: %s", formatted_prompt.c_str());
   auto error = runner1->generate(formatted_prompt, config);
   if (error != Error::Ok) {
     ET_LOG(Error, "Failed to generate with model %s, error code %zu.",
diff --git a/program-data-separation/cpp/lora_example/quick_start.md b/program-data-separation/cpp/lora_example/quick_start.md
@@ -0,0 +1,135 @@
+# Quick Start
+
+Use the provided export scripts to generate and run LoRA models.
+
+## Virtual environment setup.
+Create and activate a Python virtual environment:
+```bash
+python3 -m venv .venv && source .venv/bin/activate && pip install --upgrade pip
+```
+Or alternatively, [install conda on your machine](https://conda.io/projects/conda/en/latest/user-guide/install/index.html)
+```bash
+conda create -yn executorch-ptd python=3.10.0 && conda activate executorch-ptd
+```
+
+## Install executorch
+Please install executorch. If you are using your own trained adapter (not the example one), please use a recent nightly build or install from source.
+
+```
+pip install executorch==1.0.0
+```
+
+You can also install from the nightly build.
+```
+pip install executorch==1.1.0.devYYYYMMDD --extra-index-url https://download.pytorch.org/whl/nightly/cpu
+```
+
+Or [install from source](https://docs.pytorch.org/executorch/stable/using-executorch-building-from-source.html#install-executorch-pip-package-from-source).
+
+
+## Export the model/s.
+Change into the program-data-separation directory and create a directory to hold exported artifacts.
+```bash
+cd ~/executorch-examples/program-data-separation
+mkdir models
+```
+
+Export models into the `models` directory.
+- The first command generates a regular llama_3_2_1B model.
+- The second command generates a llama_3_2_1B lora model.
+
+```bash
+sh export_lora.sh
+```
+Expect the files:
+- llama_3_2_1B.pte
+- llama_3_2_1B.ptd
+- llama_3_2_1B_lora.pte
+- foundation_weights.ptd
+- tokenizer.model
+
+llama_3_2_1B.ptd and foundation_weights.ptd contain the same contents, and you can remove llama_3_2_1B.ptd.
+tokenizer.model is copied from the temp directory where we downloaded the HF artifacts. It is used at runtime.
+
+Note:
+- PTE: contains the program execution logic.
+- PTD: contains the constant tensors used by the PTE. This format is similar to safetensors. It relies on flatbuffers instead of json for serde.
+
+Sample file sizes:
+```
+-rw-r--r-- 1 lfq users 5994013600 Oct 17 14:31 foundation.ptd
+-rw-r--r-- 1 lfq users   27628928 Oct 17 14:31 llama_3_2_1B_lora.pte
+-rw-r--r-- 1 lfq users     317248 Oct 17 14:28 llama_3_2_1B.pte
+```
+
+Notice the lora - llama file size difference is about 27.3MB. This is the size of the adapter weights, and changes depending on the LoRA config. This demo is using the config from https://huggingface.co/lucylq/llama3_1B_lora/blob/main/adapter_config.json.
+```
+{"r": 64, "lora_alpha": 128, "target_modules": ["q_proj", "v_proj", "o_proj"], "peft_type": "LORA", "base_model_name_or_path": "meta-llama/Llama-3.2-1B-Instruct"}
+```
+
+## Install runtime dependencies.
+The ExecuTorch repository is configured as a git submodule at `~/executorch-examples/program-data-separation/cpp/executorch`.  To initialize it:
+```bash
+cd ~/executorch-examples/
+git submodule sync
+git submodule update --init --recursive
+```
+Install dev requirements for ExecuTorch:
+
+```bash
+cd ~/executorch-examples/program-data-separation/cpp/executorch
+pip install -r requirements-dev.txt
+```
+
+## Build the runtime.
+Install some dependencies:
+```bash
+cd ~/executorch-examples/program-data-separation/cpp/executorch
+sh examples/models/llama/install_requirements.sh
+```
+
+Build the executable:
+```bash
+cd ~/executorch-examples/program-data-separation/cpp/lora_example
+sh build_example.sh
+```
+
+## Run the executable.
+```bash
+cd ~/executorch-examples/program-data-separation/cpp/lora_example
+
+./build/bin/executorch_program_data_separation \
+    --tokenizer_path="../../tokenizer.model" \
+    --model1="../../models/llama_3_2_1B_lora.pte" \
+    --model2="../../models/llama_3_2_1B.pte"  \
+    --weights="../../models/foundation.ptd"
+```
+
+You should see some logs showing the Resident Set Size (RSS) at various points of the execution. Some sample logs may look like this:
+
+```
+Generating with model <model file path>
+RSS after loading model: 6909.328125 MiB
+RSS after prompt prefill: 6909.328125 MiB
+RSS after finishing text generation: 6909.328125 MiB
+
+Generating with model <model file path>...
+RSS after loading model: 7941.667969 MiB
+RSS after prompt prefill: 7941.667969 MiB
+RSS after finishing text generation: 7941.667969 MiB
+```
+There is about ~1.4GB memory increase between running the two models.
+~1GB comes from embeddings that are not lowered to XNNPACK (and currently are not shared). This can be alleviated by quantizing the embeddings by adding the config `quantization.embedding_quantize=\'4,32\'` to the export command.
+~40MB comes from running the non-lora model, to running the lora model.
+
+You can see the difference without weight-sharing by removing the flag `-DEXECUTORCH_XNNPACK_ENABLE_WEIGHT_CACHE=True` from `build_example.sh`. Expect to see almost double the memory usage, ie. ~14-15GB instead of ~8GB.
+
+## Fine-tuned adapter output.
+
+
+## Clean up.
+```bash
+rm -rf build
+cd ~/executorch-examples/program-data-separation
+rm -rf models/
+```

Original file line number	Diff line number	Diff line change
`@@ -130,7 +130,6 @@ int main(int argc, char *argv[]) {`
`130`	`130`	`}`
`131`	`131`
`132`	`132`	`ET_LOG(Info, "Generating with model %s...", model1);`
`133`		`- ET_LOG(Info, "Formatted prompt: %s", formatted_prompt.c_str());`
`134`	`133`	`auto error = runner1->generate(formatted_prompt, config);`
`135`	`134`	`if (error != Error::Ok) {`
`136`	`135`	`ET_LOG(Error, "Failed to generate with model %s, error code %zu.",`