Add AI Toolkit model conversion docs (#47)

ssss141414 · web-flow · commit 3456114e3093 · 2025-05-18T21:02:45.000+02:00
* docs

* fix comments

* fix comments

* fix comments

* fix comments

* fix comments

* fix comments

* fix comments

* fix comments

* fix comments

* fix comments

* remove files

* reupload

* fix comments

* fix comments

* try fix comments

* try fix comments

* try fix comments

* fix comments

* fix comments

* update path
diff --git a/docs/intelligentapps/images/modelconversion/Re-evaluate.png b/docs/intelligentapps/images/modelconversion/Re-evaluate.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8bb549e5642128683316c25d8bca6cf147ed8f30b893f512361911e23093d9e5
+size 34828
diff --git a/docs/intelligentapps/images/modelconversion/Run.png b/docs/intelligentapps/images/modelconversion/Run.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a6a7f27a514129073683f34434c98979c921ebdaedf2c1ae1ca8410fac745bfd
+size 263202
diff --git a/docs/intelligentapps/images/modelconversion/TemplateFiles.png b/docs/intelligentapps/images/modelconversion/TemplateFiles.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:93452d9f2dc6c6562d9c46d593badd18b3056c842d02ec887350503472e0c7e5
+size 9350
diff --git a/docs/intelligentapps/images/modelconversion/create_project_add_models.png b/docs/intelligentapps/images/modelconversion/create_project_add_models.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1acae8f8bd5440575e48a5cacfd3d3f3a1dd5e0f090dbda720e9df757eea38aa
+size 3863
diff --git a/docs/intelligentapps/images/modelconversion/create_project_default.png b/docs/intelligentapps/images/modelconversion/create_project_default.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cb64834bf10aae610989790d80663a1ee8d0439e1e22ff2742372c279315bace
+size 209320
diff --git a/docs/intelligentapps/images/modelconversion/create_project_input_name.png b/docs/intelligentapps/images/modelconversion/create_project_input_name.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:17969a023d8a68c64f6c5cb5a24759c446a31a27ef951ce2b01a8d6e2dd2ed5a
+size 8468
diff --git a/docs/intelligentapps/images/modelconversion/create_project_model_list.png b/docs/intelligentapps/images/modelconversion/create_project_model_list.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d851cedfcd727fc356f201f7e88ecbbe3097f46efe2f58157e74bc70dcbd95d5
+size 246680
diff --git a/docs/intelligentapps/images/modelconversion/create_project_readme.png b/docs/intelligentapps/images/modelconversion/create_project_readme.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2528b9028912a2989f62b32365915d267cfeaef84ea607d56af21492f7d2f46d
+size 71202
diff --git a/docs/intelligentapps/images/modelconversion/create_project_select_folder.png b/docs/intelligentapps/images/modelconversion/create_project_select_folder.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f1fbc982b37ba8ebbb3ed1b8c51b93c00d294fca6450d2b72d92d1c657ea0e30
+size 8661
diff --git a/docs/intelligentapps/images/modelconversion/history.png b/docs/intelligentapps/images/modelconversion/history.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:403111196b2bfefeeca2874d777e75801da22b5e66d77864b94ab4d8777e0b2b
+size 149901
diff --git a/docs/intelligentapps/images/modelconversion/historyaction.png b/docs/intelligentapps/images/modelconversion/historyaction.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b1becfcb9ba1e09cbdcd1245b464e4fab296cdee3d4f9fcf8f6722c0d07ca7a
+size 145297
diff --git a/docs/intelligentapps/images/modelconversion/run_disclaimer.png b/docs/intelligentapps/images/modelconversion/run_disclaimer.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7769884cfe08dc3509987cf369c02c8bd29ff8d350a126e13ca8f4c96675cfbb
+size 77856
diff --git a/docs/intelligentapps/images/modelconversion/run_token_1.png b/docs/intelligentapps/images/modelconversion/run_token_1.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd4ac018a9253191116063ea2e8da56c5218ca4e50cd571bedb36c9abd974821
+size 35571
diff --git a/docs/intelligentapps/images/modelconversion/run_token_2.png b/docs/intelligentapps/images/modelconversion/run_token_2.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e4be8fc7ac7bae47857efb9725afceb6fd091ea222e91e04a05c99b7a10dca36
+size 56146
diff --git a/docs/intelligentapps/images/modelconversion/run_token_3.png b/docs/intelligentapps/images/modelconversion/run_token_3.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ce70eb3e897957e327d982056d1925829c9216322a76e6b9ff91a6a5ae8a1dd1
+size 5268
diff --git a/docs/intelligentapps/modelconversion.md b/docs/intelligentapps/modelconversion.md
diff --git a/docs/intelligentapps/reference/FileStructure.md b/docs/intelligentapps/reference/FileStructure.md
@@ -0,0 +1,82 @@
+---
+ContentId: fea54c41-e5bb-49e2-89bd-518af096ba8e
+DateApproved: 05/13/2025
+MetaDescription: Model Conversion reference about project structure.
+---
+
+# Model conversion file structure
+This article explains the file structure generated during model conversion workflows and the purpose of each folder and file, including cache handling, history tracking, inference and so on.
+
+After creating the model project and running it several times, the file structure might look like this:
+
+```
+model_project_name/
+├── model_lab.workspace.config
+└── huggingface_microsoft_resnet-50_v1/
+    ├── .gitignore
+    ├── imagenet.py
+    ├── inference_sample.ipynb
+    ├── model_project.config
+    ├── README.md
+    ├── requirements.txt
+    ├── resnet_ptq_qnn.json
+    ├── cache/
+    └── history/
+        └── history_1(20250414_161046)/
+            ├── model/
+            ├── footprints.json
+            ├── history.config
+            ├── history.config.user
+            ├── inference_sample.ipynb
+            ├── log.txt
+            ├── metrics.json
+            ├── model_config.json
+            ├── olive_config.json
+            ├── output_footprint.json
+            └── run_history.txt
+        └── history_2/
+        └── history_3/
+```
+
+In the `model_project_name` folder, workflows for each model are stored in a separate folder.
+
+- `requirements.txt`: lists the dependencies required to run the workflow and the inference sample.
+- `resnet_ptq_qnn.json`, `imagenet.py`: the JSON file used to convert the model by Olive. Some might require additional Python files for customization.
+- `README.md`: describes model details such as the model's task, performance metrics, and usage instructions.
+- `model_project.config`: contains project template settings. Some settings can be overridden to suit your specific requirements.
+- `inference_sample.ipynb`: sample to test the output model. This file will be copied into the `history` folder, to enable using different Jupyter notebooks to compare models from different histories.
+
+## Cache folder
+
+The `cache` folder stores cache files that were generated during workflow execution. These cached results can help accelerate repeated runs of the workflow.
+
+You can delete this folder to free up space if it's no longer needed.
+
+## History folder
+
+The timestamp in the `history` folder name indicates the run time, like: `April 14, 2025 at 16:10:46`.
+
+- `model`: model files.
+- `model_config.json`: contains details about the model.
+- `footprints.json`, `output_footprint.json`, `run_history.txt`: Olive output.
+- `history.config`, `history.config.user`: history configurations used by Model Conversion.
+- `inference_sample.ipynb`: sample to test the output model.
+- `log.txt`: contain logs.
+- `metrics.json`: contain evaluation result if evaluation is enabled.
+- `olive_config.json`: the config used to run the conversion.
+
+## About git
+
+By default, the `cache` and `history` folders are excluded from version control (`.gitignore`), except for the following two configuration files: `history.config` and `olive_config.json`
+
+You might update the `.gitignore` file to include specific history folders that are worth saving.
+
+When someone else clones the repo, these two files ensure that they can rerun these histories to reproduce the conversion result.
+
+```
+__pycache__
+/cache
+/history/*/*
+!/history/*/history.config
+!/history/*/olive_config.json
+```
diff --git a/docs/intelligentapps/reference/ManualConversionOnGPU.md b/docs/intelligentapps/reference/ManualConversionOnGPU.md
@@ -0,0 +1,82 @@
+---
+ContentId: 47b2131d-2e68-46c6-a0aa-9c3586b5a2a9
+DateApproved: 05/13/2025
+MetaDescription: Model Conversion reference about manual model conversion for LLM models using GPU.
+---
+# Manual model conversion on GPU
+This article introduces the manual workflow for converting LLM models using a local Nvidia GPU. It describes the required environment setup, execution steps, and how to run inference on a Windows Copilot+ PC with a Qualcomm NPU.
+
+Conversion of LLM models requires a Nvidia GPU. If you want model lab to manage your local GPU, follow the steps in [Convert Model](/docs/intelligentapps/modelconversion.md#conversion). Otherwise, follow the steps in this article.
+
+## Manual run model conversion on GPU
+This workflow is configured using the `qnn_config.json` file and requires two separate Python environments.
+- The first environment is used for model conversion with GPU acceleration and includes packages like onnxruntime-gpu and AutoGPTQ.
+- The second environment is used for QNN optimization and includes packages like onnxruntime-qnn with specific dependencies.
+
+### First environment setup
+In a **Python 3.10** [x64 Python environment with Olive installed](https://github.com/microsoft/Olive/blob/main/examples/README.md#important), install the required packages:
+
+```bash
+# Install common dependencies
+pip install -r requirements.txt
+
+# Install ONNX Runtime GPU packages
+pip install "onnxruntime-gpu>=1.21.0" "onnxruntime-genai-cuda>=0.6.0"
+
+# AutoGPTQ: Install from source (stable package may be slow for weight packing)
+# Disable CUDA extension build (not required)
+# Linux
+export BUILD_CUDA_EXT=0
+# Windows
+# set BUILD_CUDA_EXT=0
+
+# Install AutoGPTQ from source
+pip install --no-build-isolation git+https://github.com/PanQiWei/AutoGPTQ.git
+
+# Please update CUDA version if needed
+pip install torch --index-url https://download.pytorch.org/whl/cu121
+```
+
+> ⚠️ Only set up the environment and install the packages. Do not run the `olive run` command at this point.
+
+### Second environment setup
+In a **Python 3.10** [x64 Python environment with Olive installed](https://github.com/microsoft/Olive/blob/main/examples/README.md#important), install the required packages:
+
+```bash
+# Install ONNX Runtime QNN
+pip install -r https://raw.githubusercontent.com/microsoft/onnxruntime/refs/heads/main/requirements.txt
+pip install -U --pre --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple onnxruntime-qnn --no-deps
+```
+
+Replace `/path/to/qnn/env/bin` in `qnn_config.json` with the path to the directory containing the **second environment**'s Python executable.
+
+### Run the config
+Activate the **first environment** and run the workflow:
+
+```bash
+olive run --config qnn_config.json
+```
+
+After completing this command, the optimized model is saved in: `./model/model_name`.
+
+> ⚠️ If optimization fails due to out of memory, please remove `calibration_providers` in config file.
+
+> ⚠️ If optimization fails during context binary generation, rerun the command. The process will resume from the last completed step.
+
+
+## Manual run inference samples
+The optimized model can be used for inference using [ONNX Runtime QNN Execution Provider](https://onnxruntime.ai/docs/execution-providers/QNN-ExecutionProvider.html) and [ONNX Runtime GenAI](https://onnxruntime.ai/docs/genai/). Inference must be run on a **Windows Copilot+ PC with a Qualcomm NPU.**
+
+### Install required packages on arm64 Python environment
+Model compilation using QNN Execution Provider requires a Python environment with onnxruntime-qnn installed. In a separate Python environment with Olive installed, install the required packages:
+
+```bash
+pip install -r https://raw.githubusercontent.com/microsoft/onnxruntime/refs/heads/main/requirements.txt
+pip install -U --pre --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple onnxruntime-qnn --no-deps
+pip install "onnxruntime-genai>=0.7.0rc2"
+```
+
+### Run interface sample
+Execute the provided `inference_sample.ipynb` notebook. Select ipykernel to this **arm64** Python environment.
+
+> ⚠️ If you get a `6033` error, replace `genai_config.json` in the `./model/model_name` folder.
diff --git a/docs/intelligentapps/reference/ManualModelConversion.md b/docs/intelligentapps/reference/ManualModelConversion.md
@@ -0,0 +1,109 @@
+---
+ContentId: d40f8c19-a7a4-4d53-9303-b9cb128af591
+DateApproved: 05/13/2025
+MetaDescription: Model Conversion reference about manual model conversion.
+---
+
+# Convert models to ONNX format
+
+The AI Toolkit supports the [Open Neural Network Exchange](https://onnx.ai) (ONNX) format for running models locally. ONNX is an open standard for representing machine learning models, defining a common set of operators and a file format that enables models to run across various hardware platforms.
+
+To use models from other catalogs, such as Azure AI Foundry or Hugging Face, in the AI Toolkit, you must first convert them to ONNX format.
+
+This tutorial guides you through converting Hugging Face models to ONNX format and loading them into the AI Toolkit.
+
+## Set up the environment
+
+To convert models from Hugging Face or Azure AI Foundry, you need the [Model Builder](https://onnxruntime.ai/docs/genai/howto/build-model.html) tool.
+
+Follow these steps to set up your environment:
+
+1. Ensure you have either [Anaconda](https://www.anaconda.com/download) or [Miniconda](https://www.anaconda.com/docs/getting-started/miniconda/install) installed on your device.
+
+2. Create a dedicated conda environment for Model Builder and install the necessary dependencies (`onnx`, `torch`, `onnxruntime_genai`, and `transformers`):
+
+    ```powershell
+    conda create -n model_builder python==3.11 -y
+    conda activate model_builder
+    pip install onnx torch onnxruntime_genai==0.6.0 transformers
+    ```
+
+    > Note: For certain newer models, such as Phi-4-mini, you may need to install the latest development version of transformers directly from GitHub:
+
+    ```powershell
+    pip install git+https://github.com/huggingface/transformers
+    ```
+
+## Access Hugging Face models
+
+There are multiple ways to access Hugging Face models. In this tutorial, we use the `huggingface_hub` CLI as an example to demonstrate managing a model repository.
+
+> Note: Ensure your Python environment is properly set up before proceeding.
+
+To download models from Hugging Face:
+
+1. [Install the CLI](https://huggingface.co/docs/huggingface_hub/main/en/guides/cli#getting-started):
+
+    ```powershell
+    pip install -U "huggingface_hub[cli]"
+    ```
+
+2. [Download the model repository](https://huggingface.co/docs/huggingface_hub/main/en/guides/cli#download-an-entire-repository).
+
+3. All files in the downloaded repository will be used during conversion.
+
+## Create the directory structure
+
+The AI Toolkit loads ONNX models from its working directory:
+
+* Windows: `%USERPROFILE%\.aitk\models`
+* Unix-like systems (macOS): `$HOME/.aitk/models`
+
+To ensure your models load correctly, create the required four-layer directory structure within the AI Toolkit's working directory. For example:
+
+```powershell
+mkdir C:\Users\Administrator\.aitk\models\microsoft\Phi-3.5-vision-instruct-onnx\cpu\phi3.5-cpu-int4-rtn-block-32
+```
+
+In this example, the four-layer directory structure is `microsoft\Phi-3.5-vision-instruct-onnx\cpu\phi3.5-cpu-int4-rtn-block-32`.
+
+> [!IMPORTANT]
+> The naming of the four-layer directory structure is important. Each directory layer corresponds with a specific system parameter: `$publisherName\$modelName\$runtime\$displayName`. The `$displayName` appears in the local model tree view at the top-left side of the extension. Use distinct `displayName` values for different models to avoid confusion.
+
+## Convert models to ONNX format
+
+Run the following command to convert your model to ONNX format:
+
+```powershell
+python -m onnxruntime_genai.models.builder -m $modelPath -p $precision -e $executionProvider -o $outputModelPath -c $cachePath --extra_options include_prompt_templates=1
+```
+
+> [!TIP]
+> Common precision and execution provider combinations include: `FP32 CPU`, `FP32 CUDA`, `FP16 CUDA`, `FP16 DML`, `INT4 CPU`, `INT4 CUDA`, and `INT4 DML`.
+
+Here is a complete example command for converting a model to ONNX format:
+
+```powershell
+python -m onnxruntime_genai.models.builder -m C:\hfmodel\phi3 -p fp16 -e cpu -o C:\Users\Administrator\.aitk\models\microsoft\Phi-3-mini-4k-instruct\cpu\phi3-cpu-int4-rtn-block-32-acc-level-4 -c C:\temp --extra_options include_prompt_templates=1
+```
+
+For more details on precision and execution providers, refer to these tutorials:
+
+- [Precision basics](https://huggingface.co/docs/optimum/en/concept_guides/quantization)
+- [Execution provider basics](https://onnxruntime.ai/docs/execution-providers)
+
+## Load models into AI Toolkit
+
+After conversion, move your ONNX model file into the newly created directory. The AI Toolkit automatically loads ONNX models from this directory upon activation.
+
+You can find your models in the `MY MODELS` view. To use a model, double-click its name or open `TOOLS` > `Playground` and select the model from the dropdown list to start interacting with it.
+
+> Note: The AI Toolkit does not support deleting manually added models directly. To remove a model, delete its directory manually.
+
+## Supported models for conversion
+
+The following table lists models supported for conversion to ONNX format in the AI Toolkit:
+
+| Support Matrix | Supported now | Under development | On the roadmap |
+| :------------: | :-----------: | :---------------: | :------------: |
+| Model architectures | `DeepSeek`, `Gemma`, `Llama`, `Mistral`, `Phi (Language + Vision)`, `Qwen`, `Nemotron`, `Granite`, `AMD OLMo` | `Whisper` | `Stable Diffusion` |
diff --git a/docs/intelligentapps/reference/SetupWithoutAITK.md b/docs/intelligentapps/reference/SetupWithoutAITK.md
@@ -0,0 +1,51 @@
+---
+ContentId: 8493ea9e-061a-4519-b807-57bd320cc60c
+DateApproved: 05/13/2025
+MetaDescription: Model Conversion reference about setup environment.
+---
+
+# Setup model conversion environment manually
+This article introduces how to manually set up Python environments for model conversion across different hardware targets, including CPU, QNN, AMD NPU, and Intel NPU.
+
+We recommend installing your environment in a [Python virtual environment](https://docs.python.org/3/library/venv.html) or a [Conda environment](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html).
+
+> Note: if you want to do inference on NPU devices with QNN EP, we recommend using a [Python virtual environment](https://docs.python.org/3/library/venv.html). The reason is because inference with some models might require the ARM version of Python, while Conda doesn't support creating an ARM version of Python virtual environments yet.
+
+- CPU: [Setup CPU environment](#setup-cpu-environment)
+- QNN: [Setup QNN environment](#setup-qnn-environment)
+- AMD NPU: [Setup AMD NPU environment](#setup-amd-npu-environment)
+- Intel NPU: [Setup Intel NPU environment](#setup-intel-npu-environment)
+
+## Setup CPU environment
+Tips: you can directly setup NPU environment such as [Setup QNN environment](#setup-qnn-environment). It will fallback to CPU EP when running inference on a model.
+
+Alternatively, you can choose to setup a CPU environment.
+
+Python version: 3.12.9
+```
+pip install requirements-CPU.txt
+```
+
+## Setup QNN environment
+Python version: 3.12.9
+```
+pip install requirements-QNN.txt
+```
+
+## Setup AMD NPU environment
+Python version: 3.10.9
+```
+pip install requirements-AMDNPU.txt
+```
+
+## Setup Intel NPU environment
+Python version: 3.10.9
+```
+pip install requirements-IntelNPU.txt
+```
+
+## Install requirements for target project
+Navigate to [model project folder](./FileStructure.md).
+```
+pip install requirements.txt
+```
diff --git a/docs/intelligentapps/reference/TemplateProject.md b/docs/intelligentapps/reference/TemplateProject.md
diff --git a/docs/toc.json b/docs/toc.json
