|
1 | | -The current quantization overview page is a bit sparse: https://pytorch.org/executorch/main/quantization-overview.html. I'd like to update it as follows: |
2 | | - |
3 | | -Move under Usage/ since it's the only page under Quantization/ currently. |
4 | | -Split out information intended for backend authors (info about writing a quantizer, for example). Focus on user-facing APIs. |
5 | | -Document backend-invariant quantization flows (embeddings, ao kernels, etc.). Include info (and example) on composable quantizer. |
6 | | -Document PT2E and quantize_ flows. |
7 | | -Cover the general, high level approach to quantizing different types of models. |
8 | | -CV models |
9 | | -Transformers / language models |
10 | | -Talk briefly about options for evaluating quantized model accuracy (running in eager mode vs pybindings vs on-device, for example) |
11 | | ------ |
12 | | - |
13 | | -# Quantizing ExecuTorch Models |
14 | | - |
15 | | -ExecuTorch uses [torchao](https://github.com/pytorch/ao) for quantization. In general, ExecuTorch quantization is backend specific, and we allow each backned to define exactly how model quantization is done based on the capability of the underlying hardware. |
| 1 | +# Quantization Overview |
16 | 2 |
|
| 3 | +Quantization is a technique that reduces the precision of numbers used in a model’s computations and stored weights—typically from 32-bit floats to 8-bit integers. This reduces the model’s memory footprint, speeds up inference, and lowers power consumption, often with minimal loss in accuracy. |
17 | 4 |
|
18 | | -Each backend defines its own PT2E quantizers. |
| 5 | +Quantization is especially important for deploying models on edge devices such as wearables, embedded systems, and microcontrollers, which often have limited compute, memory, and battery capacity. By quantizing models, we can make them significantly more efficient and suitable for these resource-constrained environments. |
19 | 6 |
|
20 | | -PT2E quantization happens after model export, but before lowering to a backend. |
21 | 7 |
|
| 8 | +# Quantization in ExecuTorch |
| 9 | +ExecuTorch uses [torchao](https://github.com/pytorch/ao/tree/main/torchao) as its quantization library. This integration allows ExecuTorch to leverage PyTorch-native tools for preparing, calibrating, and converting quantized models. |
22 | 10 |
|
23 | | -* [XNNPACK quantization example](backends-xnnpack.md#quantization) |
24 | | -* [CoreML quantization example](backends-coreml.md#quantization) |
25 | 11 |
|
| 12 | +Quantization in ExecuTorch is backend-specific. Each backend defines how models should be quantized based on its hardware capabilities. Most ExecuTorch backends use the torchao [PT2E quantization](https://docs.pytorch.org/ao/main/tutorials_source/pt2e_quant_ptq.html) flow, which works on models exported with torch.export and enables quantization that is tailored for each backend. |
26 | 13 |
|
27 | | -``` |
| 14 | +The PT2E quantization workflow has three main steps: |
28 | 15 |
|
29 | | -``` |
| 16 | +1. Create a backend-specific quantizer. |
| 17 | +2. Prepare, calibrate, convert, and evalute the quantized model in PyTorch |
| 18 | +3. Lower the model to the target backend |
30 | 19 |
|
| 20 | +## 1. Create a Backend-Specific Quantizer |
31 | 21 |
|
| 22 | +Each backend provides its own quantizer (e.g., XNNPACKQuantizer, CoreMLQuantizer) that defines how quantization should be applied to a model in a way that is compatible with the target hardware. |
| 23 | +These quantizers usually support configs that allow users to specify quantization options such as: |
32 | 24 |
|
| 25 | +* Precision (e.g., 8-bit or 4-bit) |
| 26 | +* Quantization type (e.g., dynamic, static, or weight-only quantization) |
| 27 | +* Granularity (e.g., per-tensor, per-channel) |
33 | 28 |
|
| 29 | +Not all quantization options are supported by all backends. Consult backend-specific guides for supported quantization modes and configuration, and how to initialize the backend-specific PT2E quantizer: |
34 | 30 |
|
35 | | -# Quantization Overview |
36 | | -Quantization is a process that reduces the precision of computations and lowers memory footprint in the model. To learn more, please visit the [ExecuTorch concepts page](concepts.md#quantization). This is particularly useful for edge devices including wearables, embedded devices and microcontrollers, which typically have limited resources such as processing power, memory, and battery life. By using quantization, we can make our models more efficient and enable them to run effectively on these devices. |
| 31 | +* [XNNPACK quantization](backends-xnnpack.md#quantization) |
| 32 | +* [CoreML quantization](backends-coreml.md#quantization) |
37 | 33 |
|
38 | | -In terms of flow, quantization happens early in the ExecuTorch stack: |
39 | 34 |
|
40 | | - |
41 | 35 |
|
42 | | -A more detailed workflow can be found in the [ExecuTorch tutorial](https://pytorch.org/executorch/main/tutorials/export-to-executorch-tutorial). |
| 36 | +## 2. Quantize and evaluate the model |
43 | 37 |
|
44 | | -Quantization is usually tied to execution backends that have quantized operators implemented. Thus each backend is opinionated about how the model should be quantized, expressed in a backend specific ``Quantizer`` class. ``Quantizer`` provides API for modeling users in terms of how they want their model to be quantized and also passes on the user intention to quantization workflow. |
| 38 | +After the backend specific quantizer is defined, the PT2E quantization flow is the same for all backends. A generic example is provided below, but specific examples are given in backend documentation: |
45 | 39 |
|
46 | | -Backend developers will need to implement their own ``Quantizer`` to express how different operators or operator patterns are quantized in their backend. This is accomplished via [Annotation API](https://pytorch.org/tutorials/prototype/pt2e_quantizer.html) provided by quantization workflow. Since ``Quantizer`` is also user facing, it will expose specific APIs for modeling users to configure how they want the model to be quantized. Each backend should provide their own API documentation for their ``Quantizer``. |
| 40 | +```python |
| 41 | +from torchao.quantization.pt2e.quantize_pt2e import convert_pt2e, prepare_pt2e |
47 | 42 |
|
48 | | -Modeling users will use the ``Quantizer`` specific to their target backend to quantize their model, e.g. ``XNNPACKQuantizer``. |
| 43 | +training_gm = torch.export.export(model, sample_inputs).module() |
49 | 44 |
|
50 | | -For an example quantization flow with ``XNNPACKQuantizer``, more documentation and tutorials, please see ``Performing Quantization`` section in [ExecuTorch tutorial](https://pytorch.org/executorch/main/tutorials/export-to-executorch-tutorial). |
| 45 | +# Prepare the model for quantization using the backend-specific quantizer instance |
| 46 | +prepared_model = prepare_pt2e(training_gm, quantizer) |
51 | 47 |
|
52 | | -## Source Quantization: Int8DynActInt4WeightQuantizer |
53 | 48 |
|
54 | | -In addition to export based quantization (described above), ExecuTorch wants to highlight source based quantizations, accomplished via [torchao](https://github.com/pytorch/ao). Unlike export based quantization, source based quantization directly modifies the model prior to export. One specific example is `Int8DynActInt4WeightQuantizer`. |
| 49 | +# Calibrate the model on representative data |
| 50 | +for sample in calibration_data: |
| 51 | + prepared_model(sample) |
55 | 52 |
|
56 | | -This scheme represents 4-bit weight quantization with 8-bit dynamic quantization of activation during inference. |
| 53 | +# Convert the calibrated model to a quantized model |
| 54 | +quantized_model = convert_pt2e(prepared_model) |
| 55 | +``` |
57 | 56 |
|
58 | | -Imported with ``from torchao.quantization.quant_api import Int8DynActInt4WeightQuantizer``, this class uses a quantization instance constructed with a specified dtype precision and groupsize, to mutate a provided ``nn.Module``. |
| 57 | +The quantized_model is a PyTorch model like any other, and can be evaluated on different tasks for accuracy. |
| 58 | +Tasks specific benchmarks are the recommended way to evaluate your quantized model, but as crude alternative you can compare to outputs with the original model using generic error metrics like SQNR: |
59 | 59 |
|
| 60 | +```python |
| 61 | +from torchao.quantization.utils import compute_error |
| 62 | +out_reference = model(sample) |
| 63 | +out_quantized = quantized_model(sample) |
| 64 | +sqnr = compute_error(out_reference, out_quantized) # SQNR error |
60 | 65 | ``` |
61 | | -# Source Quant |
62 | | -from torchao.quantization.quant_api import Int8DynActInt4WeightQuantizer |
63 | 66 |
|
64 | | -model = Int8DynActInt4WeightQuantizer(precision=torch_dtype, groupsize=group_size).quantize(model) |
| 67 | +Note that numerics on device can differ those in PyTorch even for unquantized models, and accuracy evaluation can also be done with pybindings or on device. |
65 | 68 |
|
66 | | -# Export to ExecuTorch |
67 | | -from executorch.exir import to_edge |
68 | | -from torch.export import export |
69 | 69 |
|
70 | | -exported_model = export(model, ...) |
71 | | -et_program = to_edge(exported_model, ...).to_executorch(...) |
72 | | -``` |
| 70 | +## 3. Lower the model |
| 71 | + |
| 72 | +The final step is to lower the quantized_model to the desired backend, as you would an unquantized one. See backend-specific pages for lowering information. |
0 commit comments