OpenVINOQuantizer

Author: daniil-lyakhov

prototype_source/openvino_nncf_quantization.rst renamed to prototype_source/openvino_quantizer.rst

@@ -1,4 +1,4 @@
-PyTorch 2 Export Quantization with NNCF quantization and OpenVINO runtime
+PyTorch 2 Export Quantization with OpenVINO backend
 ===========================================================================
 
 **Author**: dlyakhov, asuslov, aamir, # TODO: add required authors
@@ -9,13 +9,13 @@ Introduction
 This tutorial introduces the steps for utilizing the `Neural Network Compression Framework (nncf) <https://github.com/openvinotoolkit/nncf/tree/develop>`_ to generate a quantized model customized
 for the `OpenVINO torch.compile backend <https://docs.openvino.ai/2024/openvino-workflow/torch-compile.html>`_ and explains how to lower the quantized model into the `OpenVINO <https://docs.openvino.ai/2024/index.html>`_ representation.
 
-The pytorch 2 export quantization flow uses the torch.export to capture the model into a graph and perform quantization transformations on top of the ATen graph.
+The pytorch 2 export quantization flow uses the torch.export to capture the model into a graph and performs quantization transformations on top of the ATen graph.
 This approach is expected to have significantly higher model coverage, better programmability, and a simplified UX.
 OpenVINO is the new backend that compiles the FX Graph generated by TorchDynamo into an optimized OpenVINO model.
 
-The quantization flow mainly includes three steps:
+The quantization flow mainly includes four steps:
 
-- Step 1: OpenVINO and NNCF installation.
+- Step 1: Install OpenVINO and NNCF.
 - Step 2: Capture the FX Graph from the eager Model based on the `torch export mechanism <https://pytorch.org/docs/main/export.html>`_.
 - Step 3: Apply the Quantization flow based on the captured FX Graph.
 - Step 4: Lower the quantized model into OpenVINO representation with the API ``torch.compile``.
@@ -33,9 +33,14 @@ The high-level architecture of this flow could look like this:
                                 |
                         FX Graph in ATen
                                 |
-                                |
+                                |           OpenVINOQuantizer
+                                |                 /
     —--------------------------------------------------------
-    |                     nncf.quantize                     |
+    |                      prepare_pt2e                     |
+    |                           |                           |
+    |                       Calibrate
+    |                           |                           |
+    |                      convert_pt2e                     |
     —--------------------------------------------------------
                                 |
                          Quantized Model
@@ -69,10 +74,13 @@ We will start by performing the necessary imports, capturing the FX Graph from t
 
 .. code-block:: python
 
-    import torch
-    import torchvision.models as models
     import copy
     import openvino.torch
+    import torch
+    import torchvision.models as models
+    from torch.ao.quantization.quantize_pt2e import convert_pt2e
+    from torch.ao.quantization.quantize_pt2e import prepare_pt2e
+    from torch.ao.quantization.quantizer.openvino_quantizer import OpenVINOQuantizer
 
     import nncf
     from nncf.torch import disable_patching
@@ -90,109 +98,90 @@ We will start by performing the necessary imports, capturing the FX Graph from t
     example_inputs = (x,)
 
     # Capture the FX Graph to be quantized
-    with torch.no_grad():
-        with disable_patching():
-            exported_model = torch.export.export(model, example_inputs).module()
+    with torch.no_grad(), disable_patching():
+        exported_model = torch.export.export(model, example_inputs).module()
 
 
-Next, we will have the FX Module to be quantized.
 
 3. Apply Quantization
 ^^^^^^^^^^^^^^^^^^^^^^^
 
-Before the quantization, we need to create an instance of the nncf.Dataset class that represents the calibration dataset.
-The ``nncf.Dataset`` class can be a wrapper over the framework dataset object that is used for model training or validation
-The class constructor receives the dataset object and an optional transformation function.
+After we capture the FX Module to be quantized, we will import the OpenVINOQuantizer.
 
-The transformation function is a function that takes a sample from the dataset and returns data that can be passed to the model for inference.
-For example, this function can take a tuple of a data tensor and labels tensor and return the former while ignoring the latter.
-The transformation function is used to avoid modifying the dataset code to make it compatible with the quantization API.
-The function is applied to each sample from the dataset before passing it to the model for inference.
-The following code snippet shows how to create an instance of the ``nncf.Dataset`` class:
 
 .. code-block:: python
 
-    calibration_loader = torch.utils.data.DataLoader([example_inputs])
+    quantizer = OpenVINOQuantizer()
 
-    def transform_fn(data_item):
-        # In the transformation function,
-        # user can separate labels and input data
-        # from the given data item:
-        # images, _ = data_item
-        return data_item
+``OpenVINOQuantizer`` has several optional parameters that allow tuning the quantization process to get a more accurate model.
+Below is the list of essential parameters and their description:
 
-    calibration_dataset = nncf.Dataset(calibration_loader, transform_fn)
 
-If there is no framework dataset object, you can create your own entity that implements the Iterable interface in Python,
-for example, the list of images, and returns data samples feasible for inference. In this case, a transformation function is not required.
+* ``preset`` - defines quantization scheme for the model. Two types of presets are available:
 
-Once the dataset is ready and the model object is instantiated, you can apply 8-bit quantization to it.
+    * ``PERFORMANCE`` (default) - defines symmetric quantization of weights and activations
 
-.. code-block:: python
+    * ``MIXED`` - weights are quantized with symmetric quantization and the activations are quantized with asymmetric quantization. This preset is recommended for models with non-ReLU and asymmetric activation functions, e.g. ELU, PReLU, GELU, etc.
 
-    with disable_patching():
-        quantized_model = nncf.quantize(exported_model, calibration_dataset)
+    .. code-block:: python
 
-``nncf.quantize()`` function has several optional parameters that allow tuning the quantization process to get a more accurate model.
-Below is the list of parameters and their description:
+        OpenVINOQuantizer(preset=nncf.QuantizationPreset.MIXED)
 
 * ``model_type`` - used to specify quantization scheme required for specific type of the model. Transformer is the only supported special quantization scheme to preserve accuracy after quantization of Transformer models (BERT, DistilBERT, etc.). None is default, i.e. no specific scheme is defined.
-.. code-block:: python
 
-    nncf.quantize(model, dataset, model_type=nncf.ModelType.Transformer)
+    .. code-block:: python
 
-* ``preset`` - defines quantization scheme for the model. Two types of presets are available:
+        OpenVINOQuantizer(model_type=nncf.ModelType.Transformer)
 
-    * ``PERFORMANCE`` (default) - defines symmetric quantization of weights and activations
+* ``ignored_scope`` - this parameter can be used to exclude some layers from the quantization process to preserve the model accuracy.  For example, when you want to exclude the last layer of the model from quantization.  Below are some examples of how to use this parameter:
 
-    * ``MIXED`` - weights are quantized with symmetric quantization and the activations are quantized with asymmetric quantization. This preset is recommended for models with non-ReLU and asymmetric activation functions, e.g. ELU, PReLU, GELU, etc.
+    .. code-block:: python
 
-.. code-block:: python
+        #Exclude by layer name:
+        names = ['layer_1', 'layer_2', 'layer_3']
+        OpenVINOQuantizer(ignored_scope=nncf.IgnoredScope(names=names))
 
-    nncf.quantize(model, dataset, preset=nncf.QuantizationPreset.MIXED)
+        #Exclude by layer type:
+        types = ['Conv2d', 'Linear']
+        OpenVINOQuantizer(ignored_scope=nncf.IgnoredScope(types=types))
 
-* ``fast_bias_correction`` - when set to False, enables a more accurate bias (error) correction algorithm that can be used to improve the accuracy of the model. True is used by default to minimize quantization time.
+        #Exclude by regular expression:
+        regex = '.*layer_.*'
+        OpenVINOQuantizer(ignored_scope=nncf.IgnoredScope(patterns=regex))
 
-.. code-block:: python
+        #Exclude by subgraphs:
+        # In this case, all nodes along all simple paths in the graph
+        # from input to output nodes will be excluded from the quantization process.
+        subgraph = nncf.Subgraph(inputs=['layer_1', 'layer_2'], outputs=['layer_3'])
+        OpenVINOQuantizer(ignored_scope=nncf.IgnoredScope(subgraphs=[subgraph]))
 
-    nncf.quantize(model, dataset, fast_bias_correction=False)
 
-* ``subset_size`` - defines the number of samples from the calibration dataset that will be used to estimate quantization parameters of activations. The default value is 300.
+* ``target_device`` - defines the target device, the specificity of which will be taken into account during optimization. The following values are supported: ``ANY`` (default), ``CPU``, ``CPU_SPR``, ``GPU``, and ``NPU``.
 
-.. code-block:: python
+    .. code-block:: python
 
-    nncf.quantize(model, dataset, subset_size=1000)
+        OpenVINOQuantizer(target_device=nncf.TargetDevice.CPU)
 
-* ``ignored_scope`` - this parameter can be used to exclude some layers from the quantization process to preserve the model accuracy.  For example, when you want to exclude the last layer of the model from quantization.  Below are some examples of how to use this parameter:
 
-.. code-block:: python
+After we import the backend-specific Quantizer, we will prepare the model for post-training quantization.
+``prepare_pt2e`` folds BatchNorm operators into preceding Conv2d operators, and inserts observers in appropriate places in the model.
 
-    #Exclude by layer name:
-    names = ['layer_1', 'layer_2', 'layer_3']
-    nncf.quantize(model, dataset, ignored_scope=nncf.IgnoredScope(names=names))
+.. code-block:: python
 
-    #Exclude by layer type:
-    types = ['Conv2d', 'Linear']
-    nncf.quantize(model, dataset, ignored_scope=nncf.IgnoredScope(types=types))
+    prepared_model = prepare_pt2e(exported_model, quantizer)
 
-    #Exclude by regular expression:
-    regex = '.*layer_.*'
-    nncf.quantize(model, dataset, ignored_scope=nncf.IgnoredScope(patterns=regex))
+Now, we will calibrate the ``prepared_model`` after the observers are inserted in the model.
 
-    #Exclude by subgraphs:
-    # In this case, all nodes along all simple paths in the graph
-    # from input to output nodes will be excluded from the quantization process.
-    subgraph = nncf.Subgraph(inputs=['layer_1', 'layer_2'], outputs=['layer_3'])
-    nncf.quantize(model, dataset, ignored_scope=nncf.IgnoredScope(subgraphs=[subgraph]))
+.. code-block:: python
 
+    # We use the dummy data as an example here
+    prepared_model(*example_inputs)
 
-* ``target_device`` - defines the target device, the specificity of which will be taken into account during optimization. The following values are supported: ``ANY`` (default), ``CPU``, ``CPU_SPR``, ``GPU``, and ``NPU``.
+Finally, we will convert the calibrated Model to a quantized Model. ``convert_pt2e`` takes a calibrated model and produces a quantized model.
 
 .. code-block:: python
 
-    nncf.quantize(model, dataset, target_device=nncf.TargetDevice.CPU)
-
-* ``advanced_parameters`` - used to specify advanced quantization parameters for fine-tuning the quantization algorithm.  Defined by nncf.quantization.advanced_parameters NNCF submodule.  None is default.
+    quantized_model = convert_pt2e(prepared_model)
 
 After these steps, we finished running the quantization flow, and we will get the quantized model.
 
@@ -204,18 +193,19 @@ After that the FX Graph can utilize OpenVINO optimizations using `torch.compile(
 
 .. code-block:: python
 
-    with torch.no_grad():
+    with torch.no_grad(), disable_patching():
         optimized_model = torch.compile(quantized_model, backend="openvino")
 
         # Running some benchmark
         optimized_model(*example_inputs)
 
 
+
 The optimized model is using low-level kernels designed specifically for Intel CPU.
 This should significantly speed up inference time in comparison with the eager model.
 
 Conclusion
 ------------
 
-With this tutorial, we introduce how to use torch.compile with the OpenVINO backend with models quantized via ``nncf.quantize``.
-For further information, please visit `complete example on renset18 model <https://github.com/openvinotoolkit/nncf/tree/v2.14.0/examples/post_training_quantization/torch_fx/resnet18>`_.
+With this tutorial, we introduce how to use torch.compile with the OpenVINO backend and the OpenVINO quantizer.
+For further information, please visit `OpenVINO deploymet via torch.compile documentation <https://docs.openvino.ai/2024/openvino-workflow/torch-compile.html>`_.