How to use NNCF to perform unsigned asymmetric quantization?

[huangqiu15444]

my cdoe:
import nncf
from nncf.quantization.advanced_parameters import AdvancedQuantizationParameters,QuantizationParameters
from nncf import QuantizationMode
weights_quantization_params = QuantizationParameters()
weights_quantization_params.per_channel = False
weights_quantization_params.signedness_to_force = False
weights_quantization_params.mode = 'asymmetric'

add_quant_params = AdvancedQuantizationParameters()
add_quant_params.weights_quantization_params = weights_quantization_params

quantized_model = nncf.quantize(model, quantization_dataset, advanced_parameters=add_quant_params)

errr report:
ValueError: Quantization parameter constraints specified in NNCF config are incompatible with HW capabilities as specified in HW config type 'CPU'. First conflicting quantizer location: ResNet/Conv2d[conv1]/conv2d_0

[alexsu52]

Hello @huangqiu15444,

The provided example specifies the symmetric for activation and asymmetric for weights quantization scheme. May I know your motivation for using this quantization scheme?

NNCF returns an error because this quantization scheme is not hardware friendly and does not supported in OpenVINO.

[huangqiu15444]

When deploying my model to the device, it needs to go through “asymmetric_quantized - u8” quantization. I want to conduct quantization-aware training through NNCF to adjust the weight distribution of the model.

[huangqiu15444]

If I want to set the quantization methods for both activation and weights to “asymmetric_quantized - u8”, does NNCF support this?

weight quantization

weights_quantization_params = QuantizationParameters()
weights_quantization_params.per_channel = False
weights_quantization_params.signedness_to_force = False 
weights_quantization_params.mode = 'asymmetric'


#  activation quantization
activations_quantization_params = QuantizationParameters()
activations_quantization_params.per_channel = False
activations_quantization_params.signedness_to_force = False 
activations_quantization_params.mode = 'asymmetric'

[alexsu52]

Yes, this should work on the develop branch. Please update NNCF.

You can try the following code in the NNCF PyTorch QAT example. If you have any problems let me know.

    quantized_model = nncf.quantize(
        model,
        quantization_dataset,
        advanced_parameters=nncf.AdvancedQuantizationParameters(
            overflow_fix=OverflowFix.DISABLE,
            activations_quantization_params=QuantizationParameters(
                mode=QuantizationScheme.ASYMMETRIC, per_channel=False
            ),
            weights_quantization_params=QuantizationParameters(
                mode=QuantizationScheme.ASYMMETRIC, per_channel=False
            ),
        ),
    )

[huangqiu15444]

Excuse me, shouldn’t the parameter ‘signedness_to_force’ be set to False? I’ve found that even when I set ‘signedness_to_force’ to False, the above error still occurs.

[alexsu52]

@huangqiu15444, you should not set signedness_to_force=False for the asymmetric quantization scheme, as it's parameter of NNCF Quantizer and does not affect the data type of the quantized weights after model export. signedness_to_force is only applicable for symmetric quantization to force the use of a symmetric range around zero [-2^(num_bits - 1), 2^(num_bits - 1) - 1] for activations and weights, even if the actual range is positive (symmetric quantization).

What format are you going to export the PyTorch model to after QAT?

[huangqiu15444]

Onnx format.

[alexsu52]

Onnx format.

The quantized PyTorch model should be prepared before exporting to onnx as follows:

stripped_model = nncf.strip(quantized_model)

The stip function replaces the custom NNCF FakeQuantize modules with PyTorch one. The quantized model can then be exported to onnx in QDQ format, where the weights will be stored in the original precision of the model and the output type of QuantizeLinear will be uint8.