Quantization - nncf

[Philius95]

Hello,

I am currently working on a project to perform inference with anomaly detection (AD) models on an embedded system and am investigating ways to perform quantization.

Until now I trained a PaDiM model on a custom dataset, which performes very well.

I have already worked through the OpenVINO NNCF notebook but realized that the insertion of "Fake Quantization Layers" does not suit my project's needs.

I also attempted to quantize the trained PaDiM model, which I converted to TensorFlow, using the TensorFlow Lite converter, but this approach also seems to be a dead end.

Is there a way to perform "true" quantization to int8 with nncf?
Or could someone point me in the right direction?

[samet-akcay]

@AlexKoff88, @alexsu52, anyone can help from your end

[AlexKoff88]

Hi @Philius95,

Are you planning to use OpenVINO for inference? We mostly support optimization for OpenVINO in NNCF.
Have you seen this example: https://github.com/openvinotoolkit/nncf/tree/develop/examples/quantization_aware_training/torch/anomalib

and this one: https://github.com/openvinotoolkit/nncf/tree/develop/examples/post_training_quantization/openvino/anomaly_stfpm_quantize_with_accuracy_control

@alexsu52, can you please comment if PaDiM works OOB now? I guess it should work at least if we quantize OpenVINO IR.