[Tutorial] OpenVINOQuantizer #2

daniil-lyakhov · 2025-01-28T15:09:05Z

Fixes #ISSUE_NUMBER

Description

Checklist

The issue that is being fixed is referred in the description (see above "Fixes #ISSUE_NUMBER")
Only one issue is addressed in this pull request
Labels from the issue that this PR is fixing are added to this pull request
No unnecessary issues are included into this pull request.

prototype_source/openvino_quantizer.rst

ynimmaga · 2025-02-04T05:27:50Z

prototype_source/openvino_quantizer.rst

+
+    # Create the data, using the dummy data here as an example
+    traced_bs = 50
+    x = torch.randn(traced_bs, 3, 224, 224).contiguous(memory_format=torch.channels_last)


why do we need the memory format to be channels_last?

This is a copy past from the original tutorial, removed, thanks!

ynimmaga · 2025-02-04T05:28:45Z

prototype_source/openvino_quantizer.rst

+    example_inputs = (x,)
+
+    # Capture the FX Graph to be quantized
+    with torch.no_grad(), disable_patching():


is disable_patching() needed both during export and inference with torch.compile?

Unfortunately yes: export will fail with an error and performance of the compiled model will be ruined without it

prototype_source/openvino_quantizer.rst

alexsu52 · 2025-02-07T07:45:23Z

prototype_source/openvino_quantizer.rst

+===========================================================================
+
+**Author**: dlyakhov, asuslov, aamir, # TODO: add required authors
+


Prerequisites

PyTorch 2 Export Post Training Quantization

How to Write a Quantizer for PyTorch 2 Export Quantization

prototype_source/openvino_quantizer.rst

alexsu52 · 2025-02-07T07:58:37Z

prototype_source/openvino_quantizer.rst

+    import nncf
+    from nncf.torch import disable_patching


Suggested change

import nncf

from nncf.torch import disable_patching

import nncf

Unfortunately, that does not work. We can do import nncf.torch and then do nncf.torch.disable_patching

import nncf.torch is introduced, please check

prototype_source/openvino_quantizer.rst

alexsu52 · 2025-02-07T08:02:17Z

prototype_source/openvino_quantizer.rst

+        # from input to output nodes will be excluded from the quantization process.
+        subgraph = nncf.Subgraph(inputs=['layer_1', 'layer_2'], outputs=['layer_3'])
+        OpenVINOQuantizer(ignored_scope=nncf.IgnoredScope(subgraphs=[subgraph]))
+


Where can I find more information about OpenVINOQuantizer parameters?

That's a good question, we don't have a dedicated page about the OpenVINOQuantizer yet. We have a dedicated page for the nncf.quantize and its parameters, but the subset of parameters is not equivalent

I've added a link to nncf API docs, which should be updated with this PR: openvinotoolkit/nncf#3277

alexsu52 · 2025-02-07T08:04:41Z

prototype_source/openvino_quantizer.rst

+Conclusion
+------------
+
+With this tutorial, we introduce how to use torch.compile with the OpenVINO backend and the OpenVINO quantizer.


I would suggest to add somethink like that:
For more information about NNCF and NNCF Quantization Flow for PyTorch models, please visit

Done, please check

Co-authored-by: Alexander Suslov <[email protected]> Co-authored-by: Yamini Nimmagadda <[email protected]>

prototype_source/openvino_quantizer.rst

AlexKoff88 · 2025-02-24T14:07:21Z

prototype_source/openvino_quantizer.rst

+
+The quantization flow mainly includes four steps:
+
+- Step 1: Install OpenVINO and NNCF.


I think the quantization flow itself does not includer step 1. It is just a prerequisite.

Agree, fixed

AlexKoff88 · 2025-02-24T14:13:18Z

prototype_source/openvino_quantizer.rst

+Introduction
+--------------
+
+This tutorial demonstrates how to use `OpenVINOQuantizer` from `Neural Network Compression Framework (NNCF) <https://github.com/openvinotoolkit/nncf/tree/develop>`_ in PyTorch 2 Export Quantization flow to generate a quantized model customized for the `OpenVINO torch.compile backend <https://docs.openvino.ai/2024/openvino-workflow/torch-compile.html>`_ and explains how to lower the quantized model into the `OpenVINO <https://docs.openvino.ai/2024/index.html>`_ representation.


It would be more attractive if to give the user an idea why it may need to use OpenVINOQuantizer (e.g. it is more accurate, performant, etc.)

Make sense! Description of advantages of OpenVINOQuantizer was added

Co-authored-by: Alexander Suslov <[email protected]>

Co-authored-by: Svetlana Karslioglu <[email protected]>

Removing the docs survey banner

Fix code snippet format issue in inductor_windows --------- Co-authored-by: Svetlana Karslioglu <[email protected]>

* Add a note that foreach feature is a prototype

Update the What's New section. --------- Co-authored-by: Svetlana Karslioglu <[email protected]>

* Adjust torch.compile() best practices 1. Add best practice to prefer `mod.compile` over `torch.compile(mod)`, which avoids `_orig_` naming problems. Repro steps: - opt_mod = torch.compile(mod) - train opt_mod - save checkpoint In another script, potentially on a machine that does NOT support `torch.compile`: load checkpoint. This fails with an error, because the checkpoint on `opt_mod` got its params renamed by `torch.compile`: ``` RuntimeError: Error(s) in loading state_dict for VQVAE: Missing key(s) in state_dict: "embedding.weight", "encoder.encoder.net.0.weight", "encoder.encoder.net.0.bias", ... Unexpected key(s) in state_dict: "_orig_mod.embedding.weight", "_orig_mod.encoder.encoder.net.0.weight", "_orig_mod.encoder.encoder.net.0.bias", ... ``` - Add best practice to use, or at least try, `fullgraph=True`. This doesn't always work, but we should encourage it. --------- Co-authored-by: Svetlana Karslioglu <[email protected]>

Co-authored-by: Svetlana Karslioglu <[email protected]>

WIP

f0ab805

daniil-lyakhov force-pushed the dl/fx/openvino_quantizer branch 8 times, most recently from 4b67782 to acf1647 Compare January 28, 2025 19:05

daniil-lyakhov changed the title ~~Dl/fx/openvino quantizer~~ [Tutorial] OpenVINOQuantizer Jan 28, 2025

OpenVINOQuantizer

acf1647

ynimmaga reviewed Feb 4, 2025

View reviewed changes

prototype_source/openvino_quantizer.rst Outdated Show resolved Hide resolved

ynimmaga reviewed Feb 4, 2025

View reviewed changes

prototype_source/openvino_quantizer.rst Outdated Show resolved Hide resolved

ynimmaga reviewed Feb 4, 2025

View reviewed changes

prototype_source/openvino_quantizer.rst Outdated Show resolved Hide resolved

alexsu52 reviewed Feb 7, 2025

View reviewed changes

Apply suggestions from code review

5b1c99a

Co-authored-by: Alexander Suslov <[email protected]> Co-authored-by: Yamini Nimmagadda <[email protected]>

daniil-lyakhov requested review from alexsu52 and ynimmaga February 7, 2025 12:33

Comments

b2eaa82

daniil-lyakhov force-pushed the dl/fx/openvino_quantizer branch 3 times, most recently from f4f592f to af4eb02 Compare February 24, 2025 12:33

NNCF API docs

810899a

daniil-lyakhov force-pushed the dl/fx/openvino_quantizer branch from af4eb02 to 810899a Compare February 24, 2025 12:35

AlexKoff88 reviewed Feb 24, 2025

View reviewed changes

prototype_source/openvino_quantizer.rst Outdated Show resolved Hide resolved

AlexKoff88 reviewed Feb 24, 2025

View reviewed changes

Comments

82a47a5

daniil-lyakhov force-pushed the dl/fx/openvino_quantizer branch from c6c6e46 to 82a47a5 Compare February 24, 2025 14:44

daniil-lyakhov requested a review from AlexKoff88 February 24, 2025 14:46

daniil-lyakhov and others added 3 commits February 24, 2025 15:57

fold_quantize=False

26f044b

Update prototype_source/openvino_quantizer.rst

75d3549

Co-authored-by: Alexander Suslov <[email protected]>

Merge branch 'main' into dl/fx/openvino_quantizer

e8e94d3

daniil-lyakhov force-pushed the dl/fx/openvino_quantizer branch from 1c6bc7c to f09a85f Compare April 14, 2025 13:58

daniil-lyakhov and others added 18 commits April 14, 2025 16:05

Spelling / comments

f09a85f

Merge branch 'main' into dl/fx/openvino_quantizer

2c766e7

Merge branch 'main' into dl/fx/openvino_quantizer

b424f92

prototype_index.rst is updated

f3137be

Apply suggestions from code review

b7d2781

Co-authored-by: Svetlana Karslioglu <[email protected]>

Merge remote-tracking branch 'origin/main' into dl/fx/openvino_quantizer

bb3c2f8

Update prototype_source/openvino_quantizer.rst

c093c76

Co-authored-by: Svetlana Karslioglu <[email protected]>

Remove Docs Survey Banner (pytorch#3340)

ccc02d6

Removing the docs survey banner

Merge branch 'main' into dl/fx/openvino_quantizer

090823f

Fix code snippet format issue in inductor_windows (pytorch#3339)

71695c7

Fix code snippet format issue in inductor_windows --------- Co-authored-by: Svetlana Karslioglu <[email protected]>

Add a note that foreach feature is a prototype (pytorch#3341)

35c68ea

* Add a note that foreach feature is a prototype

Updating tutorials for 2.7. (pytorch#3338)

a5632da

Update the What's New section. --------- Co-authored-by: Svetlana Karslioglu <[email protected]>

Merge branch 'main' into dl/fx/openvino_quantizer

0a422c2

fix index format (pytorch#3343)

bdeca26

Co-authored-by: Svetlana Karslioglu <[email protected]>

fix a typo in optimization_tutorial.py (pytorch#3333)

1988e26

Co-authored-by: Svetlana Karslioglu <[email protected]>

fix a typo in zeroing_out_gradients.py (pytorch#3337)

70d2154

Co-authored-by: Svetlana Karslioglu <[email protected]>

Merge branch 'main' into dl/fx/openvino_quantizer

7e97977

		===========================================================================

		Author: dlyakhov, asuslov, aamir, # TODO: add required authors

	import nncf
	from nncf.torch import disable_patching
	import nncf


		The quantization flow mainly includes four steps:

		- Step 1: Install OpenVINO and NNCF.

[Tutorial] OpenVINOQuantizer #2

Are you sure you want to change the base?

[Tutorial] OpenVINOQuantizer #2

Uh oh!

Conversation

daniil-lyakhov commented Jan 28, 2025

Description

Checklist

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daniil-lyakhov Feb 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Prerequisites

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

daniil-lyakhov Feb 7, 2025 •

edited

Loading