[YN]Openvino backend #2
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alexsu52
reviewed
Feb 11, 2025
examples/openvino/aot/README.md
Outdated
| Enable model quantization: Default is False. | ||
|
|
||
| - **`--dataset`** (optional): | ||
| Path to the calibration dataset. TODO: It is necessary to think in what form to support the dataset. For the experiment, tiny-imagenet is used, which can be downloaded from here http://cs231n.stanford.edu/tiny-imagenet-200.zip and specify the path to it. |
| default="CPU", | ||
| help="Target device for compiling the model (e.g., CPU, GPU). Default is CPU.", | ||
| ) | ||
|
|
There was a problem hiding this comment.
What do you think about adding a --qunatization_flow argument that would have values "pt2e" and "nncf" to specify which flow should be used during quantization?
There was a problem hiding this comment.
Done, please check
| if suite == "torchvision": | ||
| transform = torchvision_models.get_model_weights(model_name).DEFAULT.transforms() | ||
| else: | ||
| transform = create_transform(**resolve_data_config(model.pretrained_cfg, model=model)) |
There was a problem hiding this comment.
Does it work for the huggingface suite?
There was a problem hiding this comment.
No, an exception is added
| elif isinstance(input_shape, list): | ||
| input_shape = tuple(input_shape) | ||
| else: | ||
| msg = "Input shape must be a list or tuple." |
There was a problem hiding this comment.
It looks like if input_shape is tuple I will be here.
| # Export the model to the aten dialect | ||
| aten_dialect: ExportedProgram = export(model, example_args) | ||
|
|
||
| if quantize: |
There was a problem hiding this comment.
Could you implement quantization flow in the separate function?
| print(f"Model exported and saved as {model_name}.pte on {device}.") | ||
| print(f"Model exported and saved as {model_file_name} on {device}.") | ||
|
|
||
| if validate: |
There was a problem hiding this comment.
Could you implement validation in the separate function?
alexsu52
reviewed
Feb 11, 2025
| from nncf.common.graph.graph import NNCFGraph | ||
|
|
||
| QUANT_ANNOTATION_KEY = "quantization_annotation" | ||
|
|
There was a problem hiding this comment.
I suggest to introduce the following class:
class QuantizationMode(StrEnum):
"""
Defines special quantization modes.
- INT8_SYM: INT8 symmetric quantization for both activations and weights.
- INT8_MIXED: INT8 asymmetric quantization for activations, symmetric for weights.
- INT8_TRANSFORMER: Optimized INT8 quantization for transformer-based models
"""
INT8_SYM = "int8_sym"
INT8_MIXED = "int8_mixed"
INT8_TRANSFORMER = "int8_transformer"
Comment on lines
36
to
54
| def __init__( | ||
| self, | ||
| *, | ||
| mode: Optional[p.QuantizationMode] = None, | ||
| preset: Optional[q.structs.QuantizationPreset] = None, | ||
| target_device: p.TargetDevice = p.TargetDevice.ANY, | ||
| transformer_model: bool = False, | ||
| ignored_scope: Optional[nncf.IgnoredScope] = None, | ||
| overflow_fix: Optional[advanced_p.OverflowFix] = None, | ||
| quantize_outputs: bool = False, | ||
| activations_quantization_params: Optional[advanced_p.QuantizationParameters] = None, | ||
| weights_quantization_params: Optional[advanced_p.QuantizationParameters] = None, | ||
| ): |
There was a problem hiding this comment.
What do you think?
Suggested change
| def __init__( | |
| self, | |
| *, | |
| mode: Optional[p.QuantizationMode] = None, | |
| preset: Optional[q.structs.QuantizationPreset] = None, | |
| target_device: p.TargetDevice = p.TargetDevice.ANY, | |
| transformer_model: bool = False, | |
| ignored_scope: Optional[nncf.IgnoredScope] = None, | |
| overflow_fix: Optional[advanced_p.OverflowFix] = None, | |
| quantize_outputs: bool = False, | |
| activations_quantization_params: Optional[advanced_p.QuantizationParameters] = None, | |
| weights_quantization_params: Optional[advanced_p.QuantizationParameters] = None, | |
| ): | |
| def __init__( | |
| self, | |
| *, | |
| mode: QuantizationMode = QuantizationMode.INT8_MIXED, | |
| ignored_scope: Optional[nncf.IgnoredScope] = None, | |
| **kwargs | |
| ): |
There was a problem hiding this comment.
I like the constructor parameters subset, refactored
There was a problem hiding this comment.
Also, I set default QuantizationMode == INT8_SYM to alight with the default MinMax parameters (which were default parameters before) https://github.com/openvinotoolkit/nncf/blob/develop/nncf/quantization/algorithms/min_max/algorithm.py#L211-L215
daniil-lyakhov
force-pushed
the
dl/openvino/model_enabling
branch
from
ddcbb11 to
01b88f8
Compare
alexsu52
reviewed
Feb 12, 2025
| self, | ||
| *, | ||
| mode: Optional[QuantizationMode] = QuantizationMode.INT8_SYM, | ||
| ignored_scope: Optional[nncf.IgnoredScope] = None, |
There was a problem hiding this comment.
Ignored scope is model specific information. What do you think about introducing additional method to set ignored scope?
def set_ignored_scope(names: Optional[List[str]] = None, patterns: Optional[List[str]] = None, types: Optional[List[str]] = None, subgraphs: Optional[List[Tuple[List[str], List[str]]]] = None, validate: bool = True)
573f316 to
c499e3c
Compare
Co-authored-by: Alexander Suslov <[email protected]>
daniil-lyakhov
force-pushed
the
dl/openvino/model_enabling
branch
2 times, most recently
from
fa6744b to
c7e0758
Compare
alexsu52
pushed a commit
to openvinotoolkit/nncf
that referenced
this pull request
Feb 24, 2025
### Changes Mark quantizer and quantize_pt2e as API ### Reason for changes To introduce `OpenVINOQuantizer` and `quantize_pt2e` in the api docs: https://openvinotoolkit.github.io/nncf/index.html ### Related tickets daniil-lyakhov/executorch#2
daniil-lyakhov
pushed a commit
that referenced
this pull request
Jun 4, 2025
Differential Revision: D75034439 Pull Request resolved: pytorch#11011
daniil-lyakhov
pushed a commit
that referenced
this pull request
Jun 11, 2025
Differential Revision: D75982351 Pull Request resolved: pytorch#11456
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
[PLEASE REMOVE] See CONTRIBUTING.md's Pull Requests for ExecuTorch PR guidelines.
[PLEASE REMOVE] If this PR closes an issue, please add a
Fixes #<issue-id>line.[PLEASE REMOVE] If this PR introduces a fix or feature that should be the upcoming release notes, please add a "Release notes: " label. For a list of available release notes labels, check out CONTRIBUTING.md's Pull Requests.
Test plan
[PLEASE REMOVE] How did you test this PR? Please write down any manual commands you used and note down tests that you have written if applicable.