Update OV / NNCF/ ModelAPI / Datumaro (#2090)

* Update ov to a pre-release version

* Update OV to 2023 release

* dummy fix for unit tests

* Update

* Apply a workaround for OV 23 bug

* Fix getting rt info

* Tmp update of NNCF

* Changes for supporting NNCF 2.5 (WIP)

* Add new model api to reqs

* Integration for classification

* Update model contaner

* Fix OTX segm wrapper

* Fix linter

* Update detection inference

* Fix MRCNN inference

* Fix tiling inference

* Fix cls unit tests

* Fix detection unit tests

* Fix semantic segmentation UT

* Fix datumaro tests

* Ugrade mapi in action and anomaly

* Update exp code commit ver

* Add raw scores to cls config

* Revert changes in graph parsing test

* Fix model creation in modelAdapter

* Fix metadata shape in ModelAdapter

* Fix tile classifier config

* Update exportable code reqs

* Fix model container

* Del unused import

* Update NNCF to release version

* Xfail movinet inference

* Updatemapi

* Update exportable code commit

* Fix act test xfail

* Fix tiling inference

Fix tiling inference

* Fix imports

* Fix exports in unit tests

* Fix tiling test

* Swithc classification to PTQ

* Extract quantization check

* Move transform to dataset

* Switch to ptq in segmentation

* Fix mypy

* Fix ptq naming

* Switch to PTQ in action task

* Fix cls optimize test

* Fix segmentation ptq test

* Fix action tests

* PTQ for detection task

* Use ignored_scope in segm pqt

* Extract OV data loader to a standalone class

* PTQ for anomaly

* Shuffler for action PTQ dataset

* Fix det unit tests

* Del unused import

* Update exportable code deps

* Fix exportable code tiling

* Update exportable code reqs

* Fix tiling in exportable code

* Use py configs for PTQ segmentation

* Add py style PTQ configs

* Del outdated pot segm configs

* Update ptq configs for anomaly

* Fix cls explain inference

* Fix activation cls wrapper

* PArtially fix OMZ plugin

* Fix opset version collisions

* Partially fix nncf ckpt

* Fix MRCNN wrapper

* Fix softmax opset ver

* Update changelog

* Update exportable code reqs

* Update ov related imports

* Update fq num refs for semantic segmentation

* Update fq refs for classification

* Add update mode for FQ refs

* Update fq refs for detection

* Xfaail e2e for movinet

* Update sses fqs pt2

* Upgrade nncf anomalib integration

* Upgrade nncf weights loading workaround in anomaly task

* Update FQ refs for anomaly

* Bypass PTQ for tile classifier

* Update IS FQ refs

* Update detection FQ refs

* Fix tiling classifier PTQ bypass

* Minor updates in tests

* Fix ruff

* Fix tile classifier PTQ bypass condition

* Fix a typo

* Update det ut

* Update fq refs for tiling iseg

* Fix tiling inference configuration

* Fix readme

* Update datumaro commit

* Update datumaro commit

* Update masks postprocessing in OV wrapper

* Update mrcnn unit tests

* Fix yolox inference config

* Update docs for NNCF state dict hooks

* Fix comments formatting

* Skip action det ov e2e tests

* Update IS tests skips

* Skip some of SS PTQ tests

* Fix a type in skip comment

* Fix data pipeline for tiling yolox

* Revert ss PTQ->POT

* Re-enable POT tests for sseg

* Fix ptq config managing

* Fix black

* Update refs to PTQ configs in docs

* Update FQs refs for anomaly

* Update FQ refs for SS models

---------

Co-authored-by: Nikita Savelyev <[email protected]>
Co-authored-by: Jaeguk Hyun <[email protected]>

Author: 3 people

CHANGELOG.md

@@ -21,6 +21,7 @@ All notable changes to this project will be documented in this file.
 - Set persistent_workers and pin_memory as True in detection task (<https://github.com/openvinotoolkit/training_extensions/pull/2224>)
 - New algorithm for Semi-SL semantic segmentation based on metric lerning via class prototypes (https://github.com/openvinotoolkit/training_extensions/pull/2156)
 - Self-SL for classification now can recieve just folder with any images to start contrastive pretraining (https://github.com/openvinotoolkit/training_extensions/pull/2219)
+- Update OpenVINO version to 2023.0, and NNCF verion to 2.5 (<https://github.com/openvinotoolkit/training_extensions/pull/2090>)
 - Improve XAI saliency map generation for tiling detection and tiling instance segmentation (https://github.com/openvinotoolkit/training_extensions/pull/2240)
 
 ### Bug fixes
@@ -30,7 +31,7 @@ All notable changes to this project will be documented in this file.
 
 ### Known issues
 
-- OpenVINO(==2022.3) IR inference is not working well on 2-stage models (e.g. Mask-RCNN) exported from torch==1.13.1
+- OpenVINO(==2023.0) IR inference is not working well on 2-stage models (e.g. Mask-RCNN) exported from torch==1.13.1
 
 ## \[v1.3.1\]
 

docs/source/guide/explanation/additional_features/models_optimization.rst

@@ -4,14 +4,14 @@ Models Optimization
 OpenVINO™ Training Extensions provides two types of optimization algorithms: `Post-training Optimization Tool (POT) <https://docs.openvino.ai/latest/pot_introduction.html#doxid-pot-introduction>`_ and `Neural Network Compression Framework (NNCF) <https://github.com/openvinotoolkit/nncf>`_.
 
 *******************************
-Post-training Optimization Tool 
+Post-training Optimization Tool
 *******************************
 
 POT is designed to optimize the inference of models by applying post-training methods that do not require model retraining or fine-tuning. If you want to know more details about how POT works and to be more familiar with model optimization methods, please refer to `documentation <https://docs.openvino.ai/latest/pot_introduction.html#doxid-pot-introduction>`_.
 
 To run Post-training optimization it is required to convert the model to OpenVINO™ intermediate representation (IR) first. To perform fast and accurate quantization we use ``DefaultQuantization Algorithm`` for each task. Please, see the `DefaultQuantization Parameters <https://docs.openvino.ai/latest/pot_compression_algorithms_quantization_default_README.html#doxid-pot-compression-algorithms-quantization-default-r-e-a-d-m-e>`_ for further information about configuring the optimization.
 
-POT parameters can be found and configured in ``template.yaml`` and ``configuration.yaml`` for each task. For Anomaly and Semantic Segmentation tasks, we have separate configuration files for POT, that can be found in the same directory with ``template.yaml``, for example for `PaDiM <https://github.com/openvinotoolkit/training_extensions/blob/develop/src/otx/algorithms/anomaly/configs/classification/padim/pot_optimization_config.json>`_, `OCR-Lite-HRNe-18-mod2 <https://github.com/openvinotoolkit/training_extensions/blob/develop/src/otx/algorithms/segmentation/configs/ocr_lite_hrnet_18_mod2/pot_optimization_config.json>`_ model.
+POT parameters can be found and configured in ``template.yaml`` and ``configuration.yaml`` for each task. For Anomaly and Semantic Segmentation tasks, we have separate configuration files for POT, that can be found in the same directory with ``template.yaml``, for example for `PaDiM <https://github.com/openvinotoolkit/training_extensions/blob/develop/src/otx/algorithms/anomaly/configs/classification/padim/ptq_optimization_config.py>`_, `OCR-Lite-HRNe-18-mod2 <https://github.com/openvinotoolkit/training_extensions/blob/develop/src/otx/algorithms/segmentation/configs/ocr_lite_hrnet_18_mod2/ptq_optimization_config.py>`_ model.
 
 ************************************
 Neural Network Compression Framework
@@ -23,8 +23,8 @@ The process of optimization is controlled by the NNCF configuration file. A JSON
 You can refer to configuration files for default templates for each task accordingly: `Classification <https://github.com/openvinotoolkit/training_extensions/blob/develop/src/otx/algorithms/classification/configs/efficientnet_b0_cls_incr/compression_config.json>`_, `Object Detection <https://github.com/openvinotoolkit/training_extensions/blob/develop/src/otx/algorithms/detection/configs/detection/mobilenetv2_atss/compression_config.json>`_, `Semantic segmentation <https://github.com/openvinotoolkit/training_extensions/blob/develop/src/otx/algorithms/segmentation/configs/ocr_lite_hrnet_18_mod2/compression_config.json>`_, `Instance segmentation <https://github.com/openvinotoolkit/training_extensions/blob/develop/src/otx/algorithms/detection/configs/instance_segmentation/efficientnetb2b_maskrcnn/compression_config.json>`_, `Anomaly classification <https://github.com/openvinotoolkit/training_extensions/blob/develop/src/otx/algorithms/anomaly/configs/classification/padim/compression_config.json>`_, `Anomaly Detection <https://github.com/openvinotoolkit/training_extensions/blob/develop/src/otx/algorithms/anomaly/configs/detection/padim/compression_config.json>`_, `Anomaly segmentation <https://github.com/openvinotoolkit/training_extensions/blob/develop/src/otx/algorithms/anomaly/configs/segmentation/padim/compression_config.json>`_. Configs for other templates can be found in the same directory.
 
 
-NNCF tends to provide better quality in terms of preserving accuracy as it uses training compression approaches. 
-Compression results achievable with the NNCF can be found `here <https://github.com/openvinotoolkit/nncf#nncf-compressed-model-zoo>`_ . 
+NNCF tends to provide better quality in terms of preserving accuracy as it uses training compression approaches.
+Compression results achievable with the NNCF can be found `here <https://github.com/openvinotoolkit/nncf#nncf-compressed-model-zoo>`_ .
 Meanwhile, the POT is faster but can degrade accuracy more than the training-enabled approach.
 
 .. note::

requirements/base.txt

@@ -4,7 +4,7 @@ natsort>=6.0.0
 prettytable
 protobuf>=3.20.0
 pyyaml
-datumaro==1.3.2
+datumaro@ git+https://github.com/openvinotoolkit/datumaro@3e77b3138d063db68a4efba3c03a6bac7df086b1#egg=datumaro
 psutil
 scipy>=1.8
 bayesian-optimization>=1.2.0

requirements/openvino.txt

@@ -1,8 +1,8 @@
 # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
 # OpenVINO Requirements.                                                      #
-nncf==2.4.0
+nncf==2.5.0
 onnx==1.13.0
-openmodelzoo-modelapi==2022.3.0
-openvino==2022.3.0
-openvino-dev==2022.3.0
+openvino-model-api==0.1.2
+openvino==2023.0
+openvino-dev==2023.0
 openvino-telemetry>=2022.1.0

src/otx/algorithms/action/adapters/openvino/dataloader.py

@@ -18,13 +18,12 @@
 from typing import Dict, List
 
 import numpy as np
-from compression.api import DataLoader
 
 from otx.api.entities.annotation import AnnotationSceneEntity
 from otx.api.entities.datasets import DatasetEntity, DatasetItemEntity
 
 
-def get_ovdataloader(dataset: DatasetEntity, task_type: str, clip_len: int, width: int, height: int) -> DataLoader:
+def get_ovdataloader(dataset: DatasetEntity, task_type: str, clip_len: int, width: int, height: int):
     """Find proper dataloader for dataset and task type.
 
     If dataset has only a single video, this returns DataLoader for online demo
@@ -49,7 +48,7 @@ def _is_multi_video(dataset: DatasetEntity) -> bool:
     return False
 
 
-class ActionOVDemoDataLoader(DataLoader):
+class ActionOVDemoDataLoader:
     """DataLoader for online demo purpose.
 
     Since it is for online demo purpose it selects background frames from neighbor of key frame
@@ -91,7 +90,7 @@ def add_prediction(self, data: List[DatasetItemEntity], prediction: AnnotationSc
             dataset_item.append_annotations(prediction.annotations)
 
 
-class ActionOVClsDataLoader(DataLoader):
+class ActionOVClsDataLoader:
     """DataLoader for evaluation of action classification models.
 
     It iterates through clustered video, and it samples frames from given video
@@ -151,7 +150,7 @@ def add_prediction(self, dataset: DatasetEntity, data: List[DatasetItemEntity],
                 dataset_item.append_labels(prediction.annotations[0].get_labels())
 
 
-class ActionOVDetDataLoader(DataLoader):
+class ActionOVDetDataLoader:
     """DataLoader for evaluation of spatio-temporal action detection models.
 
     It iterates through DatasetEntity, which only contains non-empty frame(frame with actor annotation)

src/otx/algorithms/action/adapters/openvino/model_wrappers/openvino_models.py

@@ -19,22 +19,16 @@
 from typing import Any, Dict, List
 
 import numpy as np
+from openvino.model_api.adapters import OpenvinoAdapter
+from openvino.model_api.models.model import Model
+from openvino.model_api.models.utils import (
+    RESIZE_TYPES,
+    Detection,
+    InputTransform,
+)
 
 from otx.api.entities.datasets import DatasetItemEntity
 
-try:
-    from openvino.model_zoo.model_api.adapters import OpenvinoAdapter
-    from openvino.model_zoo.model_api.models.model import Model
-    from openvino.model_zoo.model_api.models.utils import (
-        RESIZE_TYPES,
-        Detection,
-        InputTransform,
-    )
-except ImportError as e:
-    import warnings
-
-    warnings.warn(f"{e}, ModelAPI was not found.")
-
 
 def softmax_numpy(x: np.ndarray):
     """Softmax numpy."""

src/otx/algorithms/action/adapters/openvino/task.py

@@ -18,25 +18,26 @@
 import json
 import logging
 import os
+import random
 import tempfile
 from typing import Any, Dict, List, Optional, Tuple, Union
 from zipfile import ZipFile
 
+import nncf
 import numpy as np
-from addict import Dict as ADDict
-from compression.api import DataLoader
-from compression.engines.ie_engine import IEEngine
-from compression.graph import load_model, save_model
-from compression.graph.model_utils import compress_model_weights, get_nodes_by_type
-from compression.pipeline.initializer import create_pipeline
+import openvino.runtime as ov
 from mmcv.utils import ProgressBar
+from nncf.common.quantization.structs import QuantizationPreset
+from openvino.model_api.adapters import OpenvinoAdapter, create_core
+from openvino.model_api.models import Model
 
 from otx.algorithms.action.adapters.openvino import (
     ActionOVClsDataLoader,
     get_ovdataloader,
     model_wrappers,
 )
 from otx.algorithms.action.configs.base import ActionConfig
+from otx.algorithms.common.utils.ir import check_if_quantized
 from otx.api.entities.annotation import AnnotationSceneEntity
 from otx.api.entities.datasets import DatasetEntity, DatasetItemEntity
 from otx.api.entities.inference_parameters import (
@@ -53,6 +54,7 @@
 )
 from otx.api.entities.optimization_parameters import OptimizationParameters
 from otx.api.entities.resultset import ResultSetEntity
+from otx.api.entities.subset import Subset
 from otx.api.entities.task_environment import TaskEnvironment
 from otx.api.serialization.label_mapper import LabelSchemaMapper, label_schema_to_bytes
 from otx.api.usecases.evaluation.metrics_helper import MetricsHelper
@@ -71,14 +73,6 @@
     OptimizationType,
 )
 
-try:
-    from openvino.model_zoo.model_api.adapters import OpenvinoAdapter, create_core
-    from openvino.model_zoo.model_api.models import Model
-except ImportError:
-    import warnings
-
-    warnings.warn("ModelAPI was not found.")
-
 logger = logging.getLogger(__name__)
 
 
@@ -115,7 +109,7 @@ def __init__(
             create_core(), model_file, weight_file, device=device, max_num_requests=num_requests
         )
         self.configuration: Dict[Any, Any] = {}
-        self.model = Model.create_model(self.task_type, model_adapter, self.configuration, preload=True)
+        self.model = Model.create_model(model_adapter, self.task_type, self.configuration, preload=True)
         self.converter: IPredictionToAnnotationConverter
         if self.task_type == "ACTION_CLASSIFICATION":
             self.converter = ClassificationToAnnotationConverter(self.label_schema)
@@ -147,20 +141,26 @@ def forward(self, image: Dict[str, np.ndarray]) -> Dict[str, np.ndarray]:
         return self.model.infer_sync(image)
 
 
-class DataLoaderWrapper(DataLoader):
+class DataLoaderWrapper:
     """DataLoader implementation for ActionOpenVINOTask."""
 
-    def __init__(self, dataloader: DataLoader, inferencer: BaseInferencer):
-        super().__init__(config=None)
+    def __init__(self, dataloader: Any, inferencer: BaseInferencer, shuffle: bool = True):
         self.dataloader = dataloader
         self.inferencer = inferencer
+        self.shuffler = None
+        if shuffle:
+            self.shuffler = list(range(len(dataloader)))
+            random.shuffle(self.shuffler)
 
     def __getitem__(self, index: int):
         """Get item from dataset."""
+        if self.shuffler is not None:
+            index = self.shuffler[index]
+
         item = self.dataloader[index]
         annotation = item[len(item) // 2].annotation_scene
-        inputs, metadata = self.inferencer.pre_process(item)
-        return (index, annotation), inputs, metadata
+        inputs, _ = self.inferencer.pre_process(item)
+        return inputs, annotation
 
     def __len__(self):
         """Get length of dataset."""
@@ -272,13 +272,15 @@ def optimize(
         """Optimize function of OpenVINOTask."""
 
         if optimization_type is not OptimizationType.POT:
-            raise ValueError("POT is the only supported optimization type for OpenVino models")
+            raise ValueError("PTQ is the only supported optimization type for OpenVino models")
 
         clip_len = self.inferencer.model.t
         width = self.inferencer.model.w
         height = self.inferencer.model.h
+        dataset = dataset.get_subset(Subset.TRAINING)
         data_loader = get_ovdataloader(dataset, self.task_type, clip_len, width, height)
         data_loader = DataLoaderWrapper(data_loader, self.inferencer)
+        quantization_dataset = nncf.Dataset(data_loader, lambda data: data[0])
 
         if self.model is None:
             raise RuntimeError("optimize failed, model is None")
@@ -291,47 +293,30 @@ def optimize(
             with open(bin_path, "wb") as f:
                 f.write(self.model.get_data("openvino.bin"))
 
-            model_config = ADDict({"model_name": "openvino_model", "model": xml_path, "weights": bin_path})
-
-            model = load_model(model_config)
-
-            if get_nodes_by_type(model, ["FakeQuantize"]):
-                raise RuntimeError("Model is already optimized by POT")
+            ov_model = ov.Core().read_model(xml_path)
+            if check_if_quantized(ov_model):
+                raise RuntimeError("Model is already optimized by PTQ")
 
         if optimization_parameters is not None:
             optimization_parameters.update_progress(10, None)
 
-        engine_config = ADDict({"device": "CPU"})
-
         stat_subset_size = self.hparams.pot_parameters.stat_subset_size
-        preset = self.hparams.pot_parameters.preset.name.lower()
+        preset = QuantizationPreset(self.hparams.pot_parameters.preset.name.lower())
 
-        algorithms = [
-            {
-                "name": "DefaultQuantization",
-                "params": {
-                    "target_device": "ANY",
-                    "preset": preset,
-                    "stat_subset_size": min(stat_subset_size, len(data_loader)),
-                    "shuffle_data": True,
-                },
-            }
-        ]
-
-        engine = IEEngine(config=engine_config, data_loader=data_loader, metric=None)
-
-        pipeline = create_pipeline(algorithms, engine)
-
-        compressed_model = pipeline.run(model)
-
-        compress_model_weights(compressed_model)
+        compressed_model = nncf.quantize(
+            ov_model,
+            quantization_dataset,
+            subset_size=min(stat_subset_size, len(data_loader)),
+            preset=preset,
+        )
 
         if optimization_parameters is not None:
             optimization_parameters.update_progress(90, None)
 
         with tempfile.TemporaryDirectory() as tempdir:
-            save_model(compressed_model, tempdir, model_name="model")
-            with open(os.path.join(tempdir, "model.xml"), "rb") as f:
+            xml_path = os.path.join(tempdir, "model.xml")
+            ov.serialize(compressed_model, xml_path)
+            with open(xml_path, "rb") as f:
                 output_model.set_data("openvino.xml", f.read())
             with open(os.path.join(tempdir, "model.bin"), "rb") as f:
                 output_model.set_data("openvino.bin", f.read())

src/otx/algorithms/anomaly/adapters/anomalib/exportable_code/base.py

@@ -7,8 +7,8 @@
 from typing import Union
 
 import numpy as np
-from openvino.model_zoo.model_api.models import SegmentationModel
-from openvino.model_zoo.model_api.models.types import DictValue, NumericalValue
+from openvino.model_api.models import SegmentationModel
+from openvino.model_api.models.types import DictValue, NumericalValue
 
 
 class AnomalyBase(SegmentationModel):

src/otx/algorithms/anomaly/configs/classification/padim/pot_optimization_config.json

@@ -0,0 +1,25 @@
+"""PTQ config file."""
+from nncf import IgnoredScope
+from nncf.common.quantization.structs import QuantizationPreset
+from nncf.quantization.advanced_parameters import AdvancedQuantizationParameters
+from nncf.quantization.range_estimator import (
+    AggregatorType,
+    RangeEstimatorParameters,
+    StatisticsCollectorParameters,
+    StatisticsType,
+)
+
+advanced_parameters = AdvancedQuantizationParameters(
+    activations_range_estimator_params=RangeEstimatorParameters(
+        min=StatisticsCollectorParameters(
+            statistics_type=StatisticsType.QUANTILE, aggregator_type=AggregatorType.MIN, quantile_outlier_prob=1e-4
+        ),
+        max=StatisticsCollectorParameters(
+            statistics_type=StatisticsType.QUANTILE, aggregator_type=AggregatorType.MAX, quantile_outlier_prob=1e-4
+        ),
+    )
+)
+
+preset = QuantizationPreset.MIXED
+
+ignored_scope = IgnoredScope(names=["/anomaly_map_generator/Mul", "/anomaly_map_generator/Sqrt"])