diff --git a/python/OpenVINO_EP/yolov7_object_detection_fp32_int8/README.md b/python/OpenVINO_EP/yolov7_object_detection_fp32_int8/README.md new file mode 100644 index 000000000..f75db255d --- /dev/null +++ b/python/OpenVINO_EP/yolov7_object_detection_fp32_int8/README.md @@ -0,0 +1,51 @@ +# Object detection with YOLOv7 in Python using OpenVINO™ Execution Provider +## Virtual Environement +Create a virtual environment either using a the `conda` virtual environment or `python venv`. + +### Python venv +``` +$ python3 -m venv yv7-pyenv +$ source yv7-pyenv/bin/activate +$ pip install --upgrade pip +``` +### Conda +If system python3 version is < 3.8.13 it's recommended to either upgrade to python3.8 or use create virtual environment python3.8 using conda. +``` +$ conda create -n yolov7 python=3.8 +$ conda activate yolov7 +``` +## Install required packages +After creating and activating virtual environment, install the required packages: + +### NNCF Experimental + +``` +$ git clone https://github.com/openvinotoolkit/nncf.git +$ cd nncf && python setup.py develop --onnx +$ pip install --no-cache-dir onnxconverter_common onnxruntime-openvino==1.11.0 +$ cd .. && rm -fR nncf +``` + +### YoloV7 requirements + +``` +$ cd notebooks +$ git clone https://github.com/WongKinYiu/yolov7 +$ cp -R yolov7/* . +$ pip install --no-cache-dir -r requirements.txt +$ rm -fR yolov7 +``` + +Launch the notebook from here. +## Dataset +The dataset used in this sample is the coco-validation2017 dataset. It will be downloaded automatically in the desired location while running the notebook. + +## Outputs + +- `yolov7-tiny.onnx` +- `yolov7-tiny-quantized.onnx` +- `cat-yolov7-detected.jpg` + +## References +### [ONNX Rruntime](https://onnxruntime.ai/docs/install/) +### [NNCF](https://github.com/openvinotoolkit/nncf/tree/develop) \ No newline at end of file diff --git a/python/OpenVINO_EP/yolov7_object_detection_fp32_int8/notebooks/OVEP_yoloV7-tiny_object_detection.ipynb b/python/OpenVINO_EP/yolov7_object_detection_fp32_int8/notebooks/OVEP_yoloV7-tiny_object_detection.ipynb new file mode 100644 index 000000000..e29b3c004 --- /dev/null +++ b/python/OpenVINO_EP/yolov7_object_detection_fp32_int8/notebooks/OVEP_yoloV7-tiny_object_detection.ipynb @@ -0,0 +1,753 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "b6157254", + "metadata": { + "id": "b6157254" + }, + "source": [ + "Copyright (C) 2021-2022, Intel Corporation\n", + "\n", + "SPDX-License-Identifier: Apache-2.0" + ] + }, + { + "cell_type": "markdown", + "id": "af482d97", + "metadata": { + "id": "b6157254" + }, + "source": [ + "Major Portions of this code are copyright of their respective authors and released under the General Public License Version 3.0:\n", + "- For licensing see https://github.com/WongKinYiu/yolov7/blob/main/LICENSE.md" + ] + }, + { + "cell_type": "markdown", + "id": "dfed77f6", + "metadata": { + "id": "dfed77f6" + }, + "source": [ + "# Object detection with YOLOv7 in Python using OpenVINO™ Execution Provider:\n", + "\n", + "1. The Object detection sample uses a YOLOv7 Deep Learning ONNX Model.\n", + "\n", + "\n", + "2. The sample involves presenting an image to ONNX Runtime (RT), which uses the OpenVINO™ Execution Provider to run inference on various Intel hardware devices as mentioned before and perform object detection to detect up to 80 different objects like person, bicycle, car, motorbike and much more from the coco dataset." + ] + }, + { + "cell_type": "markdown", + "id": "fbf1db77", + "metadata": {}, + "source": [ + "**Before starting with this notebook please make sure to perform the required installations as mentioned below:**\n", + "1. [YoloV7 installation requirements](https://github.com/WongKinYiu/yolov7#installation)\n", + "2. [NNCF Onnx (Experimental Requirements)](https://github.com/openvinotoolkit/nncf/tree/54a444fb6ef1806c7ab8e647f762e2547ceb95bf#installation)\n", + "\n", + "## Download COCO validation dataset" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1c97ae99", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "import sys\n", + "import warnings\n", + "from urllib import request\n", + "from zipfile import ZipFile\n", + "import torch\n", + "\n", + "warnings.filterwarnings(action=\"ignore\")\n", + "operating_platform = sys.platform\n", + "\n", + "# Download COCO val\n", + "if not os.path.exists(\"datasets\"):\n", + " torch.hub.download_url_to_file('https://ultralytics.com/assets/coco2017val.zip', 'tmp.zip')\n", + " if operating_platform == \"win32\":\n", + " with ZipFile(\"tmp.zip\", \"r\") as zipped_data:\n", + " zipped_data.extractall(path=\"datasets\")\n", + " zipped_data.close()\n", + " os.remove(\"tmp.zip\")\n", + " else:\n", + " !unzip -q tmp.zip -d datasets && rm tmp.zip\n" + ] + }, + { + "cell_type": "markdown", + "id": "660fc7b3", + "metadata": {}, + "source": [ + "## Export yolov7 model to ONNX format with NMS\n", + "**Note: Adjust the values of `iou-thres` and `conf-thres` according to requirements.**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "98b58e62", + "metadata": {}, + "outputs": [], + "source": [ + "if not os.path.exists(\"yolov7.onnx\"):\n", + " if operating_platform == \"win32\":\n", + " model_url = \"https://github.com/WongKinYiu/yolov7/releases/download/v0.1/yolov7.pt\"\n", + " request.urlretrieve(model_url, \"yolov7.pt\")\n", + " else:\n", + " !wget https://github.com/WongKinYiu/yolov7/releases/download/v0.1/yolov7.pt\n", + " \n", + " !python export.py --weights yolov7.pt --grid --end2end --simplify --topk-all 100 --iou-thres 0.65 --conf-thres 0.35 --img-size 640 640 --max-wh 640" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1d702110", + "metadata": {}, + "outputs": [], + "source": [ + "if not os.path.exists('cat.jpg'):\n", + " if operating_platform == \"win32\":\n", + " img_url = \"https://storage.openvinotoolkit.org/data/test_data/images/cat.jpg\"\n", + " request.urlretrieve(img_url, \"cat.jpg\")\n", + " else:\n", + " !wget https://storage.openvinotoolkit.org/data/test_data/images/cat.jpg" + ] + }, + { + "cell_type": "markdown", + "id": "a22f6b6a", + "metadata": {}, + "source": [ + "## Import Required Libraries" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5ec4fb7c", + "metadata": {}, + "outputs": [], + "source": [ + "# Inference for ONNX model\n", + "import cv2\n", + "import time\n", + "import requests\n", + "import random\n", + "import logging\n", + "import numpy as np\n", + "import onnxruntime\n", + "from PIL import Image\n", + "from pathlib import Path\n", + "from collections import OrderedDict,namedtuple\n", + "import matplotlib.pyplot as plt" + ] + }, + { + "cell_type": "markdown", + "id": "435fb7dd", + "metadata": {}, + "source": [ + "## Define necessary helper functions\n", + "### Pre-Processing\n", + "\n", + "When we are using a pre-trained model, which is trained & fine-tuned using a fixed image size as input, we should resize our image to a shape which is expected by the model. The image reshaped using a scaling factor which is a ratio between the desired height/width and the actual image height/width. \n", + "$$scale = min \\biggl( \\frac{\\text{target height}}{\\text{input image height}}, \\frac{\\text{target width}}{\\text{input image width}} \\biggl)$$ \n", + "Using the $scale$-factor, image height & width are calculated which is then re-shaped to the desired image size using the `opencv` package. Here this is acheived by the `preProcess_image` helper function." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "0bed58f8", + "metadata": {}, + "outputs": [], + "source": [ + "def letterbox(im, new_shape=(640, 640), color=(114, 114, 114), auto=True, scaleup=True, stride=32):\n", + " # Resize and pad image while meeting stride-multiple constraints\n", + " shape = im.shape[:2] # current shape [height, width]\n", + " if isinstance(new_shape, int):\n", + " new_shape = (new_shape, new_shape)\n", + "\n", + " # Scale ratio (new / old)\n", + " r = min(new_shape[0] / shape[0], new_shape[1] / shape[1])\n", + " if not scaleup: # only scale down, do not scale up (for better val mAP)\n", + " r = min(r, 1.0)\n", + "\n", + " # Compute padding\n", + " new_unpad = int(round(shape[1] * r)), int(round(shape[0] * r))\n", + " dw, dh = new_shape[1] - new_unpad[0], new_shape[0] - new_unpad[1] # wh padding\n", + "\n", + " if auto: # minimum rectangle\n", + " dw, dh = np.mod(dw, stride), np.mod(dh, stride) # wh padding\n", + "\n", + " dw /= 2 # divide padding into 2 sides\n", + " dh /= 2\n", + "\n", + " if shape[::-1] != new_unpad: # resize\n", + " im = cv2.resize(im, new_unpad, interpolation=cv2.INTER_LINEAR)\n", + " top, bottom = int(round(dh - 0.1)), int(round(dh + 0.1))\n", + " left, right = int(round(dw - 0.1)), int(round(dw + 0.1))\n", + " im = cv2.copyMakeBorder(im, top, bottom, left, right, cv2.BORDER_CONSTANT, value=color) # add border\n", + " return im, r, (dw, dh)" + ] + }, + { + "cell_type": "markdown", + "id": "d9c15307", + "metadata": {}, + "source": [ + "## Labels and Colors for Lables" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f9ed9518", + "metadata": {}, + "outputs": [], + "source": [ + "random.seed(42)\n", + "names = ['person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'traffic light', \n", + " 'fire hydrant', 'stop sign', 'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', \n", + " 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', \n", + " 'skis', 'snowboard', 'sports ball', 'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard', \n", + " 'tennis racket', 'bottle', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', \n", + " 'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza', 'donut', 'cake', 'chair', 'couch', \n", + " 'potted plant', 'bed', 'dining table', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cell phone', \n", + " 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddy bear', \n", + " 'hair drier', 'toothbrush']\n", + "colors = {name:[random.randint(0, 255) for _ in range(3)] for i,name in enumerate(names)}" + ] + }, + { + "cell_type": "markdown", + "id": "32da9963", + "metadata": {}, + "source": [ + "## Read & Pre-Process Image" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1d35b1a3", + "metadata": {}, + "outputs": [], + "source": [ + "def set_logging(rank=-1):\n", + " logging.basicConfig(\n", + " format=\"%(message)s\",\n", + " level=logging.INFO if rank in [-1, 0] else logging.WARN)\n", + " \n", + "set_logging(0) # run before defining LOGGER\n", + "LOGGER = logging.getLogger(\"yolov7\")\n", + "\n", + "def preProcess_image(img_path):\n", + " \n", + "\n", + " if img_path.split('.')[-1] not in ['jpg', 'png']:\n", + " raise Exception(img_path, \"is an unknown file format. Use the image of .jpg or .png format\")\n", + " \n", + " if not os.path.exists(img_path):\n", + " raise FileNotFoundError(img_path, \"File Not Found: Please enter a valid image file path!\")\n", + "\n", + " if os.path.islink(img_path):\n", + " raise Exception(img_path, \"is a symbolic link. Please provide a valid image file path!\")\n", + "\n", + " img = cv2.imread(img_path)\n", + " img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n", + "\n", + " image = img.copy()\n", + " image, ratio, dwdh = letterbox(image, auto=False)\n", + " image = image.transpose((2, 0, 1))\n", + " image = np.expand_dims(image, 0)\n", + " image = np.ascontiguousarray(image)\n", + "\n", + " im = image.astype(np.float32)\n", + " im /= 255\n", + " print(\"Image Shape:\", im.shape, sep='\\t')\n", + " return im, ratio, dwdh" + ] + }, + { + "cell_type": "markdown", + "id": "bf37989c", + "metadata": {}, + "source": [ + "## Create a session for inference based on the device selected\n", + "Inferencing using OpenVINO Execution provider under ONNX-Runtime, is performed using the below simple steps:\n", + "\n", + "1. Create a ONNX Runtime Inference Session instance using `onnxruntime.InferenceSession()`\n", + "2. To create an Inference Session object pass the model and the execution provider as arguments. Execution Providers are the hardware device options e.g. CPU, Myriad, GPU, etc. on which the session will be executed.\n", + "\n", + "The below create_sess function actually takes care of the above steps. All we need to do is pass the device arguement to it. It'll return the appropriate session according to the selected device along with the input name for the model." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d217bb8b", + "metadata": {}, + "outputs": [], + "source": [ + "def check_model_extension(fp):\n", + " # Split the extension from the path and normalise it to lowercase.\n", + " ext = os.path.splitext(fp)[-1].lower()\n", + "\n", + " # Now we can simply use != to check for inequality, no need for wildcards.\n", + " if(ext != \".onnx\"):\n", + " raise Exception(fp, \"is an unknown file format. Use the model ending with .onnx format\")\n", + " \n", + " if not os.path.exists(fp):\n", + " raise Exception(\"[ ERROR ] Path of the onnx model file is Invalid\")\n", + " \n", + " if os.path.islink(fp):\n", + " raise Exception(fp, \"is a symbolic link. Please provide a valid .onnx model path!\")\n", + "\n", + "def create_session(model_path, device='CPU_FP32'):\n", + " \n", + " check_model_extension(model_path)\n", + "\n", + " if device == 'CPU_FP32':\n", + " providers = ['OpenVINOExecutionProvider']\n", + " elif device == 'cpu':\n", + " providers = ['CPUExecutionProvider']\n", + " else:\n", + " print('Invalid or No provider passed, using default CPU Execution Provider.')\n", + " providers = ['CPUExecutionProvider']\n", + " \n", + " print(f'Use ORT providers: {providers}')\n", + " onnxruntime.set_default_logger_severity(4)\n", + "\n", + " sess = onnxruntime.InferenceSession(model_path,\n", + " providers=providers,\n", + " provider_options=[{'device_type': device}])\n", + "\n", + " outname = [i.name for i in sess.get_outputs()]\n", + " inname = [i.name for i in sess.get_inputs()]\n", + " \n", + " return sess, outname, inname\n" + ] + }, + { + "cell_type": "markdown", + "id": "976db139", + "metadata": {}, + "source": [ + "# Run Inference Function\n", + "\n", + "The `run-inference` function takes inputs as:\n", + "- `model_path` $\\rightarrow$ `.onnx`\n", + "- `img_input` $\\rightarrow$ `.jpg`\n", + "- `device` $\\rightarrow$ `device using which the inference should be run e.g. cpu, CPU_FP32` \n", + " Based on the device arguement the execution providers are selected.\n", + "\n", + "It performs the following tasks:\n", + "- Pre-process the input image as per model requirements\n", + "- Creates the appropriate onnxruntime session as per the device arguement\n", + "- Runs inferences along with non-max suppression on the predictions for the number of runs passed\n", + "- Adds bounding boxes on the detected objects\n", + "- Saves a copy of the inferred image." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "62c95da2", + "metadata": {}, + "outputs": [], + "source": [ + "def run_inference(model_path, img_input, device='CPU_FP32', num_runs=10, warm_up=3):\n", + " \n", + " img0, proces_ratio, dwdh = preProcess_image(img_input)\n", + " session, outputs, inputs = create_session(model_path, device)\n", + " \n", + " pred_times = []\n", + " inp = {inputs[0]:img0}\n", + "\n", + " \n", + " # Inference\n", + " for iter_num in range(num_runs + 2):\n", + "\n", + " # warmup session\n", + " if iter_num <= warm_up:\n", + " session.run(outputs, inp)[0]\n", + " continue\n", + "\n", + " start = time.time()\n", + " pred = session.run(outputs, inp)[0]\n", + " end = time.time()\n", + " inference_time = end - start\n", + " pred_times.append(inference_time)\n", + " \n", + "\n", + " ori_images = cv2.imread(img_input)\n", + " ori_images = [cv2.cvtColor(ori_images, cv2.COLOR_BGR2RGB)]\n", + " \n", + " # Add bounding box to detected samples\n", + " for i,(batch_id,x0,y0,x1,y1,cls_id,score) in enumerate(pred):\n", + " image = ori_images[int(batch_id)]\n", + " box = np.array([x0,y0,x1,y1])\n", + " box -= np.array(dwdh*2)\n", + " box /= proces_ratio\n", + " box = box.round().astype(np.int32).tolist()\n", + " cls_id = int(cls_id)\n", + " score = round(float(score),3)\n", + " name = names[cls_id]\n", + " color = colors[name]\n", + " name += ' '+str(score)\n", + " cv2.rectangle(image,box[:2],box[2:],color,2)\n", + " cv2.putText(image,name,(box[0], box[1] - 2),cv2.FONT_HERSHEY_SIMPLEX,0.45,[225, 255, 255],thickness=1)\n", + " cv2.putText(image, 'FPS: {:.8f}'.format(1.0 / inference_time),\n", + " (10, 40), cv2.FONT_HERSHEY_COMPLEX, 0.45, (255, 255, 255), 0) \n", + " print('Avg Inference time in ms: %f' %\n", + " (sum(pred_times) / len(pred_times) * 1000))\n", + " plt.imshow(Image.fromarray(ori_images[0]))\n", + " plt.axis('off')\n", + " f = f\"{img_input.split('.')[0]}_{model_path.split('.')[0]}_{device}.jpg\"\n", + " plt.imsave(f, Image.fromarray(ori_images[0]))" + ] + }, + { + "cell_type": "markdown", + "id": "572f353e", + "metadata": {}, + "source": [ + "### Inference using CPU Execution Provider (MLAS)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2d976d50", + "metadata": {}, + "outputs": [], + "source": [ + "model_path = 'yolov7.onnx'\n", + "img_path = 'cat.jpg'\n", + "num_iters = 100" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2a729964", + "metadata": {}, + "outputs": [], + "source": [ + "params = {\"model_path\":model_path,\n", + " \"img_input\": img_path,\n", + " \"device\":'cpu',\n", + " \"num_runs\": num_iters}\n", + "\n", + "run_inference(**params)" + ] + }, + { + "cell_type": "markdown", + "id": "4a1714ff", + "metadata": {}, + "source": [ + "### Inference using OpenVino Execution Provider" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "af79760a", + "metadata": {}, + "outputs": [], + "source": [ + "params = {\"model_path\": model_path,\n", + " \"img_input\": img_path,\n", + " \"device\":'CPU_FP32',\n", + " \"num_runs\": num_iters}\n", + "\n", + "run_inference(**params)" + ] + }, + { + "cell_type": "markdown", + "id": "8cfa14f7", + "metadata": {}, + "source": [ + "# NNCF PTQ for YoloV7" + ] + }, + { + "cell_type": "markdown", + "id": "4afe9d00", + "metadata": {}, + "source": [ + "## Quantize yolov7-tiny model" + ] + }, + { + "cell_type": "markdown", + "id": "624178f9", + "metadata": {}, + "source": [ + "### Build PTQ API dataset" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b149b53d", + "metadata": {}, + "outputs": [], + "source": [ + "import onnx\n", + "from nncf.experimental.post_training.compression_builder import CompressionBuilder\n", + "from nncf.experimental.post_training.algorithms.quantization import PostTrainingQuantization\n", + "from nncf.experimental.post_training.algorithms.quantization import PostTrainingQuantizationParameters\n", + "from nncf.common.utils.logger import logger as nncf_logger\n", + "from nncf.experimental.post_training.api import dataset as ptq_api_dataset\n", + "from nncf.experimental.onnx.tensor import ONNXNNCFTensor\n", + "from utils.datasets import LoadImagesAndLabels\n", + "\n", + "class YoloV7Dataset(ptq_api_dataset.Dataset):\n", + " def __init__(self, path, batch_size, shuffle):\n", + " super().__init__(batch_size, shuffle)\n", + " self.load_images = LoadImagesAndLabels(path)\n", + "\n", + " def __getitem__(self, item):\n", + " img, _, _, _ = self.load_images[item]\n", + " # Input should be in [0,1].\n", + " img = (1 / 255.) * img\n", + " return {\"images\": ONNXNNCFTensor(img.numpy())}\n", + "\n", + " def __len__(self):\n", + " return len(self.load_images)\n", + "\n", + "dataset = YoloV7Dataset(\"datasets/coco/images/val2017\", 1, True)" + ] + }, + { + "cell_type": "markdown", + "id": "4d8e8007", + "metadata": {}, + "source": [ + "### Run PTQ" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c7562f51", + "metadata": {}, + "outputs": [], + "source": [ + "original_model = onnx.load(\"yolov7-tiny.onnx\")\n", + "num_init_samples = 100\n", + "# We'll ignore detector head not to quantize them\n", + "ignored_scopes = [\n", + " # Head branch 1\n", + " \"Mul_217\",\n", + " \"Add_219\",\n", + " \"Mul_221\",\n", + " \"Mul_223\",\n", + " \"Mul_227\",\n", + " # Head branch 2\n", + " \"Mul_251\",\n", + " \"Add_253\",\n", + " \"Mul_255\",\n", + " \"Mul_257\",\n", + " \"Mul_261\",\n", + " # Head branch 3\n", + " \"Mul_285\",\n", + " \"Add_287\",\n", + " \"Mul_289\",\n", + " \"Mul_291\",\n", + " \"Mul_295\",\n", + "]\n", + "output_model_path = \"yolov7-tiny-quantized.onnx\"" + ] + }, + { + "cell_type": "markdown", + "id": "30d85dd4", + "metadata": {}, + "source": [ + "### Step 1: Create a pipeline of compression algorithms." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d75116cd", + "metadata": {}, + "outputs": [], + "source": [ + "builder = CompressionBuilder()" + ] + }, + { + "cell_type": "markdown", + "id": "3fbeb310", + "metadata": {}, + "source": [ + "### Step 2: Create the quantization algorithm and add to the builder." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5d6f0c82", + "metadata": {}, + "outputs": [], + "source": [ + "quantization_parameters = PostTrainingQuantizationParameters(\n", + " number_samples=num_init_samples,\n", + " ignored_scopes=ignored_scopes\n", + ")\n", + "quantization = PostTrainingQuantization(quantization_parameters)\n", + "builder.add_algorithm(quantization)" + ] + }, + { + "cell_type": "markdown", + "id": "a71d7715", + "metadata": {}, + "source": [ + "### Step 3: Execute the pipeline." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "96272e26", + "metadata": {}, + "outputs": [], + "source": [ + "nncf_logger.info(\"Post-Training Quantization has just started!\")\n", + "nncf_logger.setLevel(4)\n", + "quantized_model = builder.apply(original_model, dataset)" + ] + }, + { + "cell_type": "markdown", + "id": "90590d88", + "metadata": {}, + "source": [ + "### Step 4: Save the quantized model." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "faa8caea", + "metadata": {}, + "outputs": [], + "source": [ + "onnx.save(quantized_model, output_model_path)\n", + "nncf_logger.info(\n", + " \"The quantized model is saved on {}\".format(output_model_path))\n", + "\n", + "onnx.checker.check_model(output_model_path)" + ] + }, + { + "cell_type": "markdown", + "id": "8b602028", + "metadata": {}, + "source": [ + "## Run Inference on INT-8 model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4b4c78d0", + "metadata": {}, + "outputs": [], + "source": [ + "model_path = 'yolov7-tiny-quantized.onnx'\n", + "img_path = 'cat.jpg'\n", + "num_iters = 100" + ] + }, + { + "cell_type": "markdown", + "id": "6059c1c0", + "metadata": {}, + "source": [ + "### Inference using OpenVino Execution Provider" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "72e93755", + "metadata": {}, + "outputs": [], + "source": [ + "params = {\"model_path\": model_path,\n", + " \"img_input\": img_path,\n", + " \"device\":'CPU_FP32',\n", + " \"num_runs\": num_iters}\n", + "\n", + "run_inference(**params)" + ] + }, + { + "cell_type": "markdown", + "id": "45ea764a", + "metadata": {}, + "source": [ + "# Benchmarking" + ] + }, + { + "cell_type": "markdown", + "id": "38c93e68", + "metadata": {}, + "source": [ + "- Benchmarks can be performed using:\n", + " - openvino's [benchmark app](https://github.com/openvinotoolkit/openvino/tree/master/tools/benchmark_tool)\n", + "```\n", + "./benchmark_app -m /yolov7-tiny.onnx --shape [1,3,640,640]\n", + "```\n", + " \n", + "\n", + "\n", + "\n", + "" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.13" + }, + "vscode": { + "interpreter": { + "hash": "ea635b6725f857ac32fe5a69d2a976b6cb34422ba04243780c7fe011e9e618a5" + } + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}