notebooks/226-yolov7-optimization/226-yolov7-optimization.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convert and Optimize YOLOv7 with OpenVINO™\n",
    "\n",
    "The YOLOv7 algorithm is making big waves in the computer vision and machine learning communities.\n",
    "It is a real-time object detection algorithm that performs image recognition tasks by taking an image as input \n",
    "and then predicting bounding boxes and class probabilities for each object in the image.\n",
    "\n",
    "YOLO stands for “You Only Look Once”, it is a popular family of real-time object detection algorithms.\n",
    "The original YOLO object detector was first released in 2016. Since then, different versions and variants of YOLO have been proposed, each providing a significant increase in performance and efficiency.\n",
    "YOLOv7 is next stage of evolution of YOLO models family, which provides a greatly improved real-time object detection accuracy without increasing the inference costs.\n",
    "More details about its realization can be found in original model [paper](https://arxiv.org/abs/2207.02696) and [repository](https://github.com/WongKinYiu/yolov7)\n",
    "\n",
    "Real-time object detection is often used as a key component in computer vision systems.\n",
    "Applications that use real-time object detection models include video analytics, robotics, autonomous vehicles, multi-object tracking and object counting, medical image analysis, and many others.\n",
    "\n",
    "\n",
    "This tutorial demonstrates step-by-step instructions on how to run and optimize PyTorch Yolo V7 with OpenVINO.\n",
    "\n",
    "The tutorial consists of the following steps:\n",
    "- Prepare PyTorch model\n",
    "- Download and prepare dataset\n",
    "- Validate original model\n",
    "- Convert PyTorch model to ONNX\n",
    "- Convert ONNX model to OpenVINO IR\n",
    "- Validate converted model\n",
    "- Prepare and run optimization pipeline\n",
    "- Compare accuracy of the FP32 and quantized models.\n",
    "- Compare performance of the FP32 and quantized models."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Pytorch model\n",
    "\n",
    "Generally, PyTorch models represent an instance of the [torch.nn.Module](https://pytorch.org/docs/stable/generated/torch.nn.Module.html) class, initialized by a state dictionary with model weights.\n",
    "We will use the YOLOv7 tiny model pre-trained on a COCO dataset, which is available in this [repo](https://github.com/WongKinYiu/yolov7).\n",
    "Typical steps to obtain pre-trained model:\n",
    "1. Create instance of model class.\n",
    "2. Load checkpoint state dict, which contains pre-trained model weights.\n",
    "3. Turn model to evaluation for switching some operations to inference mode.\n",
    "\n",
    "In this case, the model creators provide a tool that enables converting the YOLOv7 model to ONNX, so we do not need to do these steps manually."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "from pathlib import Path\n",
    "sys.path.append(\"../utils\")\n",
    "from notebook_utils import download_file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clone YOLOv7 repo\n",
    "if not Path('yolov7').exists():\n",
    "    !git clone https://github.com/WongKinYiu/yolov7\n",
    "%cd yolov7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download pre-trained model weights\n",
    "MODEL_LINK = \"https://github.com/WongKinYiu/yolov7/releases/download/v0.1/yolov7-tiny.pt\"\n",
    "DATA_DIR = Path(\"data/\")\n",
    "MODEL_DIR = Path(\"model/\")\n",
    "MODEL_DIR.mkdir(exist_ok=True)\n",
    "DATA_DIR.mkdir(exist_ok=True)\n",
    "\n",
    "download_file(MODEL_LINK, directory=MODEL_DIR, show_progress=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check model inference\n",
    "\n",
    "`detect.py` script run pytorch model inference and save image as result,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -W ignore detect.py --weights model/yolov7-tiny.pt --conf 0.25 --img-size 640 --source inference/images/horses.jpg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from PIL import Image\n",
    "# visualize prediction result\n",
    "Image.open('runs/detect/exp/horses.jpg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Export to ONNX"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To export an ONNX format of the model, we will use `export.py` script. Let us check its arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python export.py --help"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The most important parameters:\n",
    "* `--weights` - path to model weigths checkpoint\n",
    "* `--img-size` - size of input image for onnx tracing\n",
    "\n",
    "When exporting the ONNX model from PyTorch, there is an opportunity to setup configurable parameters for including post-processing results in model:\n",
    "* `--end2end` - export full model to onnx including post-processing\n",
    "* `--grid` - export Detect layer as part of model\n",
    "* `--topk-all` - topk elements for all images\n",
    "* `--iou-thres` - intersection over union threshold for NMS\n",
    "* `--conf-thres` - minimal confidence threshold\n",
    "* `--max-wh` - max bounding box width and height for NMS\n",
    "\n",
    "Including whole post-processing to model can help to achieve more performant results, but in the same time it makes the model less flexible and does not guarantee full accuracy reproducibility.\n",
    "It is the reason why we will add only `--grid` parameter to preserve original pytorch model result format.\n",
    "If you want to understand how to work with an end2end ONNX model, you can check this [notebook](https://github.com/WongKinYiu/yolov7/blob/main/tools/YOLOv7onnx.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -W ignore export.py --weights model/yolov7-tiny.pt --grid"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert ONNX Model to OpenVINO Intermediate Representation (IR)\n",
    "While ONNX models are directly supported by OpenVINO runtime, it can be useful to convert them to IR format to take the advantage of OpenVINO optimization tools and features.\n",
    "The `mo.convert_model` python function in OpenVINO Model Optimizer can be used for converting the model.\n",
    "The function returns instance of OpenVINO Model class, which is ready to use in Python interface. However, it can also be serialized to OpenVINO IR format for future execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openvino.tools import mo\n",
    "from openvino.runtime import serialize\n",
    "\n",
    "model = mo.convert_model('model/yolov7-tiny.onnx')\n",
    "# serialize model for saving IR\n",
    "serialize(model, 'model/yolov7-tiny.xml')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Verify model inference\n",
    "\n",
    "To test model work, we create inference pipeline similar to `detect.py`. The pipeline consists of preprocessing step, inference of OpenVINO model, and results post-processing to get bounding boxes."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing\n",
    "\n",
    "Model input is a tensor with the `[1, 3, 640, 640]` shape in `N, C, H, W` format, where\n",
    "* `N` - number of images in batch (batch size)\n",
    "* `C` - image channels\n",
    "* `H` - image height\n",
    "* `W` - image width\n",
    "\n",
    "Model expects images in RGB channels format and normalized in [0, 1] range.\n",
    "To resize images to fit model size `letterbox` resize approach is used where the aspect ratio of width and height is preserved. It is defined in yolov7 repository.\n",
    "\n",
    "To keep specific shape, preprocessing automatically enables padding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "from PIL import Image\n",
    "from utils.datasets import letterbox\n",
    "from utils.plots import plot_one_box\n",
    "\n",
    "\n",
    "def preprocess_image(img0: np.ndarray):\n",
    "    \"\"\"\n",
    "    Preprocess image according to YOLOv7 input requirements. \n",
    "    Takes image in np.array format, resizes it to specific size using letterbox resize, converts color space from BGR (default in OpenCV) to RGB and changes data layout from HWC to CHW.\n",
    "    \n",
    "    Parameters:\n",
    "      img0 (np.ndarray): image for preprocessing\n",
    "    Returns:\n",
    "      img (np.ndarray): image after preprocessing\n",
    "      img0 (np.ndarray): original image\n",
    "    \"\"\"\n",
    "    # resize\n",
    "    img = letterbox(img0, auto=False)[0]\n",
    "    \n",
    "    # Convert\n",
    "    img = img.transpose(2, 0, 1)\n",
    "    img = np.ascontiguousarray(img)\n",
    "    return img, img0\n",
    "\n",
    "\n",
    "def prepare_input_tensor(image: np.ndarray):\n",
    "    \"\"\"\n",
    "    Converts preprocessed image to tensor format according to YOLOv7 input requirements. \n",
    "    Takes image in np.array format with unit8 data in [0, 255] range and converts it to torch.Tensor object with float data in [0, 1] range\n",
    "    \n",
    "    Parameters:\n",
    "      image (np.ndarray): image for conversion to tensor\n",
    "    Returns:\n",
    "      input_tensor (torch.Tensor): float tensor ready to use for YOLOv7 inference\n",
    "    \"\"\"\n",
    "    input_tensor = image.astype(np.float32)  # uint8 to fp16/32\n",
    "    input_tensor /= 255.0  # 0 - 255 to 0.0 - 1.0\n",
    "    \n",
    "    if input_tensor.ndim == 3:\n",
    "        input_tensor = np.expand_dims(input_tensor, 0)\n",
    "    return input_tensor\n",
    "\n",
    "\n",
    "# label names for visualization\n",
    "NAMES = ['person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'traffic light',\n",
    "         'fire hydrant', 'stop sign', 'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow',\n",
    "         'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee',\n",
    "         'skis', 'snowboard', 'sports ball', 'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard',\n",
    "         'tennis racket', 'bottle', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple',\n",
    "         'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza', 'donut', 'cake', 'chair', 'couch',\n",
    "         'potted plant', 'bed', 'dining table', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cell phone',\n",
    "         'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddy bear',\n",
    "         'hair drier', 'toothbrush']\n",
    "\n",
    "# colors for visualization\n",
    "COLORS = {name: [np.random.randint(0, 255) for _ in range(3)]\n",
    "          for i, name in enumerate(NAMES)}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Postprocessing\n",
    "\n",
    "Model output contains detection boxes candidates. It is a tensor with the `[1,25200,85]` shape in the `B, N, 85` format, where:\n",
    "\n",
    "- `B` - batch size\n",
    "- `N` - number of detection boxes\n",
    "\n",
    "Detection box has the [`x`, `y`, `h`, `w`, `box_score`, `class_no_1`, ..., `class_no_80`] format, where:\n",
    "\n",
    "- (`x`, `y`) - raw coordinates of box center\n",
    "- `h`, `w` - raw height and width of box\n",
    "- `box_score` - confidence of detection box\n",
    "- `class_no_1`, ..., `class_no_80` - probability distribution over the classes.\n",
    "\n",
    "For getting final prediction, we need to apply non maximum suppression algorithm and rescale boxes coordinates to original image size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import List, Tuple, Dict\n",
    "from utils.general import scale_coords, non_max_suppression\n",
    "from openvino.runtime import Model\n",
    "\n",
    "\n",
    "def detect(model: Model, image_path: Path, conf_thres: float = 0.25, iou_thres: float = 0.45, classes: List[int] = None, agnostic_nms: bool = False):\n",
    "    \"\"\"\n",
    "    OpenVINO YOLOv7 model inference function. Reads image, preprocess it, runs model inference and postprocess results using NMS.\n",
    "    Parameters:\n",
    "        model (Model): OpenVINO compiled model.\n",
    "        image_path (Path): input image path.\n",
    "        conf_thres (float, *optional*, 0.25): minimal accpeted confidence for object filtering\n",
    "        iou_thres (float, *optional*, 0.45): minimal overlap score for remloving objects duplicates in NMS\n",
    "        classes (List[int], *optional*, None): labels for prediction filtering, if not provided all predicted labels will be used\n",
    "        agnostic_nms (bool, *optiona*, False): apply class agnostinc NMS approach or not\n",
    "    Returns:\n",
    "       pred (List): list of detections with (n,6) shape, where n - number of detected boxes in format [x1, y1, x2, y2, score, label] \n",
    "       orig_img (np.ndarray): image before preprocessing, can be used for results visualization\n",
    "       inpjut_shape (Tuple[int]): shape of model input tensor, can be used for output rescaling\n",
    "    \"\"\"\n",
    "    output_blob = model.output(0)\n",
    "    img = np.array(Image.open(image_path))\n",
    "    preprocessed_img, orig_img = preprocess_image(img)\n",
    "    input_tensor = prepare_input_tensor(preprocessed_img)\n",
    "    predictions = torch.from_numpy(model(input_tensor)[output_blob])\n",
    "    pred = non_max_suppression(predictions, conf_thres, iou_thres, classes=classes, agnostic=agnostic_nms)\n",
    "    return pred, orig_img, input_tensor.shape\n",
    "\n",
    "\n",
    "def draw_boxes(predictions: np.ndarray, input_shape: Tuple[int], image: np.ndarray, names: List[str], colors: Dict[str, int]):\n",
    "    \"\"\"\n",
    "    Utility function for drawing predicted bounding boxes on image\n",
    "    Parameters:\n",
    "        predictions (np.ndarray): list of detections with (n,6) shape, where n - number of detected boxes in format [x1, y1, x2, y2, score, label]\n",
    "        image (np.ndarray): image for boxes visualization\n",
    "        names (List[str]): list of names for each class in dataset\n",
    "        colors (Dict[str, int]): mapping between class name and drawing color\n",
    "    Returns:\n",
    "        image (np.ndarray): box visualization result\n",
    "    \"\"\"\n",
    "    if not len(predictions):\n",
    "        return image\n",
    "    # Rescale boxes from input size to original image size\n",
    "    predictions[:, :4] = scale_coords(input_shape[2:], predictions[:, :4], image.shape).round()\n",
    "\n",
    "    # Write results\n",
    "    for *xyxy, conf, cls in reversed(predictions):\n",
    "        label = f'{names[int(cls)]} {conf:.2f}'\n",
    "        plot_one_box(xyxy, image, label=label, color=colors[names[int(cls)]], line_thickness=1)\n",
    "    return image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openvino.runtime import Core\n",
    "core = Core()\n",
    "# read converted model\n",
    "model = core.read_model('model/yolov7-tiny.xml')\n",
    "# load model on CPU device\n",
    "compiled_model = core.compile_model(model, 'CPU')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "boxes, image, input_shape = detect(compiled_model, 'inference/images/horses.jpg')\n",
    "image_with_boxes = draw_boxes(boxes[0], input_shape, image, NAMES, COLORS)\n",
    "# visualize results\n",
    "Image.fromarray(image_with_boxes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Verify model accuracy"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download dataset\n",
    "\n",
    "YOLOv7 tiny is pre-trained on the COCO dataset, so in order to evaluate the model accuracy, we need to download it. According to the instructions provided in the YOLOv7 repo, we also need to download annotations in the format used by the author of the model, for use with the original model evaluation scripts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from zipfile import ZipFile\n",
    "\n",
    "sys.path.append(\"../../utils\")\n",
    "from notebook_utils import download_file\n",
    "\n",
    "DATA_URL = \"http://images.cocodataset.org/zips/val2017.zip\"\n",
    "LABELS_URL = \"https://github.com/ultralytics/yolov5/releases/download/v1.0/coco2017labels-segments.zip\"\n",
    "\n",
    "OUT_DIR = Path('.')\n",
    "\n",
    "download_file(DATA_URL, directory=OUT_DIR, show_progress=True)\n",
    "download_file(LABELS_URL, directory=OUT_DIR, show_progress=True)\n",
    "\n",
    "if not (OUT_DIR / \"coco/labels\").exists():\n",
    "    with ZipFile('coco2017labels-segments.zip' , \"r\") as zip_ref:\n",
    "        zip_ref.extractall(OUT_DIR)\n",
    "    with ZipFile('val2017.zip' , \"r\") as zip_ref:\n",
    "        zip_ref.extractall(OUT_DIR / 'coco/images')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create dataloader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import namedtuple\n",
    "import yaml\n",
    "from utils.datasets import create_dataloader\n",
    "from utils.general import check_dataset, box_iou, xywh2xyxy, colorstr\n",
    "\n",
    "# read dataset config\n",
    "DATA_CONFIG = 'data/coco.yaml'\n",
    "with open(DATA_CONFIG) as f:\n",
    "    data = yaml.load(f, Loader=yaml.SafeLoader)\n",
    "\n",
    "# Dataloader\n",
    "TASK = 'val'  # path to train/val/test images\n",
    "Option = namedtuple('Options', ['single_cls'])  # imitation of commandline provided options for single class evaluation\n",
    "opt = Option(False)\n",
    "dataloader = create_dataloader(\n",
    "    data[TASK], 640, 1, 32, opt, pad=0.5,\n",
    "    prefix=colorstr(f'{TASK}: ')\n",
    ")[0]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define validation function\n",
    "\n",
    "We will reuse validation metrics provided in the YOLOv7 repo with a modification for this case (removing extra steps). The original model evaluation procedure can be found in this [file](https://github.com/WongKinYiu/yolov7/blob/main/test.py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "test_replace": {
     "int = None": "int = 100"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from tqdm.notebook import tqdm\n",
    "from utils.metrics import ap_per_class\n",
    "\n",
    "\n",
    "def test(data,\n",
    "         model: Model,\n",
    "         dataloader: torch.utils.data.DataLoader,\n",
    "         conf_thres: float = 0.001,\n",
    "         iou_thres: float = 0.65,  # for NMS\n",
    "         single_cls: bool = False,\n",
    "         v5_metric: bool = False,\n",
    "         names: List[str] = None,\n",
    "         num_samples: int = None\n",
    "        ):\n",
    "    \"\"\"\n",
    "    YOLOv7 accuracy evaluation. Processes validation dataset and compites metrics.\n",
    "    \n",
    "    Parameters:\n",
    "        model (Model): OpenVINO compiled model.\n",
    "        dataloader (torch.utils.DataLoader): validation dataset.\n",
    "        conf_thres (float, *optional*, 0.001): minimal confidence threshold for keeping detections\n",
    "        iou_thres (float, *optional*, 0.65): IOU threshold for NMS\n",
    "        single_cls (bool, *optional*, False): class agnostic evaluation\n",
    "        v5_metric (bool, *optional*, False): use YOLOv5 evaluation approach for metrics calculation\n",
    "        names (List[str], *optional*, None): names for each class in dataset\n",
    "        num_samples (int, *optional*, None): number samples for testing\n",
    "    Returns:\n",
    "        mp (float): mean precision\n",
    "        mr (float): mean recall\n",
    "        map50 (float): mean average precision at 0.5 IOU threshold\n",
    "        map (float): mean average precision at 0.5:0.95 IOU thresholds\n",
    "        maps (Dict(int, float): average precision per class\n",
    "        seen (int): number of evaluated images\n",
    "        labels (int): number of labels\n",
    "    \"\"\"\n",
    "\n",
    "    model_output = model.output(0)\n",
    "    check_dataset(data)  # check\n",
    "    nc = 1 if single_cls else int(data['nc'])  # number of classes\n",
    "    iouv = torch.linspace(0.5, 0.95, 10)  # iou vector for mAP@0.5:0.95\n",
    "    niou = iouv.numel()\n",
    "\n",
    "    if v5_metric:\n",
    "        print(\"Testing with YOLOv5 AP metric...\")\n",
    "    \n",
    "    seen = 0\n",
    "    p, r, mp, mr, map50, map = 0., 0., 0., 0., 0., 0.\n",
    "    stats, ap, ap_class = [], [], []\n",
    "    for sample_id, (img, targets, _, shapes) in enumerate(tqdm(dataloader)):\n",
    "        if num_samples is not None and sample_id == num_samples:\n",
    "            break\n",
    "        img = prepare_input_tensor(img.numpy())\n",
    "        targets = targets\n",
    "        height, width = img.shape[2:]\n",
    "\n",
    "        with torch.no_grad():\n",
    "            # Run model\n",
    "            out = torch.from_numpy(model(img)[model_output])  # inference output            \n",
    "            # Run NMS\n",
    "            targets[:, 2:] *= torch.Tensor([width, height, width, height])  # to pixels\n",
    "\n",
    "            out = non_max_suppression(out, conf_thres=conf_thres, iou_thres=iou_thres, labels=None, multi_label=True)\n",
    "        # Statistics per image\n",
    "        for si, pred in enumerate(out):\n",
    "            labels = targets[targets[:, 0] == si, 1:]\n",
    "            nl = len(labels)\n",
    "            tcls = labels[:, 0].tolist() if nl else []  # target class\n",
    "            seen += 1\n",
    "\n",
    "            if len(pred) == 0:\n",
    "                if nl:\n",
    "                    stats.append((torch.zeros(0, niou, dtype=torch.bool), torch.Tensor(), torch.Tensor(), tcls))\n",
    "                continue\n",
    "            # Predictions\n",
    "            predn = pred.clone()\n",
    "            scale_coords(img[si].shape[1:], predn[:, :4], shapes[si][0], shapes[si][1])  # native-space pred\n",
    "            # Assign all predictions as incorrect\n",
    "            correct = torch.zeros(pred.shape[0], niou, dtype=torch.bool, device='cpu')\n",
    "            if nl:\n",
    "                detected = []  # target indices\n",
    "                tcls_tensor = labels[:, 0]\n",
    "                # target boxes\n",
    "                tbox = xywh2xyxy(labels[:, 1:5])\n",
    "                scale_coords(img[si].shape[1:], tbox, shapes[si][0], shapes[si][1])  # native-space labels\n",
    "                # Per target class\n",
    "                for cls in torch.unique(tcls_tensor):\n",
    "                    ti = (cls == tcls_tensor).nonzero(as_tuple=False).view(-1)  # prediction indices\n",
    "                    pi = (cls == pred[:, 5]).nonzero(as_tuple=False).view(-1)  # target indices\n",
    "                    # Search for detections\n",
    "                    if pi.shape[0]:\n",
    "                        # Prediction to target ious\n",
    "                        ious, i = box_iou(predn[pi, :4], tbox[ti]).max(1)  # best ious, indices\n",
    "                        # Append detections\n",
    "                        detected_set = set()\n",
    "                        for j in (ious > iouv[0]).nonzero(as_tuple=False):\n",
    "                            d = ti[i[j]]  # detected target\n",
    "                            if d.item() not in detected_set:\n",
    "                                detected_set.add(d.item())\n",
    "                                detected.append(d)\n",
    "                                correct[pi[j]] = ious[j] > iouv  # iou_thres is 1xn\n",
    "                                if len(detected) == nl:  # all targets already located in image\n",
    "                                    break\n",
    "            # Append statistics (correct, conf, pcls, tcls)\n",
    "            stats.append((correct.cpu(), pred[:, 4].cpu(), pred[:, 5].cpu(), tcls))\n",
    "    # Compute statistics\n",
    "    stats = [np.concatenate(x, 0) for x in zip(*stats)]  # to numpy\n",
    "    if len(stats) and stats[0].any():\n",
    "        p, r, ap, f1, ap_class = ap_per_class(*stats, plot=True, v5_metric=v5_metric, names=names)\n",
    "        ap50, ap = ap[:, 0], ap.mean(1)  # AP@0.5, AP@0.5:0.95\n",
    "        mp, mr, map50, map = p.mean(), r.mean(), ap50.mean(), ap.mean()\n",
    "        nt = np.bincount(stats[3].astype(np.int64), minlength=nc)  # number of targets per class\n",
    "    else:\n",
    "        nt = torch.zeros(1)\n",
    "    maps = np.zeros(nc) + map\n",
    "    for i, c in enumerate(ap_class):\n",
    "        maps[c] = ap[i]\n",
    "    return mp, mr, map50, map, maps, seen, nt.sum()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Validation function reports following list of accuracy metrics:\n",
    "\n",
    "* `Precision` is the degree of exactness of the model in identifying only relevant objects.\n",
    "* `Recall` measures the ability of the model to detect all ground truths objects.\n",
    "* `mAP@t` - mean average precision, represented as area under the Precision-Recall curve aggregated over all classes in the dataset, where `t` is Intersection Over Union (IOU) threshold, degree of overlapping between ground truth and predicted objects. Therefore, `mAP@.5` indicates that mean average precision calculated at 0.5 IOU threshold, `mAP@.5:.95` - calculated on range IOU thresholds from 0.5 to 0.95 with step 0.05."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mp, mr, map50, map, maps, num_images, labels = test(data=data, model=compiled_model, dataloader=dataloader, names=NAMES)\n",
    "# Print results\n",
    "s = ('%20s' + '%12s' * 6) % ('Class', 'Images', 'Labels', 'Precision', 'Recall', 'mAP@.5', 'mAP@.5:.95')\n",
    "print(s)\n",
    "pf = '%20s' + '%12i' * 2 + '%12.3g' * 4  # print format\n",
    "print(pf % ('all', num_images, labels, mp, mr, map50, map))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimize model using NNCF Post-training Quantization API\n",
    "\n",
    "[NNCF](https://github.com/openvinotoolkit/nncf) provides a suite of advanced algorithms for Neural Networks inference optimization in OpenVINO with minimal accuracy drop.\n",
    "We will use 8-bit quantization in post-training mode (without the fine-tuning pipeline) to optimize YOLOv7.\n",
    "\n",
    "> **Note**: NNCF Post-training Quantization is available as a preview feature in OpenVINO 2022.3 release. Fully functional support will be provided in the next releases.\n",
    "\n",
    "The optimization process contains the following steps:\n",
    "\n",
    "1. Create a Dataset for quantization.\n",
    "2. Run `nncf.quantize` for getting an optimized model.\n",
    "3. Serialize an OpenVINO IR model, using the `openvino.runtime.serialize` function."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reuse validation dataloader in accuracy testing for quantization.\n",
    "For that, it should be wrapped into the `nncf.Dataset` object and define transformation function for getting only input tensors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nncf  # noqa: F811\n",
    "\n",
    "\n",
    "def transform_fn(data_item):\n",
    "    \"\"\"\n",
    "    Quantization transform function. Extracts and preprocess input data from dataloader item for quantization.\n",
    "    Parameters:\n",
    "       data_item: Tuple with data item produced by DataLoader during iteration\n",
    "    Returns:\n",
    "        input_tensor: Input data for quantization\n",
    "    \"\"\"\n",
    "    img = data_item[0].numpy()\n",
    "    input_tensor = prepare_input_tensor(img) \n",
    "    return input_tensor\n",
    "\n",
    "\n",
    "quantization_dataset = nncf.Dataset(dataloader, transform_fn)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `nncf.quantize` function provides interface for model quantization. It requires instance of OpenVINO Model and quantization dataset. \n",
    "Optionally, some additional parameters for configuration quantization process (number of samples for quantization, preset, ignored scope etc.) can be provided.\n",
    "YOLOv7 model contains non-ReLU activation functions, which require asymmetric quantization of activations. To achieve better result, we will use `mixed` quantization preset.\n",
    "It provides symmetric quantization of weights and asymmetric quantization of activations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quantized_model = nncf.quantize(model, quantization_dataset, preset=nncf.QuantizationPreset.MIXED)\n",
    "\n",
    "serialize(quantized_model, 'model/yolov7-tiny_int8.xml')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validate Quantized model inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "int8_compiled_model = core.compile_model(quantized_model)\n",
    "boxes, image, input_shape = detect(int8_compiled_model, 'inference/images/horses.jpg')\n",
    "image_with_boxes = draw_boxes(boxes[0], input_shape, image, NAMES, COLORS)\n",
    "Image.fromarray(image_with_boxes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validate quantized model accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "int8_result = test(data=data, model=int8_compiled_model, dataloader=dataloader, names=NAMES)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mp, mr, map50, map, maps, num_images, labels = int8_result\n",
    "# Print results\n",
    "s = ('%20s' + '%12s' * 6) % ('Class', 'Images', 'Labels', 'Precision', 'Recall', 'mAP@.5', 'mAP@.5:.95')\n",
    "print(s)\n",
    "pf = '%20s' + '%12i' * 2 + '%12.3g' * 4  # print format\n",
    "print(pf % ('all', num_images, labels, mp, mr, map50, map))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, model accuracy slightly changed after quantization. However, if we look at the output image, these changes are not significant."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compare Performance of the Original and Quantized Models\n",
    "Finally, use the OpenVINO [Benchmark Tool](https://docs.openvino.ai/latest/openvino_inference_engine_tools_benchmark_tool_README.html) to measure the inference performance of the `FP32` and `INT8` models.\n",
    "\n",
    "> **NOTE**: For more accurate performance, it is recommended to run `benchmark_app` in a terminal/command prompt after closing other applications. Run `benchmark_app -m model.xml -d CPU` to benchmark async inference on CPU for one minute. Change `CPU` to `GPU` to benchmark on GPU. Run `benchmark_app --help` to see an overview of all command-line options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inference FP32 model (OpenVINO IR)\n",
    "!benchmark_app -m model/yolov7-tiny.xml -d CPU -api async"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inference INT8 model (OpenVINO IR)\n",
    "!benchmark_app -m model/yolov7-tiny_int8.xml -d CPU -api async"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8 | packaged by conda-forge | (main, Nov 24 2022, 14:07:00) [MSC v.1916 64 bit (AMD64)]"
  },
  "vscode": {
   "interpreter": {
    "hash": "30f6166f5f0cb6253cad15b1c8ca46093b160f1914c051aeccf8063f98b299b9"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}