|
| 1 | +{ |
| 2 | + "cells": [ |
| 3 | + { |
| 4 | + "cell_type": "markdown", |
| 5 | + "metadata": {}, |
| 6 | + "source": [ |
| 7 | + "# INT8 Quantization with Post-training Optimization Tool (POT) in Simplified Mode tutorial\n", |
| 8 | + "\n", |
| 9 | + "This tutorial shows how to quantize a [ResNet20](https://github.com/chenyaofo/pytorch-cifar-models) image classification model, trained on [CIFAR10 ](http://pytorch.org/vision/main/generated/torchvision.datasets.CIFAR10.html) dataset, using the Simplified Mode of OpenVINO Post-Training Optimization Tool (POT).\n", |
| 10 | + "\n", |
| 11 | + "Simplified Mode is designed to make the data preparation step easier, before model optimization. The mode is represented by an implementation of the engine interface in the POT API. It enables reading data from an arbitrary folder specified by the user. Currently, Simplified Mode is available only for image data in PNG or JPEG formats, stored in a single folder.\n", |
| 12 | + "\n", |
| 13 | + "**Note:** This mode cannot be used with the accuracy-aware method. It is not possible to control accuracy after optimization using this mode. However, Simplified Mode can be useful for estimating performance improvements when optimizing models.\n", |
| 14 | + "\n", |
| 15 | + "This tutorial includes the following steps:\n", |
| 16 | + "\n", |
| 17 | + "- Downloading and saving the CIFAR10 dataset\n", |
| 18 | + "- Preparing the model for quantization\n", |
| 19 | + "- Compressing the prepared model\n", |
| 20 | + "- Measuring and comparing the performance of the original and quantized models\n", |
| 21 | + "- Demonstrating the use of the quantized model for image classification\n" |
| 22 | + ] |
| 23 | + }, |
| 24 | + { |
| 25 | + "cell_type": "code", |
| 26 | + "execution_count": null, |
| 27 | + "metadata": {}, |
| 28 | + "outputs": [], |
| 29 | + "source": [ |
| 30 | + "import os\n", |
| 31 | + "from pathlib import Path\n", |
| 32 | + "import warnings\n", |
| 33 | + "\n", |
| 34 | + "import torch\n", |
| 35 | + "from torchvision import transforms as T\n", |
| 36 | + "from torchvision.datasets import CIFAR10\n", |
| 37 | + "\n", |
| 38 | + "import matplotlib.pyplot as plt\n", |
| 39 | + "import numpy as np\n", |
| 40 | + "\n", |
| 41 | + "from openvino.runtime import Core, Tensor\n", |
| 42 | + "\n", |
| 43 | + "warnings.filterwarnings(\"ignore\")\n", |
| 44 | + "\n", |
| 45 | + "# Set the data and model directories\n", |
| 46 | + "MODEL_DIR = 'model'\n", |
| 47 | + "CALIB_DIR = 'calib'\n", |
| 48 | + "CIFAR_DIR = 'cifar'\n", |
| 49 | + "CALIB_SET_SIZE = 300\n", |
| 50 | + "MODEL_NAME = 'resnet20'\n", |
| 51 | + "\n", |
| 52 | + "os.makedirs(MODEL_DIR, exist_ok=True)\n", |
| 53 | + "os.makedirs(CALIB_DIR, exist_ok=True)" |
| 54 | + ] |
| 55 | + }, |
| 56 | + { |
| 57 | + "cell_type": "markdown", |
| 58 | + "metadata": {}, |
| 59 | + "source": [ |
| 60 | + "## Prepare the calibration dataset\n", |
| 61 | + "The following steps are required to prepare the calibration dataset:\n", |
| 62 | + "- Download CIFAR10 dataset from Torchvision.datasets repository\n", |
| 63 | + "- Save the selected number of elements from this dataset as .png images in a separate folder" |
| 64 | + ] |
| 65 | + }, |
| 66 | + { |
| 67 | + "cell_type": "code", |
| 68 | + "execution_count": null, |
| 69 | + "metadata": {}, |
| 70 | + "outputs": [], |
| 71 | + "source": [ |
| 72 | + "transform = T.Compose([T.ToTensor()])\n", |
| 73 | + "dataset = CIFAR10(root=CIFAR_DIR, train=False, transform=transform, download=True)" |
| 74 | + ] |
| 75 | + }, |
| 76 | + { |
| 77 | + "cell_type": "code", |
| 78 | + "execution_count": null, |
| 79 | + "metadata": {}, |
| 80 | + "outputs": [], |
| 81 | + "source": [ |
| 82 | + "pil_converter = T.ToPILImage(mode=\"RGB\")\n", |
| 83 | + "\n", |
| 84 | + "for idx, info in enumerate(dataset):\n", |
| 85 | + " im = info[0]\n", |
| 86 | + " if idx >= CALIB_SET_SIZE:\n", |
| 87 | + " break\n", |
| 88 | + " label = info[1]\n", |
| 89 | + " pil_converter(im.squeeze(0)).save(Path(CALIB_DIR) / f'{label}_{idx}.png')" |
| 90 | + ] |
| 91 | + }, |
| 92 | + { |
| 93 | + "cell_type": "markdown", |
| 94 | + "metadata": {}, |
| 95 | + "source": [ |
| 96 | + "## Prepare the Model\n", |
| 97 | + "Model preparation includes the following steps:,\n", |
| 98 | + "- Download PyTorch model from Torchvision repository,\n", |
| 99 | + "- Convert the model to ONNX format,\n", |
| 100 | + "- Run OpenVINO Model Optimizer tool to convert ONNX to OpenVINO Intermediate Representation (IR)" |
| 101 | + ] |
| 102 | + }, |
| 103 | + { |
| 104 | + "cell_type": "code", |
| 105 | + "execution_count": null, |
| 106 | + "metadata": {}, |
| 107 | + "outputs": [], |
| 108 | + "source": [ |
| 109 | + "model = torch.hub.load(\"chenyaofo/pytorch-cifar-models\", \"cifar10_resnet20\", pretrained=True)\n", |
| 110 | + "dummy_input = torch.randn(1, 3, 32, 32)\n", |
| 111 | + "\n", |
| 112 | + "onnx_model_path = Path(MODEL_DIR) / '{}.onnx'.format(MODEL_NAME)\n", |
| 113 | + "ir_model_xml = onnx_model_path.with_suffix('.xml')\n", |
| 114 | + "ir_model_bin = onnx_model_path.with_suffix('.bin')\n", |
| 115 | + "\n", |
| 116 | + "torch.onnx.export(model, dummy_input, onnx_model_path)" |
| 117 | + ] |
| 118 | + }, |
| 119 | + { |
| 120 | + "cell_type": "markdown", |
| 121 | + "metadata": {}, |
| 122 | + "source": [ |
| 123 | + "Now we convert this model into the OpenVINO IR using the Model Optimizer:" |
| 124 | + ] |
| 125 | + }, |
| 126 | + { |
| 127 | + "cell_type": "code", |
| 128 | + "execution_count": null, |
| 129 | + "metadata": {}, |
| 130 | + "outputs": [], |
| 131 | + "source": [ |
| 132 | + "!mo --framework=onnx --data_type=FP32 --input_shape=[1,3,32,32] -m $onnx_model_path --output_dir $MODEL_DIR" |
| 133 | + ] |
| 134 | + }, |
| 135 | + { |
| 136 | + "cell_type": "markdown", |
| 137 | + "metadata": {}, |
| 138 | + "source": [ |
| 139 | + "## Compression stage\n", |
| 140 | + "Compress the model with the following command:\n", |
| 141 | + " \n", |
| 142 | + "`pot -q default -m <path_to_xml> -w <path_to_bin> --engine simplified --data-source <path_to_data>`" |
| 143 | + ] |
| 144 | + }, |
| 145 | + { |
| 146 | + "cell_type": "code", |
| 147 | + "execution_count": null, |
| 148 | + "metadata": {}, |
| 149 | + "outputs": [], |
| 150 | + "source": [ |
| 151 | + "!pot -q default -m $ir_model_xml -w $ir_model_bin --engine simplified --data-source $CALIB_DIR --output-dir compressed --direct-dump --name $MODEL_NAME" |
| 152 | + ] |
| 153 | + }, |
| 154 | + { |
| 155 | + "cell_type": "markdown", |
| 156 | + "metadata": {}, |
| 157 | + "source": [ |
| 158 | + "## Compare Performance of the Original and Quantized Models\n", |
| 159 | + "\n", |
| 160 | + "Finally, we will measure the inference performance of the FP32 and INT8 models. To do this, we use [Benchmark Tool](https://docs.openvino.ai/latest/openvino_inference_engine_tools_benchmark_tool_README.html) - an inference performance measurement tool for OpenVINO.\n", |
| 161 | + "\n", |
| 162 | + "**NOTE:** For more accurate performance, we recommended running benchmark_app in a terminal/command prompt after closing other applications. Run benchmark_app -m model.xml -d CPU to benchmark async inference on CPU for one minute. Change CPU to GPU to benchmark on GPU. Run benchmark_app --help to see an overview of all command line options." |
| 163 | + ] |
| 164 | + }, |
| 165 | + { |
| 166 | + "cell_type": "code", |
| 167 | + "execution_count": null, |
| 168 | + "metadata": {}, |
| 169 | + "outputs": [], |
| 170 | + "source": [ |
| 171 | + "optimized_model_path = Path('compressed/optimized')\n", |
| 172 | + "optimized_model_xml = optimized_model_path / '{}.xml'.format(MODEL_NAME)\n", |
| 173 | + "optimized_model_bin = optimized_model_path / '{}.bin'.format(MODEL_NAME)" |
| 174 | + ] |
| 175 | + }, |
| 176 | + { |
| 177 | + "cell_type": "code", |
| 178 | + "execution_count": null, |
| 179 | + "metadata": {}, |
| 180 | + "outputs": [], |
| 181 | + "source": [ |
| 182 | + "# Inference FP32 model (IR)\n", |
| 183 | + "!benchmark_app -m $ir_model_xml -d CPU -api async" |
| 184 | + ] |
| 185 | + }, |
| 186 | + { |
| 187 | + "cell_type": "code", |
| 188 | + "execution_count": null, |
| 189 | + "metadata": {}, |
| 190 | + "outputs": [], |
| 191 | + "source": [ |
| 192 | + "# Inference INT8 model (IR)\n", |
| 193 | + "!benchmark_app -m $optimized_model_xml -d CPU -api async" |
| 194 | + ] |
| 195 | + }, |
| 196 | + { |
| 197 | + "cell_type": "markdown", |
| 198 | + "metadata": {}, |
| 199 | + "source": [ |
| 200 | + "## Demonstration of the results\n", |
| 201 | + "\n", |
| 202 | + "This section demonstrates how to use the compressed model by running the optimized model on a subset of images from the CIFAR10 dataset and shows predictions using the model.\n", |
| 203 | + "\n", |
| 204 | + "The first step is to load the model:" |
| 205 | + ] |
| 206 | + }, |
| 207 | + { |
| 208 | + "cell_type": "code", |
| 209 | + "execution_count": null, |
| 210 | + "metadata": {}, |
| 211 | + "outputs": [], |
| 212 | + "source": [ |
| 213 | + "ie = Core()\n", |
| 214 | + "\n", |
| 215 | + "compiled_model = ie.compile_model(str(optimized_model_xml), \"AUTO\")" |
| 216 | + ] |
| 217 | + }, |
| 218 | + { |
| 219 | + "cell_type": "code", |
| 220 | + "execution_count": null, |
| 221 | + "metadata": {}, |
| 222 | + "outputs": [], |
| 223 | + "source": [ |
| 224 | + "# define all possible labels from CIFAR10\n", |
| 225 | + "labels_names = [\"airplane\", \"automobile\", \"bird\", \"cat\", \"deer\", \"dog\", \"frog\", \"horse\", \"ship\", \"truck\"]\n", |
| 226 | + "all_images = []\n", |
| 227 | + "all_labels = []\n", |
| 228 | + "\n", |
| 229 | + "# get all images and their labels \n", |
| 230 | + "for batch in dataset:\n", |
| 231 | + " all_images.append(torch.unsqueeze(batch[0], 0))\n", |
| 232 | + " all_labels.append(batch[1])" |
| 233 | + ] |
| 234 | + }, |
| 235 | + { |
| 236 | + "cell_type": "markdown", |
| 237 | + "metadata": {}, |
| 238 | + "source": [ |
| 239 | + "This section defines the function that shows the images and their labels using the indexes and two lists created in the previous step:" |
| 240 | + ] |
| 241 | + }, |
| 242 | + { |
| 243 | + "cell_type": "code", |
| 244 | + "execution_count": null, |
| 245 | + "metadata": {}, |
| 246 | + "outputs": [], |
| 247 | + "source": [ |
| 248 | + "def plot_pictures(indexes: list, images=all_images, labels=all_labels):\n", |
| 249 | + " \"\"\"Plot images with the specified indexes.\n", |
| 250 | + " :param indexes: a list of indexes of images to be displayed.\n", |
| 251 | + " :param images: a list of images from the dataset.\n", |
| 252 | + " :param labels: a list of labels for each image.\n", |
| 253 | + " \"\"\"\n", |
| 254 | + " num_pics = len(indexes)\n", |
| 255 | + " _, axarr = plt.subplots(1, num_pics)\n", |
| 256 | + " for idx, im_idx in enumerate(indexes):\n", |
| 257 | + " assert idx < 10000, 'Cannot get such index, there are only 10000'\n", |
| 258 | + " pic = np.rollaxis(images[im_idx].squeeze().numpy(), 0, 3)\n", |
| 259 | + " axarr[idx].imshow(pic)\n", |
| 260 | + " axarr[idx].set_title(labels_names[labels[im_idx]])" |
| 261 | + ] |
| 262 | + }, |
| 263 | + { |
| 264 | + "cell_type": "markdown", |
| 265 | + "metadata": {}, |
| 266 | + "source": [ |
| 267 | + "In this section we define a function that uses the optimized model to obtain predictions for the selected images:" |
| 268 | + ] |
| 269 | + }, |
| 270 | + { |
| 271 | + "cell_type": "code", |
| 272 | + "execution_count": null, |
| 273 | + "metadata": {}, |
| 274 | + "outputs": [], |
| 275 | + "source": [ |
| 276 | + "def infer_on_images(net, indexes: list, images=all_images):\n", |
| 277 | + " \"\"\" Inference model on a set of images.\n", |
| 278 | + " :param net: model on which do inference\n", |
| 279 | + " :param indexes: a list of indexes of images to infer on.\n", |
| 280 | + " :param images: a list of images from the dataset.\n", |
| 281 | + " \"\"\"\n", |
| 282 | + " predicted_labels = []\n", |
| 283 | + " infer_request = net.create_infer_request()\n", |
| 284 | + " for idx in indexes:\n", |
| 285 | + " assert idx < 10000, 'Cannot get such index, there are only 10000'\n", |
| 286 | + " input_tensor = Tensor(array=images[idx].detach().numpy(), shared_memory=True)\n", |
| 287 | + " infer_request.set_input_tensor(input_tensor)\n", |
| 288 | + " infer_request.start_async()\n", |
| 289 | + " infer_request.wait()\n", |
| 290 | + " output = infer_request.get_output_tensor()\n", |
| 291 | + " result = list(output.data)\n", |
| 292 | + " result = labels_names[np.argmax(result[0])]\n", |
| 293 | + " predicted_labels.append(result)\n", |
| 294 | + " return predicted_labels" |
| 295 | + ] |
| 296 | + }, |
| 297 | + { |
| 298 | + "cell_type": "code", |
| 299 | + "execution_count": null, |
| 300 | + "metadata": {}, |
| 301 | + "outputs": [], |
| 302 | + "source": [ |
| 303 | + "indexes_to_infer = [0, 1, 2] # to plot specify indexes\n", |
| 304 | + "\n", |
| 305 | + "plot_pictures(indexes_to_infer)\n", |
| 306 | + "\n", |
| 307 | + "results_quanized = infer_on_images(compiled_model, indexes_to_infer)\n", |
| 308 | + "\n", |
| 309 | + "print(f\"Image labels using the quantized model : {results_quanized}.\")" |
| 310 | + ] |
| 311 | + } |
| 312 | + ], |
| 313 | + "metadata": { |
| 314 | + "accelerator": "GPU", |
| 315 | + "colab": { |
| 316 | + "collapsed_sections": [ |
| 317 | + "K5HPrY_d-7cV", |
| 318 | + "E01dMaR2_AFL", |
| 319 | + "qMnYsGo9_MA8", |
| 320 | + "L0tH9KdwtHhV" |
| 321 | + ], |
| 322 | + "name": "INT8 Quantization with POT in Simplified Mode tutorial", |
| 323 | + "provenance": [] |
| 324 | + }, |
| 325 | + "interpreter": { |
| 326 | + "hash": "852430a53033d44ce17eefa3d9017e4bc4dca84b51e1c38a8c71aec2fd2e4d43" |
| 327 | + }, |
| 328 | + "kernelspec": { |
| 329 | + "display_name": "openvino_env", |
| 330 | + "language": "python", |
| 331 | + "name": "python3" |
| 332 | + }, |
| 333 | + "language_info": { |
| 334 | + "codemirror_mode": { |
| 335 | + "name": "ipython", |
| 336 | + "version": 3 |
| 337 | + }, |
| 338 | + "file_extension": ".py", |
| 339 | + "mimetype": "text/x-python", |
| 340 | + "name": "python", |
| 341 | + "nbconvert_exporter": "python", |
| 342 | + "pygments_lexer": "ipython3", |
| 343 | + "version": "3.7.5" |
| 344 | + } |
| 345 | + }, |
| 346 | + "nbformat": 4, |
| 347 | + "nbformat_minor": 4 |
| 348 | +} |
0 commit comments