notebooks/103-paddle-onnx-to-openvino/103-paddle-onnx-to-openvino-classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "545c60f5-afbd-41e8-bf1a-fe437c8e4f11",
   "metadata": {
    "id": "JwEAhQVzkAwA"
   },
   "source": [
    "# Convert PaddlePaddle Classification Model to ONNX and OpenVINO IR\n",
    "\n",
    "This notebook shows how to convert a MobileNet V3 [PaddleHub](https://github.com/PaddlePaddle/PaddleHub) model, pretrained on the [ImageNet](https://www.image-net.org) dataset, to OpenVINO IR. It also shows how to perform classification inference on an image using OpenVINO [Inference Engine](https://docs.openvinotoolkit.org/latest/openvino_docs_IE_DG_Deep_Learning_Inference_Engine_DevGuide.html) and compares the results of the PaddlePaddle model with the IR model. \n",
    "\n",
    "The model source is https://www.paddlepaddle.org.cn/hubdetail?name=mobilenet_v3_large_imagenet_ssld&en_category=ImageClassification"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa1a7c38-bed0-4f61-8436-e7d53e1b8aed",
   "metadata": {},
   "source": [
    "## Preparation\n",
    "\n",
    "### Install PaddlePaddle and Upgrade OpenVINO\n",
    "\n",
    "The Model Optimizer command in this notebook, `mo`, was introduced in OpenVINO 2021.4. If your OpenVINO version is lower, running this cell will upgrade openvino-dev. You can also upgrade all the packages in openvino_env by running `pip install --upgrade -r requirements.txt` in a terminal where you activated the openvino_env environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6859fa3-f09a-4df5-9b22-9c8114663d28",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openvino.inference_engine\n",
    "\n",
    "if \"2021.4\" not in openvino.inference_engine.get_version():\n",
    "    print(\"Installing OpenVINO 2021.4. This may take a while...\")\n",
    "    print(\"It is recommended to restart the Jupyter Kernel after installation, \"\n",
    "          \"with Kernel->Restart Kernel\")\n",
    "    install_result = %sx python -m pip install --upgrade openvino-dev==2021.4.*\n",
    "    install_result = [line for line in install_result if \"Requirement already \" not in line]\n",
    "    print(\"\\n\".join(install_result))\n",
    "else:\n",
    "    print(\"OpenVINO 2021.4 is installed.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75e4d91f-8f97-4690-9c0d-aa8b9c84f4a9",
   "metadata": {},
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db25c6f1-db8b-4b17-a6c7-6f351508ba90",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import time\n",
    "\n",
    "import cv2\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import paddlehub as hub\n",
    "from IPython.display import Markdown, display\n",
    "from openvino.inference_engine import IECore\n",
    "from paddle.static import InputSpec\n",
    "from PIL import Image\n",
    "from scipy.special import softmax"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb5d1b57-8cf5-493b-9cd6-b67800542ea8",
   "metadata": {},
   "source": [
    "### Settings\n",
    "\n",
    "Set `IMAGE_FILENAME` to the filename of an image to use. Set `MODEL_NAME` to the PaddlePaddle model to download from PaddleHub. `MODEL_NAME` will also be the base name for the converted ONNX and IR models. The notebook is tested with the [mobilenet_v3_large_imagenet_ssld](https://www.paddlepaddle.org.cn/hubdetail?name=mobilenet_v3_large_imagenet_ssld&en_category=ImageClassification) model. Other models may use different preprocessing methods and therefore require some modification to get the same results on the original and converted model.\n",
    "\n",
    "`hub.config.server` is the URL to the PaddleHub server. You should not need to modify this setting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cf545ce-15c9-4bb6-9994-896019091b7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "IMAGE_FILENAME = \"coco_close.png\"\n",
    "\n",
    "MODEL_NAME = \"mobilenet_v3_large_imagenet_ssld\"\n",
    "hub.config.server = \"https://paddlepaddle.org.cn/paddlehub\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03046a49-b5dc-4309-891d-4fda3ec8a018",
   "metadata": {},
   "source": [
    "## Show Inference on PaddlePaddle Model\n",
    "\n",
    "In the next cell, we load and download the model from PaddleHub, read and display an image, do inference on that image, and show the top three prediction results.\n",
    "\n",
    "The first time you run this notebook, the PaddlePaddle model is downloaded from PaddleHub. This may take a while."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63cb401f-c0a4-4e11-a386-8b023c097c75",
   "metadata": {},
   "outputs": [],
   "source": [
    "classifier = hub.Module(name=MODEL_NAME)\n",
    "# Load image in BGR format, as specified in model documentation\n",
    "image = cv2.imread(IMAGE_FILENAME)\n",
    "plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))\n",
    "result = classifier.classification(images=[image], top_k=3)\n",
    "for class_name, softmax_probability in result[0].items():\n",
    "    print(f\"{class_name}, {softmax_probability:.5f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e02f54bf-b40f-4261-93d6-8618cc297a1b",
   "metadata": {},
   "source": [
    "`classifier.classification()` takes an image as input and returns the class name of the image. By default the best network result is returned. With the `top_k` parameter, the best `k` results are returned, where `k` is a number. Preprocessing the image and converting the network result to class names is done behind the scenes. The classification model returns an array with floating point values for each of the 1000 ImageNet classes. The higher the value, the more confident the network is that the class number corresponding to that value (the index of that value in the network output array) is the class number for the image. The `classification()` function converts these numbers to class names and [softmax](https://en.wikipedia.org/wiki/Softmax_function)  probabilities. \n",
    "\n",
    "To see PaddlePaddle's implementation for the classification function and for loading and preprocessing data, uncomment the next two cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba16b03f-d886-481b-9d01-0caedc48c688",
   "metadata": {},
   "outputs": [],
   "source": [
    "# classifier??"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51263cfe-4d3d-4aa5-b920-0aec02177292",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import mobilenet_v3_large_imagenet_ssld.data_feed\n",
    "# %load $mobilenet_v3_large_imagenet_ssld.data_feed.__file__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da21823e-60ce-494a-8c41-9418877cada5",
   "metadata": {},
   "source": [
    "The `data_feed` module shows that images are normalized, resized and cropped and that the BGR image is converted to RGB before propagating through the network. In the next cell, we import the `process_image` function that is defined in this file to do inference on the OpenVINO IR model with the same method. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64a4a36f-d7ed-4173-872a-fe545b4f0477",
   "metadata": {},
   "outputs": [],
   "source": [
    "from mobilenet_v3_large_imagenet_ssld.data_feed import process_image"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fe900d4-02d5-4f4c-aca4-195efc4da3b8",
   "metadata": {},
   "source": [
    "It is useful to show the output of the `process_image()` function, to see the effect of cropping and resizing. Because of the normalization, the colors will look strange, and matplotlib will warn about clipping values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09c84353-c6f0-4ff4-b90f-70bd90dd4dea",
   "metadata": {},
   "outputs": [],
   "source": [
    "pil_image = Image.open(IMAGE_FILENAME)\n",
    "processed_image = process_image(pil_image)\n",
    "print(f\"Processed image shape: {processed_image.shape}\")\n",
    "# Processed image is in (C,H,W) format, convert to (H,W,C) to show the image\n",
    "plt.imshow(np.transpose(processed_image, (1, 2, 0)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc0cbe95-690c-4078-b8d5-f1b063357d26",
   "metadata": {},
   "source": [
    "## Convert the Model to OpenVINO IR Format\n",
    "\n",
    "To Convert the PaddlePaddle model to IR, we first convert the model to ONNX, and then convert the ONNX model to IR."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d5f3fce-c7ff-46d9-b964-90804a813cda",
   "metadata": {},
   "source": [
    "#### Preparation\n",
    "\n",
    "PaddlePaddle's MobileNet model contains information about the input shape, mean and scale values that we can use to convert the model. The next cell shows how to get these values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89b84988-81b8-4ef8-8820-6167e08e4450",
   "metadata": {},
   "outputs": [],
   "source": [
    "input_shape = list(classifier.cpu_predictor.get_input_tensor_shape().values())\n",
    "print(\"input shape:\", input_shape)\n",
    "print(\"mean:\", classifier.get_pretrained_images_mean())\n",
    "print(\"std:\", classifier.get_pretrained_images_std())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6b2f874-137b-4106-92b3-baecf371df09",
   "metadata": {},
   "source": [
    "#### Convert PaddlePaddle Model to ONNX\n",
    "\n",
    "We convert the PaddlePaddle Model to ONNX with the `.export_onnx_model()` method. This uses [Paddle2ONNX](https://github.com/PaddlePaddle/paddle2onnx). We convert the model with the input shape found in the previous cell.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87d24ca2-ed9f-422d-b9b4-3caa83eaa725",
   "metadata": {},
   "outputs": [],
   "source": [
    "target_height, target_width = next(iter(input_shape))[2:]\n",
    "x_spec = InputSpec([1, 3, target_height, target_width], \"float32\", \"x\")\n",
    "print(\n",
    "    \"Exporting PaddlePaddle model to ONNX with target_height \"\n",
    "    f\"{target_height} and target_width {target_width}\"\n",
    ")\n",
    "classifier.export_onnx_model(\".\", input_spec=[x_spec], opset_version=11)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f369e47-9c83-4b81-bd91-41a80216453a",
   "metadata": {},
   "source": [
    "#### Convert ONNX model to OpenVINO IR Format\n",
    "\n",
    "Call the OpenVINO Model Optimizer tool to convert the PaddlePaddle model to OpenVINO IR, with FP32 precision. The models are saved to the current directory. We can add the mean values to the model with `--mean_values` and scale the output with the standard deviation with `--scale_values`. With these options, it is not necessary to normalize input data before propagating it through the network. However, to get the exact same output as the PaddlePaddle model, it is necessary to preprocess in the image in the same way. For this tutorial, we therefore do not add the mean and scale values to the model, and we use the `process_image` function, as described in the previous section, to ensure that both the IR and the PaddlePaddle model use the same preprocessing methods. We do show how to get the mean and scale values of the PaddleGAN model, so you can add them to the Model Optimizer command if you want. See the [PyTorch/ONNX to OpenVINO](../102-pytorch-onnx-to-openvino/102-pytorch-onnx-to-openvino.ipynb) notebook for a notebook where these options are used.\n",
    "\n",
    "Run `! mo --help` in a code cell to show an overview of command line options for Model Optimizer. See the [Model Optimizer Developer Guide](https://docs.openvinotoolkit.org/latest/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html) for more information about Model Optimizer.\n",
    "\n",
    "In the next cell, we first construct the command for Model Optimizer, and then execute this command in the notebook by prepending the command with a `!`. Model Optimization was succesful if the last lines of the output include `[ SUCCESS ] Generated IR version 10 model`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08673c57-a703-4023-b031-18d5a484a7b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_xml = f\"{MODEL_NAME}.xml\"\n",
    "if not os.path.exists(model_xml):\n",
    "    mo_command = f'mo --input_model {MODEL_NAME}.onnx --input_shape \"[1,3,{target_height},{target_width}]\"'\n",
    "    display(Markdown(f\"Model Optimizer command to convert the ONNX model to IR: `{mo_command}`\"))\n",
    "    display(Markdown(\"_Converting model to IR. This may take a few minutes..._\"))\n",
    "    ! $mo_command\n",
    "else:\n",
    "    print(f\"{model_xml} already exists.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eae447ee-622c-44ae-8d31-27716acf3973",
   "metadata": {},
   "source": [
    "## Show Inference on OpenVINO Model\n",
    "\n",
    "Load the IR model, get model information, load the image, do inference, convert the inference to a meaningful result, and show the output. See the [Inference Engine API Notebook](../002-openvino-api/002-openvino-api.ipynb) for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "764ed735-db44-47e7-a65d-f8b2e26c71de",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load Inference Engine and IR model\n",
    "ie = IECore()\n",
    "net = ie.read_network(f\"{MODEL_NAME}.xml\")\n",
    "exec_net = ie.load_network(net, \"CPU\")\n",
    "\n",
    "# Get model input and outputs\n",
    "input_layer = next(iter(net.input_info))\n",
    "output_layer = next(iter(net.outputs))\n",
    "\n",
    "# Read, show, and preprocess input image\n",
    "# See the \"Show Inference on PaddlePaddle Model\" section for source of process_image\n",
    "image = Image.open(IMAGE_FILENAME)\n",
    "plt.imshow(image)\n",
    "input_image = process_image(image)\n",
    "\n",
    "# Do inference\n",
    "ie_result = exec_net.infer({input_layer: input_image})[output_layer][0]\n",
    "\n",
    "# Compute softmax probabilities for the inference result and find the top three values\n",
    "softmax_result = softmax(ie_result)\n",
    "top_indices = np.argsort(softmax_result)[-3:][::-1]\n",
    "top_softmax = softmax_result[top_indices]\n",
    "\n",
    "# Convert the inference results to class names, using the same labels as the PaddlePaddle classifier\n",
    "for index, softmax_probability in zip(top_indices, top_softmax):\n",
    "    print(f\"{classifier.label_list[index]}, {softmax_probability:.5f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9fe7a71-7c4e-4c05-9270-d8e4096d9ec8",
   "metadata": {},
   "source": [
    "## Timing and Comparison\n",
    "\n",
    "Measure the time it takes to do inference on fifty images and compare the result. The timing information gives an indication of performance. For a fair comparison, we include the time it takes to process the image. For more accurate benchmarking, use the [OpenVINO benchmark tool](https://github.com/openvinotoolkit/openvino/tree/master/inference-engine/tools/benchmark_tool). Note that many optimizations are possible to improve the performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "427993c9-8877-4776-bd3d-0430ff9b8e1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "num_images = 50\n",
    "# PaddlePaddle's classification method expects a BGR numpy array\n",
    "image = cv2.imread(IMAGE_FILENAME)\n",
    "# The process_image function expects a PIL image\n",
    "pil_image = Image.open(IMAGE_FILENAME)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00c0a72d-5029-4370-936b-30c0d337c282",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show devices available for OpenVINO Inference Engine\n",
    "ie = IECore()\n",
    "for device in ie.available_devices:\n",
    "    device_name = ie.get_metric(device, \"FULL_DEVICE_NAME\")\n",
    "    print(f\"{device}: {device_name}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33a04541-9e17-4352-949e-5e9f7e9926be",
   "metadata": {},
   "outputs": [],
   "source": [
    "start = time.perf_counter()\n",
    "for _ in range(num_images):\n",
    "    result = classifier.classification(images=[image], top_k=3)\n",
    "end = time.perf_counter()\n",
    "time_ir = end - start\n",
    "print(\n",
    "    f\"PaddlePaddle model on CPU: {time_ir/num_images:.4f} \"\n",
    "    f\"seconds per image, FPS: {num_images/time_ir:.2f}\\n\"\n",
    ")\n",
    "print(\"PaddlePaddle result:\")\n",
    "for class_name, softmax_probability in result[0].items():\n",
    "    print(f\"{class_name}, {softmax_probability:.5f}\")\n",
    "plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26bc8d4d-6c5e-4c25-82cd-74199cd9e9d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "device = \"CPU\"\n",
    "exec_net = ie.load_network(net, device)\n",
    "\n",
    "start = time.perf_counter()\n",
    "input_image = process_image(pil_image)\n",
    "\n",
    "for _ in range(num_images):\n",
    "    ie_result = exec_net.infer(inputs={input_layer: input_image})[output_layer][0]\n",
    "\n",
    "    result_index = np.argmax(ie_result)\n",
    "    class_name = classifier.label_list[np.argmax(ie_result)]\n",
    "    softmax_result = softmax(ie_result)\n",
    "    top_indices = np.argsort(softmax_result)[-3:][::-1]\n",
    "    top_softmax = softmax_result[top_indices]\n",
    "\n",
    "end = time.perf_counter()\n",
    "time_ir = end - start\n",
    "\n",
    "\n",
    "print(\n",
    "    f\"IR model in Inference Engine/{device}: {time_ir/num_images:.4f} \"\n",
    "    f\"seconds per image, FPS: {num_images/time_ir:.2f}\\n\"\n",
    ")\n",
    "print(\"OpenVINO result:\")\n",
    "for index, softmax_probability in zip(top_indices, top_softmax):\n",
    "    print(f\"{classifier.label_list[index]}, {softmax_probability:.5f}\")\n",
    "plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5727ce3a-a692-4e6c-abaf-3167de6b9f83",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "openvino_env",
   "language": "python",
   "name": "openvino_env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}