notebooks/103-paddle-to-openvino/103-paddle-to-openvino-classification.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "38c66e13",
   "metadata": {
    "id": "JwEAhQVzkAwA"
   },
   "source": [
    "# Convert a PaddlePaddle Model to OpenVINO™ IR\n",
    "\n",
    "This notebook shows how to convert a MobileNetV3 model from [PaddleHub](https://github.com/PaddlePaddle/PaddleHub), pre-trained on the [ImageNet](https://www.image-net.org) dataset, to OpenVINO IR. It also shows how to perform classification inference on a sample image, using [OpenVINO Runtime](https://docs.openvino.ai/latest/openvino_docs_IE_DG_Deep_Learning_Inference_Engine_DevGuide.html) and compares the results of the [PaddlePaddle](https://github.com/PaddlePaddle/Paddle) model with the IR model.\n",
    "\n",
    "Source of the [model](https://www.paddlepaddle.org.cn/hubdetail?name=mobilenet_v3_large_imagenet_ssld&en_category=ImageClassification)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9606e34e",
   "metadata": {},
   "source": [
    "## Preparation\n",
    "\n",
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a81a53c3-9dd5-4c55-8773-b812f86d715f",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install paddleclas --no-deps\n",
    "!pip install \"prettytable\" \"ujson\" \"visualdl>=2.2.0\" \"faiss-cpu>=1.7.1\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33c8d915",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import sys\n",
    "import tarfile\n",
    "from pathlib import Path\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from paddleclas import PaddleClas\n",
    "from IPython.display import Markdown, display\n",
    "from PIL import Image\n",
    "from openvino.runtime import Core\n",
    "sys.path.append(\"../utils\")\n",
    "from notebook_utils import download_file"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "137ef187",
   "metadata": {},
   "source": [
    "### Settings\n",
    "\n",
    "Set `IMAGE_FILENAME` to the filename of an image to use. Set `MODEL_NAME` to the PaddlePaddle model to download from PaddleHub. `MODEL_NAME` will also be the base name for the IR model. The notebook is tested with the [mobilenet_v3_large_x1_0](https://github.com/PaddlePaddle/PaddleClas/blob/release/2.5/docs/en/models/Mobile_en.md) model. Other models may use different preprocessing methods and therefore require some modification to get the same results on the original and converted model.\n",
    "\n",
    "First of all, we need to download and unpack model files.\n",
    "The first time you run this notebook, the PaddlePaddle model is downloaded from PaddleHub. This may take a while."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2bebd4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "IMAGE_FILENAME = \"../data/image/coco_close.png\"\n",
    "\n",
    "MODEL_NAME = \"MobileNetV3_large_x1_0\"\n",
    "MODEL_DIR = Path(\"model\")\n",
    "if not MODEL_DIR.exists():\n",
    "    MODEL_DIR.mkdir()\n",
    "MODEL_URL = 'https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/inference/{}_infer.tar'.format(MODEL_NAME)\n",
    "download_file(MODEL_URL, directory=MODEL_DIR)\n",
    "file = tarfile.open(MODEL_DIR / '{}_infer.tar'.format(MODEL_NAME))\n",
    "res = file.extractall(MODEL_DIR)\n",
    "if not res:\n",
    "    print(f\"Model Extracted to \\\"./{MODEL_DIR}\\\".\")\n",
    "else:\n",
    "    print(\"Error Extracting the model. Please check the network.\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f42abed1",
   "metadata": {},
   "source": [
    "## Show Inference on PaddlePaddle Model\n",
    "\n",
    "In the next cell, we load the model, load and display an image, do inference on that image, and then show the top three prediction results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "154962c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "classifier = PaddleClas(inference_model_dir=MODEL_DIR / '{}_infer'.format(MODEL_NAME))\n",
    "result = next(classifier.predict(IMAGE_FILENAME))\n",
    "class_names = result[0]['label_names']\n",
    "scores = result[0]['scores']\n",
    "image = Image.open(IMAGE_FILENAME)\n",
    "plt.imshow(image)\n",
    "for class_name, softmax_probability in zip(class_names, scores):\n",
    "    print(f\"{class_name}, {softmax_probability:.5f}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "510082f7",
   "metadata": {},
   "source": [
    "`classifier.predict()` takes an image file name, reads the image, preprocesses the input, then returns the class labels and scores of the image. Preprocessing the image is done behind the scenes. The classification model returns an array with floating point values for each of the 1000 ImageNet classes. The higher the value, the more confident the network is that the class number corresponding to that value (the index of that value in the network output array) is the class number for the image.\n",
    "\n",
    "To see PaddlePaddle's implementation for the classification function and for loading and preprocessing data, uncomment the next two cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1997d68a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# classifier??"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "646d337b-084a-411e-bbb1-e27eb8770481",
   "metadata": {},
   "outputs": [],
   "source": [
    "# classifier.get_config()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ec080a4d",
   "metadata": {},
   "source": [
    "The `classifier.get_config()` module shows the preprocessing configuration for the model. It should show that images are normalized, resized and cropped, and that the BGR image is converted to RGB before propagating it through the network. In the next cell, we get the `classifier.predictror.preprocess_ops` property that returns list of preprocessing operations to do inference on the OpenVINO IR model using the same method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4c0a6ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocess_ops = classifier.predictor.preprocess_ops\n",
    "\n",
    "\n",
    "def process_image(image):\n",
    "    for op in preprocess_ops:\n",
    "        image = op(image)\n",
    "    return image"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f5e47e1f",
   "metadata": {},
   "source": [
    "It is useful to show the output of the `process_image()` function, to see the effect of cropping and resizing. Because of the normalization, the colors will look strange, and matplotlib will warn about clipping values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "005cf5e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "pil_image = Image.open(IMAGE_FILENAME)\n",
    "processed_image = process_image(np.array(pil_image))\n",
    "print(f\"Processed image shape: {processed_image.shape}\")\n",
    "# Processed image is in (C,H,W) format, convert to (H,W,C) to show the image\n",
    "plt.imshow(np.transpose(processed_image, (1, 2, 0)))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "597bbd42-7706-4b96-a6e7-75cc391d00f4",
   "metadata": {},
   "source": [
    "To decode the labels predicted by the model to names of classes, we need to have a mapping between them. The model config contains information about `class_id_map_file`, which stores such mapping. The code below shows how to parse the mapping into a dictionary to use with the OpenVINO model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "979a2ce9-c059-413d-bf00-f33bdbfdcdde",
   "metadata": {},
   "outputs": [],
   "source": [
    "class_id_map_file = classifier.get_config()['PostProcess']['Topk']['class_id_map_file']\n",
    "class_id_map = {}\n",
    "with open(class_id_map_file, \"r\") as fin:\n",
    "    lines = fin.readlines()\n",
    "    for line in lines:\n",
    "        partition = line.split(\"\\n\")[0].partition(\" \")\n",
    "        class_id_map[int(partition[0])] = str(partition[-1])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "205deec3",
   "metadata": {},
   "source": [
    "## Convert the Model to OpenVINO IR Format\n",
    "\n",
    "Call the OpenVINO Model Optimizer tool to convert the PaddlePaddle model to OpenVINO IR, with FP32 precision. The models are saved to the current directory. You can add the mean values to the model with `--mean_values` and scale the output with the standard deviation with `--scale_values`. With these options, it is not necessary to normalize input data before propagating it through the network. However, to get the exact same output as the PaddlePaddle model, it is necessary to preprocess in the image in the same way. Therefore, for this tutorial, you do not add the mean and scale values to the model, and you use the `process_image` function, as described in the previous section, to ensure that both the IR and the PaddlePaddle model use the same preprocessing methods. It is explained how to get the mean and scale values of the PaddleGAN model, so you can add them to the Model Optimizer command if you want. See the [PyTorch/ONNX to OpenVINO](../102-pytorch-onnx-to-openvino/102-pytorch-onnx-to-openvino.ipynb) notebook, where these options are used.\n",
    "\n",
    "Run `! mo --help` in a code cell to show an overview of command line options for Model Optimizer. See the [Model Optimizer Developer Guide](https://docs.openvino.ai/latest/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html) for more information about Model Optimizer.\n",
    "\n",
    "In the next cell, we first construct the command for Model Optimizer, and then execute this command in the notebook by prepending the command with a `!`. When Model Optimization is successful, the last lines of the output include `[ SUCCESS ] Generated IR version 11 model`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd7628bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_xml = Path(MODEL_NAME).with_suffix('.xml')\n",
    "if not model_xml.exists():\n",
    "    mo_command = f'mo --input_model model/MobileNetV3_large_x1_0_infer/inference.pdmodel --model_name {MODEL_NAME}'\n",
    "    display(Markdown(f\"Model Optimizer command to convert the ONNX model to IR: `{mo_command}`\"))\n",
    "    display(Markdown(\"_Converting model to IR. This may take a few minutes..._\"))\n",
    "    ! $mo_command\n",
    "else:\n",
    "    print(f\"{model_xml} already exists.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3b2a7ca",
   "metadata": {},
   "source": [
    "## Show Inference on OpenVINO Model\n",
    "\n",
    "Load the IR model, get model information, load the image, do inference, convert the inference to a meaningful result, and show the output. See the [OpenVINO Runtime API Notebook](../002-openvino-api/002-openvino-api.ipynb) for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d45659a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load OpenVINO Runtime and OpenVINO IR model\n",
    "ie = Core()\n",
    "model = ie.read_model(model_xml)\n",
    "compiled_model = ie.compile_model(model=model, device_name=\"CPU\")\n",
    "\n",
    "# Get model output\n",
    "output_layer = compiled_model.output(0)\n",
    "\n",
    "# Read, show, and preprocess input image\n",
    "# See the \"Show Inference on PaddlePaddle Model\" section for source of process_image\n",
    "image = Image.open(IMAGE_FILENAME)\n",
    "plt.imshow(image)\n",
    "input_image = process_image(np.array(image))[None,]\n",
    "\n",
    "# Do inference\n",
    "ie_result = compiled_model([input_image])[output_layer][0]\n",
    "\n",
    "# find the top three values\n",
    "top_indices = np.argsort(ie_result)[-3:][::-1]\n",
    "top_scores = ie_result[top_indices]\n",
    "\n",
    "# Convert the inference results to class names, using the same labels as the PaddlePaddle classifier\n",
    "for index, softmax_probability in zip(top_indices, top_scores):\n",
    "    print(f\"{class_id_map[index]}, {softmax_probability:.5f}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7d249c27",
   "metadata": {},
   "source": [
    "## Timing and Comparison\n",
    "\n",
    "Measure the time it takes to do inference on fifty images and compare the result. The timing information gives an indication of performance. For a fair comparison, we include the time it takes to process the image. For more accurate benchmarking, use the [OpenVINO benchmark tool](https://docs.openvino.ai/latest/openvino_inference_engine_tools_benchmark_tool_README.html). Note that many optimizations are possible to improve the performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5f98c52",
   "metadata": {},
   "outputs": [],
   "source": [
    "num_images = 50\n",
    "\n",
    "image = Image.open(fp=IMAGE_FILENAME)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90d0f3d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show CPU information\n",
    "ie = Core()\n",
    "print(f\"CPU: {ie.get_property('CPU', 'FULL_DEVICE_NAME')}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbb63bc3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show inference speed on PaddlePaddle model\n",
    "start = time.perf_counter()\n",
    "for _ in range(num_images):\n",
    "    result = next(classifier.predict(np.array(image)))\n",
    "end = time.perf_counter()\n",
    "time_ir = end - start\n",
    "print(\n",
    "    f\"PaddlePaddle model on CPU: {time_ir/num_images:.4f} \"\n",
    "    f\"seconds per image, FPS: {num_images/time_ir:.2f}\\n\"\n",
    ")\n",
    "print(\"PaddlePaddle result:\")\n",
    "class_names = result[0]['label_names']\n",
    "scores = result[0]['scores']\n",
    "for class_name, softmax_probability in zip(class_names, scores):\n",
    "    print(f\"{class_name}, {softmax_probability:.5f}\")\n",
    "plt.imshow(image);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "860691f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show inference speed on OpenVINO IR model\n",
    "compiled_model = ie.compile_model(model=model, device_name=\"CPU\")\n",
    "output_layer = compiled_model.output(0)\n",
    "\n",
    "\n",
    "start = time.perf_counter()\n",
    "input_image = process_image(np.array(image))[None,]\n",
    "for _ in range(num_images):\n",
    "    ie_result = compiled_model([input_image])[output_layer][0]\n",
    "    top_indices = np.argsort(ie_result)[-5:][::-1]\n",
    "    top_softmax = ie_result[top_indices]\n",
    "\n",
    "end = time.perf_counter()\n",
    "time_ir = end - start\n",
    "\n",
    "print(\n",
    "    f\"OpenVINO IR model in OpenVINO Runtime (CPU): {time_ir/num_images:.4f} \"\n",
    "    f\"seconds per image, FPS: {num_images/time_ir:.2f}\"\n",
    ")\n",
    "print()\n",
    "print(\"OpenVINO result:\")\n",
    "for index, softmax_probability in zip(top_indices, top_softmax):\n",
    "    print(f\"{class_id_map[index]}, {softmax_probability:.5f}\")\n",
    "plt.imshow(image);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2b40736-0198-46c6-87a3-e492458cb9fd",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "* [PaddleClas](https://github.com/PaddlePaddle/PaddleClas)\n",
    "* [OpenVINO PaddlePaddle support](https://docs.openvino.ai/latest/openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_Paddle.html)\n",
    "* [OpenVINO Model Optimizer Documentation](https://docs.openvino.ai/latest/openvino_docs_MO_DG_prepare_model_convert_model_Converting_Model_General.html)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "ac3de435443db6b16b4e91fe40bfa1799db53ed91595065a9c02602f1a2f9f73"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}