notebooks/214-vision-paddle-classification/214-vision-paddle-classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5206ff2e-8ee6-47b0-abef-eb1b7f9ec5ee",
   "metadata": {},
   "source": [
    "# PaddlePaddle Image Classification with OpenVINO\n",
    "This demo shows how to run a MobileNetV3 Large PaddePaddle model using OpenVINO Runtime. Instead of exporting the PaddlePaddle model to ONNX and converting to Intermediate Representation (IR) format using Model Optimizer, we can now read the Paddle model directly without conversion."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29258d2e",
   "metadata": {},
   "source": [
    "## Import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe0dacf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# model download\n",
    "from pathlib import Path\n",
    "import os\n",
    "import urllib.request\n",
    "import tarfile\n",
    "\n",
    "# inference\n",
    "from openvino.runtime import Core\n",
    "\n",
    "# preprocessing\n",
    "import cv2\n",
    "import numpy as np\n",
    "from openvino.preprocess import PrePostProcessor, ResizeAlgorithm\n",
    "from openvino.runtime import Layout, Type, AsyncInferQueue, PartialShape\n",
    "\n",
    "# results visualization\n",
    "import time\n",
    "import json\n",
    "from IPython.display import Image"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d3dbb77-5cd6-46ad-b6e4-ba75f0ee795f",
   "metadata": {},
   "source": [
    "## Download the MobileNetV3_large_x1_0 Model\n",
    "Download the pre-trained model directly from the server. More details about the pre-trained model can be found in the PaddleClas documentation below.\n",
    "\n",
    "Source: https://github.com/PaddlePaddle/PaddleClas/blob/release/2.2/deploy/lite/readme_en.md"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1d476b3-0d6e-420b-9e51-9a1629bda494",
   "metadata": {},
   "outputs": [],
   "source": [
    "mobilenet_url = \"https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/inference/MobileNetV3_large_x1_0_infer.tar\"\n",
    "mobilenetv3_model_path = Path(\"model/MobileNetV3_large_x1_0_infer/inference.pdmodel\")\n",
    "if mobilenetv3_model_path.is_file(): \n",
    "    print(\"Model MobileNetV3_large_x1_0 already exists\")\n",
    "else:\n",
    "    # Download the model from the server, and untar it.\n",
    "    print(\"Downloading the MobileNetV3_large_x1_0_infer model (20Mb)... May take a while...\")\n",
    "    # create a directory \n",
    "    os.makedirs(\"model\")\n",
    "    urllib.request.urlretrieve(mobilenet_url, \"model/MobileNetV3_large_x1_0_infer.tar\")\n",
    "    print(\"Model Downloaded\")\n",
    "\n",
    "    file = tarfile.open(\"model/MobileNetV3_large_x1_0_infer.tar\")\n",
    "    res = file.extractall(\"model\")\n",
    "    file.close()\n",
    "    if (not res):\n",
    "        print(f\"Model Extracted to {mobilenetv3_model_path}.\")\n",
    "    else:\n",
    "        print(\"Error Extracting the model. Please check the network.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3a72a48-494e-44f3-9ee8-b2c15c845fdc",
   "metadata": {},
   "source": [
    "## Define the callback function for postprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36b52bfa-551c-4d58-a796-7a0cb7e8c357",
   "metadata": {},
   "outputs": [],
   "source": [
    "def callback(infer_request, i) -> None:\n",
    "    \"\"\"\n",
    "    Define the callback function for postprocessing\n",
    "    \n",
    "    :param: infer_request: the infer_request object\n",
    "            i: the iteration of inference\n",
    "    :retuns:\n",
    "            None\n",
    "    \"\"\"\n",
    "    imagenet_classes = json.loads(open(\"utils/imagenet_class_index.json\").read())\n",
    "    predictions = next(iter(infer_request.results.values()))\n",
    "    indices = np.argsort(-predictions[0])\n",
    "    if (i == 0):\n",
    "        # Calculate the first inference time\n",
    "        latency = (time.time() - start) * 1000\n",
    "        print(\"first inference latency: {:.2f} ms\".format(latency))\n",
    "        for n in range(5):\n",
    "            print(\n",
    "                \"class name: {}, probability: {:.5f}\"\n",
    "                .format(imagenet_classes[str(list(indices)[n])][1], predictions[0][list(indices)[n]])\n",
    "            )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8adfba1-82d2-4526-910f-2d32530d74eb",
   "metadata": {},
   "source": [
    "## Read the model\n",
    "OpenVINO Runtime reads the `PaddlePaddle` model directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "179ae1a7-966d-49fe-9c06-811257c57989",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Intialize Inference Engine with Core()\n",
    "ie = Core()\n",
    "# MobileNetV3_large_x1_0\n",
    "model = ie.read_model(mobilenetv3_model_path)\n",
    "# get the information of intput and output layer\n",
    "input_layer = model.input(0)\n",
    "output_layer = model.output(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc6a1ee1-802b-4561-af31-a613bba10fc6",
   "metadata": {},
   "source": [
    "## Integrate preprocessing steps into the execution graph with Preprocessing API\n",
    "If your input data does not fit perfectly in the model input tensor additional operations/steps are needed to transform the data to a format expected by the model. These operations are known as “preprocessing”.\n",
    "Preprocessing steps are integrated into the execution graph and performed on the selected device(s) (CPU/GPU/VPU/etc.) rather than always executed on CPU. This improves utilization on the selected device(s).\n",
    "\n",
    "Overview of `Preprocessing API`: https://docs.openvino.ai/latest/openvino_docs_OV_Runtime_UG_Preprocessing_Overview.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "998aa8d0-20d7-471a-b4c5-25dd52c51881",
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = \"../001-hello-world/data/coco.jpg\"\n",
    "test_image = cv2.imread(filename) \n",
    "test_image = np.expand_dims(test_image, 0) / 255\n",
    "_, h, w, _ = test_image.shape\n",
    "\n",
    "# Adjust model input shape to improve the performance\n",
    "model.reshape({input_layer.any_name: PartialShape([1, 3, 224, 224])})\n",
    "ppp = PrePostProcessor(model)\n",
    "# Set input tensor information:\n",
    "# - input() provides information about a single model input\n",
    "# - layout of data is \"NHWC\"\n",
    "# - set static spatial dimensions to input tensor to resize from\n",
    "ppp.input().tensor() \\\n",
    "    .set_spatial_static_shape(h, w) \\\n",
    "    .set_layout(Layout(\"NHWC\")) \n",
    "inputs = model.inputs\n",
    "# Here we assume the model has \"NCHW\" layout for input\n",
    "ppp.input().model().set_layout(Layout(\"NCHW\"))\n",
    "# Do prepocessing:\n",
    "# - apply linear resize from tensor spatial dims to model spatial dims\n",
    "# - Subtract mean from each channel\n",
    "# - Divide each pixel data to appropriate scale value\n",
    "ppp.input().preprocess() \\\n",
    "    .resize(ResizeAlgorithm.RESIZE_LINEAR, 224, 224) \\\n",
    "    .mean([0.485, 0.456, 0.406]) \\\n",
    "    .scale([0.229, 0.224, 0.225])\n",
    "# Set output tensor information:\n",
    "# - precision of tensor is supposed to be 'f32'\n",
    "ppp.output().tensor().set_element_type(Type.f32)\n",
    "# Apply preprocessing to modify the original 'model'\n",
    "model = ppp.build()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63289a21-3369-4f45-bd36-5f2e4f7aca01",
   "metadata": {},
   "source": [
    "## Run Inference\n",
    "Use `AUTO` as the device name to delegate device selection to OpenVINO. The Auto device plugin internally recognizes and selects devices from among Intel CPU and GPU depending on the device capabilities and the characteristics of the model(s) (for example, precision). Then it assigns inference requests to the best device.\n",
    "`AUTO` starts inference immediately on the CPU and then transparently shifts to the GPU (or VPU) once it is ready, dramatically reducing time to first inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6f306c2-7589-4e0a-8de6-5bfb4b94ce0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the available devices in your system\n",
    "devices = ie.available_devices\n",
    "for device in devices:\n",
    "    device_name = ie.get_property(device_name=device, name=\"FULL_DEVICE_NAME\")\n",
    "    print(f\"{device}: {device_name}\")\n",
    "\n",
    "# Load model to a device selected by AUTO from the available devices list\n",
    "compiled_model = ie.compile_model(model=model, device_name=\"AUTO\")\n",
    "# Create infer request queue\n",
    "infer_queue = AsyncInferQueue(compiled_model)\n",
    "infer_queue.set_callback(callback)\n",
    "start = time.time()\n",
    "# Do inference\n",
    "infer_queue.start_async({input_layer.any_name: test_image}, 0)\n",
    "infer_queue.wait_all()\n",
    "Image(filename=filename) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7a206a3",
   "metadata": {},
   "source": [
    "## Performance Hints: Latency and Throughput\n",
    "Throughput and latency are some of the most widely used metrics that measure the overall performance of an application.\n",
    "<img align='center' src=\"https://user-images.githubusercontent.com/10940214/160096393-cfd4cfaf-5fb2-4c35-9e6e-bdd216820a82.png\" alt=\"drawing\" width=\"1000\"/>\n",
    "\n",
    "- **Latency** measures inference time (ms) required to process a single input or First inference.\n",
    "- To calculate **throughput**, divide number of inputs that were processed by the processing time.\n",
    "\n",
    "The OpenVINO performance hints are the new way to configure the performance with the portability in mind. Performance Hints will let the device to configure itself, rather than map the application needs to the low-level performance settings, and keep an associated application logic to configure each possible device separately. \n",
    "\n",
    "High-level Performance Hints: https://docs.openvino.ai/latest/openvino_docs_OV_UG_Performance_Hints.html\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be337a5a-1e28-42cc-bb11-0fe9db5986c7",
   "metadata": {},
   "source": [
    "<br/>\n",
    " \n",
    "**Run Inference with \"LATENCY\" Performance Hint**\n",
    "\n",
    "It is possible to define application-specific performance settings with a config key, letting the device adjust to achieve better `LATENCY` oriented performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d46e985-258b-4650-aede-54a83c3aa03d",
   "metadata": {},
   "outputs": [],
   "source": [
    "loop = 100\n",
    "# AUTO sets device config based on hints\n",
    "compiled_model = ie.compile_model(model=model, device_name=\"AUTO\", config={\"PERFORMANCE_HINT\": \"LATENCY\"})\n",
    "infer_queue = AsyncInferQueue(compiled_model)\n",
    "# implement AsyncInferQueue Python API to boost the performance in Async mode\n",
    "infer_queue.set_callback(callback)\n",
    "# run infernce for 100 times to get the average FPS\n",
    "start = time.time()\n",
    "for i in range(loop):\n",
    "    infer_queue.start_async({input_layer.any_name: test_image}, i)\n",
    "infer_queue.wait_all()\n",
    "end = time.time()\n",
    "# Calculate the average FPS\n",
    "fps = loop / (end - start) \n",
    "print(\"throughput: {:.2f} fps\".format(fps))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0807fd6b-16ac-4e90-b394-36a88a94abb6",
   "metadata": {},
   "source": [
    " <br/>\n",
    " \n",
    "**Run Inference with \"TRHOUGHPUT\" Performance Hint**\n",
    "\n",
    "It is possible to define application-specific performance settings with a config key, letting the device adjust to achieve better `THROUGHPUT` performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49997f5f-36d6-42b4-bce1-3eb44005fbd5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# AUTO sets device config based on hints\n",
    "compiled_model = ie.compile_model(model=model, device_name=\"AUTO\", config={\"PERFORMANCE_HINT\": \"THROUGHPUT\"})\n",
    "infer_queue = AsyncInferQueue(compiled_model)\n",
    "infer_queue.set_callback(callback)\n",
    "start = time.time()\n",
    "for i in range(loop):\n",
    "    infer_queue.start_async({input_layer.any_name: test_image}, i)\n",
    "infer_queue.wait_all()\n",
    "end = time.time()\n",
    "# Calculate the average FPS\n",
    "fps = loop / (end - start)\n",
    "print(\"throughput: {:.2f} fps\".format(fps))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e701661c",
   "metadata": {},
   "source": [
    "## benchmark_app\n",
    "To generate more accurate performance measurements, use the OpenVINO [Benchmark Tool](https://docs.openvino.ai/latest/openvino_inference_engine_tools_benchmark_tool_README.html).\n",
    "we can trigger the \"Performance hint\" by using `-hint` parameter, which instructs the OpenVINO device plugin to use the best network-specific settings for `latency` OR `throughput`.\n",
    "\n",
    "NOTE: The performance results from `benchmark_app` exclude model \"compilation and load time\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89af7e9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 'latency': device performance optimized for LATENCY.\n",
    "! benchmark_app -m $mobilenetv3_model_path -data_shape [1,3,224,224] -hint \"latency\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9671c464",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 'throughput' or 'tput': device performance optimized for THROUGHPUT.\n",
    "! benchmark_app -m $mobilenetv3_model_path -data_shape [1,3,224,224] -hint \"throughput\" "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "openvino_env",
   "language": "python",
   "name": "openvino_env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}