notebooks/110-ct-segmentation-quantize/110-ct-scan-live-inference.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Live Inference and Benchmark CT-scan Data with OpenVINO™\n",
    "\n",
    "## Kidney Segmentation with PyTorch Lightning and OpenVINO™ - Part 4 \n",
    "\n",
    "This tutorial is a part of a series on how to train, optimize, quantize and show live inference on a medical segmentation model. The goal is to accelerate inference on a kidney segmentation model. The [UNet](https://arxiv.org/abs/1505.04597) model is trained from scratch, and the data is from [Kits19](https://github.com/neheller/kits19).\n",
    "\n",
    "This tutorial shows how to benchmark performance of the model and show live inference with async API and MULTI plugin in OpenVINO.\n",
    "\n",
    "This notebook needs a quantized OpenVINO IR model and images from the [KiTS-19](https://github.com/neheller/kits19) dataset, converted to 2D images. (To learn how the model is quantized, see the [Convert and Quantize a UNet Model and Show Live Inference](110-ct-segmentation-quantize.ipynb) tutorial.)\n",
    "\n",
    "This notebook provides a pre-trained model, trained for 20 epochs with the full KiTS-19 frames dataset, which has an F1 score on the validation set of 0.9. The training code is available in the [PyTorch Monai Training](../110-ct-segmentation-quantize/pytorch-monai-training.ipynb) notebook. \n",
    "\n",
    "For demonstration purposes, this tutorial will download one converted CT scan to use for inference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import zipfile\n",
    "from pathlib import Path\n",
    "\n",
    "import numpy as np\n",
    "from monai.transforms import LoadImage\n",
    "from openvino.runtime import Core\n",
    "\n",
    "from custom_segmentation import SegmentationModel\n",
    "\n",
    "sys.path.append(\"../utils\")\n",
    "from notebook_utils import download_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Settings\n",
    "\n",
    "To use the pre-trained models, set `IR_PATH` to `\"pretrained_model/unet44.xml\"` and `COMPRESSED_MODEL_PATH` to `\"pretrained_model/quantized_unet44.xml\"`. To use a model that you trained or optimized yourself, adjust the model paths."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The directory that contains the IR model (xml and bin) files.\n",
    "MODEL_PATH = \"pretrained_model/quantized_unet_kits19.xml\"\n",
    "# Uncomment the next line to use the FP16 model instead of the quantized model.\n",
    "# MODEL_PATH = \"pretrained_model/unet_kits19.xml\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benchmark Model Performance\n",
    "To measure the inference performance of the IR model, use [Benchmark Tool](https://docs.openvino.ai/latest/openvino_inference_engine_tools_benchmark_tool_README.html) - an inference performance measurement tool in OpenVINO. Benchmark tool is a command-line application that can be run in the notebook with `! benchmark_app` or `%sx benchmark_app` commands.\n",
    "\n",
    "> **Note**: The `benchmark_app` tool is able to measure the performance of the OpenVINO Intermediate Representation (OpenVINO IR) models only. For more accurate performance, run `benchmark_app` in a terminal/command prompt after closing other applications. Run `benchmark_app -m model.xml -d CPU` to benchmark async inference on CPU for one minute. Change `CPU` to `GPU` to benchmark on GPU. Run `benchmark_app --help` to see an overview of all command-line options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ie = Core()\n",
    "# By default, benchmark on MULTI:CPU,GPU if a GPU is available, otherwise on CPU.\n",
    "device = \"MULTI:CPU,GPU\" if \"GPU\" in ie.available_devices else \"CPU\"\n",
    "# Uncomment one of the options below to benchmark on other devices.\n",
    "# device = \"GPU\"\n",
    "# device = \"CPU\"\n",
    "# device = \"AUTO\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "test_replace": {
     "-t 15": "-t 3"
    }
   },
   "outputs": [],
   "source": [
    "# Benchmark model\n",
    "! benchmark_app -m $MODEL_PATH -d $device -t 15 -api sync"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download and Prepare Data\n",
    "\n",
    "Download one validation video for live inference. \n",
    "\n",
    "This tutorial reuses the `KitsDataset` class that was also used in the training and quantization notebook that will be released later.\n",
    "\n",
    "The data is expected in `BASEDIR`. The `BASEDIR` directory should contain the `case_00000` to `case_00299` subdirectories. If the data for the case specified above does not already exist, it will be downloaded and extracted in the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Directory that contains the CT scan data. This directory should contain subdirectories\n",
    "# case_00XXX where XXX is between 000 and 299.\n",
    "BASEDIR = Path(\"kits19_frames_1\")\n",
    "# The CT scan case number. For example: 16 for data from the case_00016 directory.\n",
    "# Currently only 117 is supported.\n",
    "CASE = 117\n",
    "\n",
    "case_path = BASEDIR / f\"case_{CASE:05d}\"\n",
    "\n",
    "if not case_path.exists():\n",
    "    filename = download_file(\n",
    "        f\"https://storage.openvinotoolkit.org/data/test_data/openvino_notebooks/kits19/case_{CASE:05d}.zip\"\n",
    "    )\n",
    "    with zipfile.ZipFile(filename, \"r\") as zip_ref:\n",
    "        zip_ref.extractall(path=BASEDIR)\n",
    "    os.remove(filename)  # remove zipfile\n",
    "    print(f\"Downloaded and extracted data for case_{CASE:05d}\")\n",
    "else:\n",
    "    print(f\"Data for case_{CASE:05d} exists\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Show Live Inference\n",
    "\n",
    "To show live inference on the model in the notebook, use the asynchronous processing feature of OpenVINO Runtime.\n",
    "\n",
    "If you use a GPU device, with `device=\"GPU\"` or `device=\"MULTI:CPU,GPU\"` to do inference on an integrated graphics card, model loading will be slow the first time you run this code. The model will be cached, so after the first time model loading will be faster. For more information on OpenVINO Runtime, including Model Caching, refer to the [OpenVINO API tutorial](../002-openvino-api/002-openvino-api.ipynb).\n",
    "\n",
    "We will use [AsyncInferQueue](https://docs.openvino.ai/latest/openvino_docs_OV_UG_Python_API_exclusives.html#asyncinferqueue) to perform asynchronous inference. It can be instantiated with compiled model and a number of jobs - parallel execution threads. If you don't pass a number of jobs or pass `0`, then OpenVINO will pick the optimal number based on your device and heuristics. After acquiring the inference queue, there are two jobs to do:\n",
    "- Preprocess the data and push it to the inference queue. The preprocessing steps will remain the same.\n",
    "- Tell the inference queue what to do with the model output after the inference is finished. It is represented by the `callback` python function that takes an inference result and data that we passed to the inference queue along with the prepared input data\n",
    "\n",
    "Everything else will be handled by the `AsyncInferQueue` instance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Model and List of Image Files\n",
    "\n",
    "Load the segmentation model to OpenVINO Runtime with `SegmentationModel`, based on the Model API from [Open Model Zoo](https://github.com/openvinotoolkit/open_model_zoo/). This model implementation includes pre and post processing for the model. For `SegmentationModel` this includes the code to create an overlay of the segmentation mask on the original image/frame. Uncomment the next cell to see the implementation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "122wcKhzXn3z",
    "tags": []
   },
   "outputs": [],
   "source": [
    "ie = Core()\n",
    "segmentation_model = SegmentationModel(\n",
    "    ie=ie, model_path=Path(MODEL_PATH), sigmoid=True, rotate_and_flip=True\n",
    ")\n",
    "image_paths = sorted(case_path.glob(\"imaging_frames/*jpg\"))\n",
    "\n",
    "print(f\"{case_path.name}, {len(image_paths)} images\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preapre images\n",
    "\n",
    "Use the `reader = LoadImage()` function to read the images in the same way as in the [training](../110-ct-segmentation-quantize/pytorch-monai-training.ipynb) tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "test_replace": {
     "len(image_paths)": "len(image_paths[:5])"
    }
   },
   "outputs": [],
   "source": [
    "framebuf = []\n",
    "\n",
    "next_frame_id = 0\n",
    "reader = LoadImage(image_only=True, dtype=np.uint8)\n",
    "\n",
    "while next_frame_id < len(image_paths) - 1:\n",
    "    image_path = image_paths[next_frame_id]\n",
    "    image = reader(str(image_path))\n",
    "    framebuf.append(image)\n",
    "    next_frame_id += 1"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Specify device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Possible options for device include \"CPU\", \"GPU\", \"AUTO\", \"MULTI\".\n",
    "device = \"MULTI:CPU,GPU\" if \"GPU\" in ie.available_devices else \"CPU\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting callback function\n",
    "\n",
    "When `callback` is set, any job that ends the inference, calls the Python function. The `callback` function must have two arguments: one is the request that calls the `callback`, which provides the InferRequest API; the other is called “userdata”, which provides the possibility of passing runtime values.\n",
    "\n",
    "The `callback` function will show the results of inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2\n",
    "import copy\n",
    "from IPython import display\n",
    "\n",
    "from typing import Dict, Any\n",
    "from openvino.runtime import InferRequest\n",
    "\n",
    "\n",
    "# Define a callback function that runs every time the asynchronous pipeline completes inference on a frame\n",
    "def completion_callback(infer_request: InferRequest, user_data: Dict[str, Any],) -> None:\n",
    "    preprocess_meta = user_data['preprocess_meta']\n",
    "    \n",
    "    raw_outputs = {out.any_name: copy.deepcopy(res.data) for out, res in zip(infer_request.model_outputs, infer_request.output_tensors)}\n",
    "    frame = segmentation_model.postprocess(raw_outputs, preprocess_meta)\n",
    "\n",
    "    _, encoded_img = cv2.imencode(\".jpg\", frame, params=[cv2.IMWRITE_JPEG_QUALITY, 90])\n",
    "    # Create IPython image\n",
    "    i = display.Image(data=encoded_img)\n",
    "\n",
    "    # Display the image in this notebook\n",
    "    display.clear_output(wait=True)\n",
    "    display.display(i)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create asynchronous inference queue and perform it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from openvino.runtime import AsyncInferQueue\n",
    "\n",
    "load_start_time = time.perf_counter()\n",
    "compiled_model = ie.compile_model(segmentation_model.net, device)\n",
    "# Create asynchronous inference queue with optimal number of infer requests\n",
    "infer_queue = AsyncInferQueue(compiled_model)\n",
    "infer_queue.set_callback(completion_callback)\n",
    "load_end_time = time.perf_counter()\n",
    "\n",
    "results = [None] * len(framebuf)\n",
    "frame_number = 0\n",
    "\n",
    "# Perform inference on every frame in the framebuffer\n",
    "start_time = time.time()\n",
    "for i, input_frame in enumerate(framebuf):\n",
    "    inputs, preprocessing_meta = segmentation_model.preprocess({segmentation_model.net.input(0): input_frame})\n",
    "    infer_queue.start_async(inputs, {'preprocess_meta': preprocessing_meta})\n",
    "\n",
    "# Wait until all inference requests in the AsyncInferQueue are completed\n",
    "infer_queue.wait_all()\n",
    "stop_time = time.time()\n",
    "\n",
    "# Calculate total inference time and FPS\n",
    "total_time = stop_time - start_time\n",
    "fps = len(framebuf) / total_time\n",
    "time_per_frame = 1 / fps \n",
    "\n",
    "print(f\"Loaded model to {device} in {load_end_time-load_start_time:.2f} seconds.\")\n",
    "\n",
    "print(f'Total time to infer all frames: {total_time:.3f}s')\n",
    "print(f'Time per frame: {time_per_frame:.6f}s ({fps:.3f} FPS)')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  },
  "toc-autonumbering": false,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}