notebooks/107-speech-recognition-quantization/107-speech-recognition-quantization-wav2vec2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cacbe6b4",
   "metadata": {
    "id": "rQc-wXjqrEuR"
   },
   "source": [
    "# Quantize Speech Recognition Models with OpenVINO™ Post-Training Optimization Tool ​\n",
    "This tutorial demonstrates how to apply `INT8` quantization to the speech recognition model, known as [Wav2Vec2](https://huggingface.co/docs/transformers/model_doc/wav2vec2), using the [Post-Training Optimization Tool API (POT API)](https://docs.openvino.ai/latest/pot_compression_api_README.html) (part of the [OpenVINO Toolkit](https://docs.openvino.ai/)). This notebook uses a fine-tuned [Wav2Vec2-Base-960h](https://huggingface.co/facebook/wav2vec2-base-960h) [PyTorch](https://pytorch.org/) model trained on the [LibriSpeech ASR corpus](https://www.openslr.org/12). The tutorial is designed to be extendable to custom models and datasets. It consists of the following steps:\n",
    "\n",
    "- Download and prepare the Wav2Vec2 model and LibriSpeech dataset.\n",
    "- Define data loading and accuracy validation functionality.\n",
    "- Prepare the model for quantization.\n",
    "- Run optimization pipeline.\n",
    "- Compare performance of the original and quantized models."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d6b41e6-132b-40da-b3b9-91bacba29e31",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "771388d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import time\n",
    "import re\n",
    "import numpy as np\n",
    "import torch\n",
    "import tarfile\n",
    "from pathlib import Path\n",
    "from itertools import groupby\n",
    "import soundfile as sf\n",
    "import IPython.display as ipd\n",
    "\n",
    "from transformers import Wav2Vec2ForCTC\n",
    "from openvino.runtime import Core\n",
    "from openvino.tools.pot import Metric, DataLoader, IEEngine, \\\n",
    "    load_model, save_model, compress_model_weights, create_pipeline\n",
    "\n",
    "sys.path.append(\"../utils\")\n",
    "from notebook_utils import download_file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9e66896-d439-4065-868a-65b44d31525a",
   "metadata": {},
   "source": [
    "## Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "284e9a4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the data and model directories, model source URL and model filename.\n",
    "DATA_DIR = \"../data/datasets/librispeech\"\n",
    "MODEL_DIR = \"model\"\n",
    "\n",
    "os.makedirs(DATA_DIR, exist_ok=True)\n",
    "os.makedirs(MODEL_DIR, exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44dc335d",
   "metadata": {
    "id": "YytHDzLE0uOJ",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Prepare the Model\n",
    "Perform the following:\n",
    "- Download and unpack a pre-trained Wav2Vec2 model.\n",
    "- Convert the model to ONNX.\n",
    "- Run Model Optimizer to convert the model from the ONNX representation to the OpenVINO Intermediate Representation (OpenVINO IR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be9fc64c",
   "metadata": {
    "id": "f7i6dWUmhloy"
   },
   "outputs": [],
   "source": [
    "download_file(\"https://huggingface.co/facebook/wav2vec2-base-960h/resolve/main/pytorch_model.bin\", directory=Path(MODEL_DIR) / 'pytorch', show_progress=True)\n",
    "download_file(\"https://huggingface.co/facebook/wav2vec2-base-960h/resolve/main/config.json\", directory=Path(MODEL_DIR) / 'pytorch', show_progress=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dca2fa0",
   "metadata": {
    "id": "ehX7F6KB0uPu"
   },
   "source": [
    "Import all dependencies to load the original PyTorch model and convert it to the ONNX representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb2f6d66",
   "metadata": {
    "id": "r5as0_Yg0uQX",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "BATCH_SIZE = 1\n",
    "MAX_SEQ_LENGTH = 30480\n",
    "\n",
    "\n",
    "def export_model_to_onnx(model, path):\n",
    "    with torch.no_grad():\n",
    "        default_input = torch.zeros([1, MAX_SEQ_LENGTH], dtype=torch.float)\n",
    "        inputs = {\n",
    "            \"inputs\": default_input\n",
    "        }\n",
    "        symbolic_names = {0: \"batch_size\", 1: \"sequence_len\"}\n",
    "        torch.onnx.export(\n",
    "            model,\n",
    "            (inputs[\"inputs\"]),\n",
    "            path,\n",
    "            opset_version=11,\n",
    "            input_names=[\"inputs\"],\n",
    "            output_names=[\"logits\"],\n",
    "            dynamic_axes={\n",
    "                \"inputs\": symbolic_names,\n",
    "                \"logits\": symbolic_names,\n",
    "            },\n",
    "        )\n",
    "        print(\"ONNX model saved to {}\".format(path))\n",
    "\n",
    "\n",
    "torch_model = Wav2Vec2ForCTC.from_pretrained(Path(MODEL_DIR) / 'pytorch')\n",
    "onnx_model_path = Path(MODEL_DIR) / \"wav2vec2_base.onnx\"\n",
    "if not onnx_model_path.exists():\n",
    "    export_model_to_onnx(torch_model, onnx_model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6911365",
   "metadata": {
    "id": "sNWDAGGd0uRt"
   },
   "source": [
    "## Convert the ONNX Model to OpenVINO IR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20aeea80",
   "metadata": {
    "id": "-6P0c_960uR5",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "ir_model_xml = onnx_model_path.with_suffix(\".xml\")\n",
    "ir_model_bin = onnx_model_path.with_suffix(\".bin\")\n",
    "\n",
    "if not ir_model_xml.exists():\n",
    "    !mo --input_model $onnx_model_path --output_dir $MODEL_DIR --input_shape \"[1,-1]\" --compress_to_fp16"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "635f4b0d",
   "metadata": {
    "id": "LBbY7c4NsHzT"
   },
   "source": [
    "## Prepare LibriSpeech Dataset\n",
    "\n",
    "Use the code below to download and unpack the archive with 'test-clean' subset of LibriSpeech Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43070514",
   "metadata": {
    "id": "NN-qRME1a-Sm"
   },
   "outputs": [],
   "source": [
    "download_file(\"http://openslr.elda.org/resources/12/test-clean.tar.gz\", directory=DATA_DIR, show_progress=True)\n",
    "\n",
    "if not os.path.exists(f'{DATA_DIR}/LibriSpeech'):\n",
    "    with tarfile.open(f\"{DATA_DIR}/test-clean.tar.gz\") as tar:\n",
    "        tar.extractall(path=DATA_DIR)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "da802acd",
   "metadata": {
    "id": "E5hsOsj-0uSc"
   },
   "source": [
    "## Define DataLoader for POT\n",
    "Define `DataLoader` based on POT API, as it will be used to collect statistics for quantization and run model evaluation.\n",
    "Wav2Vec2 model accepts a raw waveform of the speech signal as input and produces vocabulary class estimations as output. Since the dataset contains\n",
    "audio files in FLAC format, use the 'soundfile' package to convert them to waveform.\n",
    "\n",
    "> **NOTE**: Consider increasing `samples_limit` to get more precise results. A suggested value is `300` or more, as it will take longer time to process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2a378cb",
   "metadata": {
    "id": "6xnl2PhM0uSn",
    "tags": [],
    "test_replace": {
     "return len(self._ds)": "return 2",
     "samples_limit = 4": "samples_limit = 2"
    }
   },
   "outputs": [],
   "source": [
    "class LibriSpeechDataLoader(DataLoader):\n",
    "    samples_limit = 4\n",
    "\n",
    "    @staticmethod\n",
    "    def read_flac(file_name):\n",
    "        speech, samplerate = sf.read(file_name)\n",
    "        assert samplerate == 16000, \"read_flac: only 16kHz supported!\"\n",
    "        return speech\n",
    "\n",
    "    # Required methods\n",
    "    def __init__(self, config):\n",
    "        \"\"\"Constructor\n",
    "        :param config: data loader specific config\n",
    "        \"\"\"\n",
    "        super().__init__(config)\n",
    "        self._data_dir = config[\"data_source\"]\n",
    "        self._ds = []\n",
    "        self._prepare_dataset()\n",
    "\n",
    "    def __len__(self):\n",
    "        \"\"\"Returns size of the dataset\"\"\"\n",
    "        return len(self._ds)\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        \"\"\"\n",
    "        Returns annotation, data and metadata at the specified index.\n",
    "        Possible formats:\n",
    "        (index, annotation), data\n",
    "        (index, annotation), data, metadata\n",
    "        \"\"\"\n",
    "        label = self._ds[index][0]\n",
    "        inputs = {'inputs': np.expand_dims(self._ds[index][1], axis=0)}\n",
    "        return label, inputs\n",
    "\n",
    "    # Methods specific to the current implementation\n",
    "    def _prepare_dataset(self):\n",
    "        pattern = re.compile(r'([0-9\\-]+)\\s+(.+)')\n",
    "        data_folder = Path(self._data_dir)\n",
    "        txts = list(data_folder.glob('**/*.txt'))\n",
    "        counter = 0\n",
    "        for txt in txts:\n",
    "            content = txt.open().readlines()\n",
    "            for line in content:\n",
    "                res = pattern.search(line)\n",
    "                if not res:\n",
    "                    continue\n",
    "                name = res.group(1)\n",
    "                transcript = res.group(2)\n",
    "                fname = txt.parent / name\n",
    "                fname = fname.with_suffix('.flac')\n",
    "                identifier = str(fname.relative_to(data_folder))\n",
    "                self._ds.append(((counter, transcript.upper()), LibriSpeechDataLoader.read_flac(os.path.join(self._data_dir, identifier))))\n",
    "                counter += 1\n",
    "                if counter >= self.samples_limit:\n",
    "                    # Limit exceeded\n",
    "                    return"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "71c867d3",
   "metadata": {
    "id": "Re9-YhbBddh3"
   },
   "source": [
    "## Define WER Metric Calculation\n",
    "In this step, the `Metric` interface for WER metric is implemented. It is used for validating the accuracy of the model. `WER` stands for Word Error Rate. For more details, refer to the [Wiki page](https://en.wikipedia.org/wiki/Word_error_rate)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f4e7fbc",
   "metadata": {
    "id": "GB8L492ztZEC"
   },
   "outputs": [],
   "source": [
    "class MetricWER(Metric):\n",
    "    alphabet = [\n",
    "        \"<pad>\", \"<s>\", \"</s>\", \"<unk>\", \"|\",\n",
    "        \"e\", \"t\", \"a\", \"o\", \"n\", \"i\", \"h\", \"s\", \"r\", \"d\", \"l\", \"u\",\n",
    "        \"m\", \"w\", \"c\", \"f\", \"g\", \"y\", \"p\", \"b\", \"v\", \"k\", \"'\", \"x\", \"j\", \"q\", \"z\"]\n",
    "    words_delimiter = '|'\n",
    "    pad_token = '<pad>'\n",
    "\n",
    "    @staticmethod\n",
    "    def decode_logits(logits):\n",
    "        decoding_vocab = dict(enumerate(MetricWER.alphabet))\n",
    "        token_ids = np.squeeze(np.argmax(logits, -1))\n",
    "        tokens = [decoding_vocab[idx] for idx in token_ids]\n",
    "        tokens = [token_group[0] for token_group in groupby(tokens)]\n",
    "        tokens = [t for t in tokens if t != MetricWER.pad_token]\n",
    "        res_string = ''.join([t if t != MetricWER.words_delimiter else ' ' for t in tokens]).strip()\n",
    "        res_string = ' '.join(res_string.split(' '))\n",
    "        res_string = res_string.lower()\n",
    "        return res_string\n",
    "\n",
    "    # Required methods\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self._name = \"WER\"\n",
    "        self._sum_score = 0\n",
    "        self._sum_words = 0\n",
    "        self._cur_score = 0\n",
    "        self._decoding_vocab = dict(enumerate(self.alphabet))\n",
    "\n",
    "    @property\n",
    "    def value(self):\n",
    "        \"\"\"Returns accuracy metric value for the last model output.\"\"\"\n",
    "        return {self._name: self._cur_score}\n",
    "\n",
    "    @property\n",
    "    def avg_value(self):\n",
    "        \"\"\"Returns accuracy metric value for all model outputs.\"\"\"\n",
    "        return {self._name: self._sum_score / self._sum_words if self._sum_words != 0 else 0}\n",
    "\n",
    "    def update(self, output, target):\n",
    "        \"\"\"\n",
    "        Updates prediction matches.\n",
    "\n",
    "        :param output: model output\n",
    "        :param target: annotations\n",
    "        \"\"\"\n",
    "        decoded = [self.decode_logits(i) for i in output]\n",
    "        target = [i.lower() for i in target]\n",
    "        assert len(output) == len(target), \"sizes of output and target mismatch!\"\n",
    "        for i in range(len(output)):\n",
    "            self._get_metric_per_sample(decoded[i], target[i])\n",
    "\n",
    "    def reset(self):\n",
    "        \"\"\"\n",
    "        Resets collected matches\n",
    "        \"\"\"\n",
    "        self._sum_score = 0\n",
    "        self._sum_words = 0\n",
    "\n",
    "    def get_attributes(self):\n",
    "        \"\"\"\n",
    "        Returns a dictionary of metric attributes {metric_name: {attribute_name: value}}.\n",
    "        Required attributes: 'direction': 'higher-better' or 'higher-worse'\n",
    "                             'type': metric type\n",
    "        \"\"\"\n",
    "        return {self._name: {\"direction\": \"higher-worse\", \"type\": \"WER\"}}\n",
    "\n",
    "    # Methods specific to the current implementation\n",
    "    def _get_metric_per_sample(self, annotation, prediction):\n",
    "        cur_score = self._editdistance_eval(annotation.split(), prediction.split())\n",
    "        cur_words = len(annotation.split())\n",
    "\n",
    "        self._sum_score += cur_score\n",
    "        self._sum_words += cur_words\n",
    "        self._cur_score = cur_score / cur_words\n",
    "\n",
    "        result = cur_score / cur_words if cur_words != 0 else 0\n",
    "        return result\n",
    "\n",
    "    def _editdistance_eval(self, source, target):\n",
    "        n, m = len(source), len(target)\n",
    "\n",
    "        distance = np.zeros((n + 1, m + 1), dtype=int)\n",
    "        distance[:, 0] = np.arange(0, n + 1)\n",
    "        distance[0, :] = np.arange(0, m + 1)\n",
    "\n",
    "        for i in range(1, n + 1):\n",
    "            for j in range(1, m + 1):\n",
    "                cost = 0 if source[i - 1] == target[j - 1] else 1\n",
    "\n",
    "                distance[i][j] = min(distance[i - 1][j] + 1,\n",
    "                                     distance[i][j - 1] + 1,\n",
    "                                     distance[i - 1][j - 1] + cost)\n",
    "        return distance[n][m]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9bbbca4a",
   "metadata": {
    "id": "CclWk-fVd9Wi"
   },
   "source": [
    "## Run Quantization Pipeline\n",
    "Use the code below to define a configuration for the quantization pipeline and run it. Keep in mind that built-in `IEEngine` implementation of `Engine` interface from the POT API for model inference is used here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16457bd4",
   "metadata": {
    "id": "PiAvrwo0tr6Z",
    "tags": [],
    "test_replace": {
     "\"stat_subset_size\": 300,": "\"stat_subset_size\": 3,"
    }
   },
   "outputs": [],
   "source": [
    "model_config = {\"model_name\": \"wav2vec2_base\", \"model\": ir_model_xml, \"weights\": ir_model_bin}\n",
    "\n",
    "engine_config = {\"device\": \"CPU\"}\n",
    "\n",
    "dataset_config = {\"data_source\": os.path.join(DATA_DIR, \"LibriSpeech/test-clean\")}\n",
    "\n",
    "algorithms = [\n",
    "    {\n",
    "        \"name\": \"DefaultQuantization\",\n",
    "        \"params\": {\n",
    "            \"target_device\": \"ANY\",\n",
    "            \"model_type\": \"transformer\",\n",
    "            \"preset\": \"performance\",\n",
    "            \"stat_subset_size\": 300,\n",
    "            \"activations\": {\n",
    "                \"range_estimator\": {\n",
    "                    \"min\": {\n",
    "                        \"aggregator\": \"min\",\n",
    "                        \"type\": \"min\"\n",
    "                    },\n",
    "                    \"max\": {\n",
    "                        \"aggregator\": \"mean\",\n",
    "                        \"type\": \"quantile\",\n",
    "                        \"outlier_prob\": 0.0001\n",
    "                    }\n",
    "                }\n",
    "            },\n",
    "            \"ignored\": {\n",
    "                \"scope\": [\"214\"]\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "]\n",
    "\n",
    "# Step 1: Load the model.\n",
    "model = load_model(model_config=model_config)\n",
    "\n",
    "# Step 2: Initialize the data loader.\n",
    "data_loader = LibriSpeechDataLoader(config=dataset_config)\n",
    "\n",
    "# Step 3 (Optional. Required for AccuracyAwareQuantization): Initialize the metric.\n",
    "metric = MetricWER()\n",
    "\n",
    "# Step 4: Initialize the engine for metric calculation and statistics collection.\n",
    "engine = IEEngine(config=engine_config, data_loader=data_loader, metric=metric)\n",
    "\n",
    "# Step 5: Create a pipeline of compression algorithms.\n",
    "pipeline = create_pipeline(algo_config=algorithms, engine=engine)\n",
    "\n",
    "# Step 6 (Optional): Evaluate the original model. Print the results.\n",
    "start_time = time.perf_counter()\n",
    "fp_results = pipeline.evaluate(model=model)\n",
    "end_time = time.perf_counter()\n",
    "print(f\"Evaluation finished in {end_time - start_time:.2f} seconds\")\n",
    "if fp_results:\n",
    "    print(\"FP16 model results:\")\n",
    "    for name, value in fp_results.items():\n",
    "        print(f\"{name}: {value:.5f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a05b3999",
   "metadata": {
    "id": "hPj_fcDAG8xG"
   },
   "outputs": [],
   "source": [
    "# Step 7: Execute the pipeline.\n",
    "print(f\"Quantizing model with {algorithms[0]['params']['preset']} preset and {algorithms[0]['name']}\")\n",
    "start_time = time.perf_counter()\n",
    "compressed_model = pipeline.run(model=model)\n",
    "end_time = time.perf_counter()\n",
    "print(f\"Quantization finished in {end_time - start_time:.2f} seconds\")\n",
    "\n",
    "# Step 8 (Optional): Compress model weights to quantized precision\n",
    "#                    in order to reduce the size of the final .bin file.\n",
    "compress_model_weights(model=compressed_model)\n",
    "\n",
    "# Step 9: Save the compressed model to the desired path.\n",
    "compressed_model_paths = save_model(model=compressed_model, save_path=MODEL_DIR, model_name=\"quantized_wav2vec2_base\")\n",
    "compressed_model_xml = compressed_model_paths[0][\"model\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b05dc0de",
   "metadata": {
    "id": "hPj_fcDAG8xG"
   },
   "outputs": [],
   "source": [
    "# Step 10 (Optional): Evaluate the compressed model and print the results.\n",
    "int_results = pipeline.evaluate(model=compressed_model)\n",
    "\n",
    "if int_results:\n",
    "    print(\"INT8 model results:\")\n",
    "    for name, value in int_results.items():\n",
    "        print(f\"{name}: {value:.5f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "754b4d84",
   "metadata": {},
   "source": [
    "## Model Usage Example with Inference Pipeline\n",
    "Both initial (`FP16`) and quantized (`INT8`) models are exactly the same in use.\n",
    "\n",
    "Start with taking one example from the dataset to show inference steps for it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17bd5485",
   "metadata": {},
   "outputs": [],
   "source": [
    "audio = LibriSpeechDataLoader.read_flac(f'{DATA_DIR}/LibriSpeech/test-clean/121/127105/121-127105-0017.flac')\n",
    "\n",
    "ipd.Audio(audio, rate=16000)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7538271f",
   "metadata": {},
   "source": [
    "Next, load the quantized model to the inference pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0431ac4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "ie = Core()\n",
    "\n",
    "model = ie.read_model(compressed_model_xml)\n",
    "compiled_model = ie.compile_model(model=model, device_name='CPU')\n",
    "\n",
    "input_data = np.expand_dims(audio, axis=0)\n",
    "output_layer = compiled_model.outputs[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d8274a6",
   "metadata": {},
   "source": [
    "Next, make a prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4fc43396",
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = compiled_model([input_data])[output_layer]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca4095f9",
   "metadata": {},
   "source": [
    "Now, you just need to decode predicted probabilities to text, using tokenizer from `MetricWER` class.\n",
    "\n",
    "Alternatively, use a built-in `Wav2Vec2Processor` tokenizer  from the `transformers` package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "516f6acc",
   "metadata": {},
   "outputs": [],
   "source": [
    "predicted_text = MetricWER.decode_logits(predictions)\n",
    "\n",
    "predicted_text"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4f67f6a2",
   "metadata": {
    "id": "vQACMfAUo52V",
    "tags": []
   },
   "source": [
    "## Compare Performance of the Original and Quantized Models\n",
    "Finally, use [Benchmark Tool](https://docs.openvino.ai/latest/openvino_inference_engine_tools_benchmark_tool_README.html) to measure the inference performance of the `FP16` and `INT8` models.\n",
    "\n",
    "> **NOTE**: For more accurate performance, it is recommended to run `benchmark_app` in a terminal/command prompt after closing other applications. Run `benchmark_app -m model.xml -d CPU` to benchmark async inference on CPU for one minute. Change `CPU` to `GPU` to benchmark on GPU. Run `benchmark_app --help` to see an overview of all command-line options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d732360",
   "metadata": {
    "tags": [],
    "test_replace": {
     "benchmark_app": "benchmark-app -t 5"
    }
   },
   "outputs": [],
   "source": [
    "# Inference FP16 model (OpenVINO IR)\n",
    "! benchmark_app -m $ir_model_xml -shape [1,30480] -d CPU -api async"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fdf41525",
   "metadata": {
    "tags": [],
    "test_replace": {
     "benchmark_app": "benchmark-app -t 5"
    }
   },
   "outputs": [],
   "source": [
    "# Inference INT8 model (OpenVINO IR)\n",
    "! benchmark_app -m $compressed_model_xml -shape [1,30480] -d CPU -api async"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}