notebooks/301-tensorflow-training-openvino/301-tensorflow-training-openvino-pot.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KwQtSOz0VrVX",
    "tags": []
   },
   "source": [
    "# Post-Training Quantization with TensorFlow Classification Model\n",
    "\n",
    "This example demonstrates how to quantize the OpenVINO model that was created in [301-tensorflow-training-openvino.ipynb](301-tensorflow-training-openvino.ipynb), to improve inference speed. Quantization is performed with [Post-Training Optimization Tool (POT)](https://docs.openvino.ai/nightly/pot_README.html). A custom dataloader and metric will be defined, and accuracy and performance will be computed for the original IR model and the quantized model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KwQtSOz0VrVX"
   },
   "source": [
    "## Preparation\n",
    "\n",
    "The notebook requires that the training notebook has been run and that the Intermediate Representation (IR) models are created. If the IR models do not exist, running the next cell will run the training notebook. This will take a while."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "hide_output"
    ],
    "test_replace": {
     "301-tensorflow-training-openvino.ipynb": "test_301-tensorflow-training-openvino.ipynb"
    }
   },
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "model_xml = Path(\"model/flower/flower_ir.xml\")\n",
    "dataset_url = (\n",
    "    \"https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz\"\n",
    ")\n",
    "data_dir = Path(tf.keras.utils.get_file(\"flower_photos\", origin=dataset_url, untar=True))\n",
    "\n",
    "if not model_xml.exists():\n",
    "    print(\"Executing training notebook. This will take a while...\")\n",
    "    %run 301-tensorflow-training-openvino.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xG4gTnxHXn3z"
   },
   "source": [
    "### Imports\n",
    "\n",
    "The Post Training Optimization API is implemented in the `compression` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Wi2JvOs1Xn3z",
    "tags": []
   },
   "outputs": [],
   "source": [
    "import copy\n",
    "import os\n",
    "import sys\n",
    "\n",
    "import cv2\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from addict import Dict\n",
    "from openvino.tools.pot.api import Metric, DataLoader\n",
    "from openvino.tools.pot.graph import load_model, save_model\n",
    "from openvino.tools.pot.graph.model_utils import compress_model_weights\n",
    "from openvino.tools.pot.engines.ie_engine import IEEngine\n",
    "from openvino.tools.pot.pipeline.initializer import create_pipeline\n",
    "from openvino.runtime import Core\n",
    "from PIL import Image\n",
    "\n",
    "sys.path.append(\"../utils\")\n",
    "from notebook_utils import benchmark_model, download_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7M6TsJjlXn30"
   },
   "source": [
    "### Settings\n",
    "\n",
    "In the next cell, the settings for running quantization are defined. The default settings use the _mixed_ preset and the _DefaultQuantization_ algorithm. This enables reasonably fast quantization, with possible drop in accuracy. The _performance_ preset can result in faster inference on the quantized model, the _AccuracyAwareQuantization_ algorithm quantizes the model to a defined maximal accuracy drop, which may not achieve the greatest performance boost but avoids further drop in accuracy.  \n",
    "\n",
    "See the [Post-Training Optimization Best Practices](https://docs.openvino.ai/latest/pot_docs_BestPractices.html) page for more information about the configurable parameters and best practices for post-training quantization.\n",
    "\n",
    "The POT methods expect configuration dictionaries as arguments. They are defined in the cell below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Lb_BnWSpXn30",
    "tags": [],
    "test_replace": {
     "\"stat_subset_size\": 1000,": "\"stat_subset_size\": 25,"
    }
   },
   "outputs": [],
   "source": [
    "model_config = Dict(\n",
    "    {\n",
    "        \"model_name\": \"flower\",\n",
    "        \"model\": \"model/flower/flower_ir.xml\",\n",
    "        \"weights\": \"model/flower/flower_ir.bin\",\n",
    "    }\n",
    ")\n",
    "\n",
    "engine_config = Dict({\"device\": \"CPU\", \"stat_requests_number\": 2, \"eval_requests_number\": 2})\n",
    "\n",
    "algorithms = [\n",
    "    {\n",
    "        \"name\": \"DefaultQuantization\",\n",
    "        \"params\": {\n",
    "            \"target_device\": \"CPU\",\n",
    "            \"preset\": \"performance\",\n",
    "            \"stat_subset_size\": 1000,\n",
    "        },\n",
    "    }\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IVZ7GGu2Xn3z"
   },
   "source": [
    "### Create DataLoader Class\n",
    "\n",
    "OpenVINO's compression library contains a DataLoader class. The DataLoader defines how to load data and annotations. For the TensorFlow flowers dataset, images are stored in a directory per category. The DataLoader loads images from a given _data_source_ directory and assigns a label based on the position of the directory in _class_names_ (where class_names is a list of directory names in alphabetical order)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "122wcKhzXn3z",
    "tags": [],
    "test_replace": {
     "return len(self.dataset)": "return 30"
    }
   },
   "outputs": [],
   "source": [
    "class ClassificationDataLoader(DataLoader):\n",
    "    \"\"\"\n",
    "    DataLoader for image data that is stored in a directory per category. For example, for\n",
    "    categories _rose_ and _daisy_, rose images are expected in data_source/rose, daisy images\n",
    "    in data_source/daisy.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, data_source):\n",
    "        \"\"\"\n",
    "        :param data_source: path to data directory\n",
    "        \"\"\"\n",
    "        self.data_source = Path(data_source)\n",
    "        self.dataset = [p for p in data_dir.glob(\"**/*\") if p.suffix in (\".png\", \".jpg\")]\n",
    "        self.class_names = sorted([item.name for item in Path(data_dir).iterdir() if item.is_dir()])\n",
    "\n",
    "    def __len__(self):\n",
    "        \"\"\"\n",
    "        Returns the number of elements in the dataset\n",
    "        \"\"\"\n",
    "        return len(self.dataset)\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        \"\"\"\n",
    "        Get item from self.dataset at the specified index.\n",
    "        Returns (annotation, image), where annotation is a tuple (index, class_index)\n",
    "        and image a preprocessed image in network shape\n",
    "        \"\"\"\n",
    "        if index >= len(self):\n",
    "            raise IndexError\n",
    "        filepath = self.dataset[index]\n",
    "        annotation = (index, self.class_names.index(filepath.parent.name))\n",
    "        image = self._read_image(filepath)\n",
    "        return annotation, image\n",
    "\n",
    "    def _read_image(self, index):\n",
    "        \"\"\"\n",
    "        Read image at dataset[index] to memory, resize, convert to BGR and to network shape\n",
    "\n",
    "        :param index: dataset index to read\n",
    "        :return ndarray representation of image batch\n",
    "        \"\"\"\n",
    "        image = cv2.imread(os.path.join(self.data_source, index))[:, :, (2, 1, 0)]\n",
    "        image = cv2.resize(image, (180, 180)).astype(np.float32)\n",
    "        return image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_eREaJb5Xn3z"
   },
   "source": [
    "### Create Accuracy Metric Class\n",
    "\n",
    "The accuracy metric is defined as the number of correct predictions divided by the total number of predictions. It is used to validate the accuracy of the quantized model.\n",
    "\n",
    "The Accuracy class in this tutorial implements the `Metric` interface of the compression library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4fBKPFMzXn30"
   },
   "outputs": [],
   "source": [
    "class Accuracy(Metric):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self._name = \"accuracy\"\n",
    "        self._matches = []\n",
    "\n",
    "    @property\n",
    "    def value(self):\n",
    "        \"\"\"Returns accuracy metric value for the last model output.\"\"\"\n",
    "        return {self._name: self._matches[-1]}\n",
    "\n",
    "    @property\n",
    "    def avg_value(self):\n",
    "        \"\"\"\n",
    "        Returns accuracy metric value for all model outputs. Results per image are stored in\n",
    "        self._matches, where True means a correct prediction and False a wrong prediction.\n",
    "        Accuracy is computed as the number of correct predictions divided by the total\n",
    "        number of predictions.\n",
    "        \"\"\"\n",
    "        num_correct = np.count_nonzero(self._matches)\n",
    "        return {self._name: num_correct / len(self._matches)}\n",
    "\n",
    "    def update(self, output, target):\n",
    "        \"\"\"Updates prediction matches.\n",
    "\n",
    "        :param output: model output\n",
    "        :param target: annotations\n",
    "        \"\"\"\n",
    "        predict = np.argmax(output[0], axis=1)\n",
    "        match = predict == target\n",
    "        self._matches.append(match)\n",
    "\n",
    "    def reset(self):\n",
    "        \"\"\"\n",
    "        Resets the Accuracy metric. This is a required method that should initialize all\n",
    "        attributes to their initial value.\n",
    "        \"\"\"\n",
    "        self._matches = []\n",
    "\n",
    "    def get_attributes(self):\n",
    "        \"\"\"\n",
    "        Returns a dictionary of metric attributes {metric_name: {attribute_name: value}}.\n",
    "        Required attributes: 'direction': 'higher-better' or 'higher-worse'\n",
    "                             'type': metric type\n",
    "        \"\"\"\n",
    "        return {self._name: {\"direction\": \"higher-better\", \"type\": \"accuracy\"}}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mgaRN7mTXn30"
   },
   "source": [
    "## POT Optimization\n",
    "\n",
    "After creating the DataLoader and Metric classes, and defining the configuration settings for POT, we can start the quantization process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RjywhxvqXn30"
   },
   "outputs": [],
   "source": [
    "# Step 1: Load the model\n",
    "model = load_model(model_config=model_config)\n",
    "original_model = copy.deepcopy(model)\n",
    "\n",
    "# Step 2: Initialize the data loader\n",
    "data_loader = ClassificationDataLoader(data_source=data_dir)\n",
    "\n",
    "# Step 3 (Optional. Required for AccuracyAwareQuantization): Initialize the metric\n",
    "#        Compute metric results on original model\n",
    "metric = Accuracy()\n",
    "\n",
    "# Step 4: Initialize the engine for metric calculation and statistics collection\n",
    "engine = IEEngine(config=engine_config, data_loader=data_loader, metric=metric)\n",
    "\n",
    "# Step 5: Create a pipeline of compression algorithms\n",
    "pipeline = create_pipeline(algo_config=algorithms, engine=engine)\n",
    "\n",
    "# Step 6: Execute the pipeline\n",
    "compressed_model = pipeline.run(model=model)\n",
    "\n",
    "# Step 7 (Optional): Compress model weights quantized precision\n",
    "#                    in order to reduce the size of final .bin file\n",
    "compress_model_weights(model=compressed_model)\n",
    "\n",
    "# Step 8: Save the compressed model and get the path to the model\n",
    "compressed_model_paths = save_model(\n",
    "    model=compressed_model, save_path=os.path.join(os.path.curdir, \"model/optimized\")\n",
    ")\n",
    "compressed_model_xml = Path(compressed_model_paths[0][\"model\"])\n",
    "print(f\"The quantized model is stored in {compressed_model_xml}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RjywhxvqXn30"
   },
   "outputs": [],
   "source": [
    "# Step 9 (Optional): Evaluate the original and compressed model. Print the results\n",
    "original_metric_results = pipeline.evaluate(original_model)\n",
    "if original_metric_results:\n",
    "    print(f\"Accuracy of the original model:  {next(iter(original_metric_results.values())):.5f}\")\n",
    "\n",
    "quantized_metric_results = pipeline.evaluate(compressed_model)\n",
    "if quantized_metric_results:\n",
    "    print(f\"Accuracy of the quantized model: {next(iter(quantized_metric_results.values())):.5f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DJbP4pdEXn30"
   },
   "source": [
    "## Run Inference on Quantized Model\n",
    "\n",
    "Copy the preprocess function from the training notebook and run inference on the quantized model with Inference Engine. See the [OpenVINO API tutorial](../002-openvino-api/002-openvino-api.ipynb) for more information about running inference with Inference Engine Python API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pre_process_image(imagePath, img_height=180):\n",
    "    # Model input format\n",
    "    n, c, h, w = [1, 3, img_height, img_height]\n",
    "    image = Image.open(imagePath)\n",
    "    image = image.resize((h, w), resample=Image.BILINEAR)\n",
    "\n",
    "    # Convert to array and change data layout from HWC to CHW\n",
    "    image = np.array(image)\n",
    "\n",
    "    input_image = image.reshape((n, h, w, c))\n",
    "\n",
    "    return input_image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "q8lGxI-1Xn31"
   },
   "outputs": [],
   "source": [
    "# Load the optimized model and get the names of the input and output layer\n",
    "ie = Core()\n",
    "model_pot = ie.read_model(model=\"model/optimized/flower_ir.xml\")\n",
    "compiled_model_pot = ie.compile_model(model=model_pot, device_name=\"CPU\")\n",
    "input_layer = compiled_model_pot.input(0)\n",
    "output_layer = compiled_model_pot.output(0)\n",
    "\n",
    "# Get the class names: a list of directory names in alphabetical order\n",
    "class_names = sorted([item.name for item in Path(data_dir).iterdir() if item.is_dir()])\n",
    "\n",
    "# Run inference on an input image...\n",
    "inp_img_url = (\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/4/48/A_Close_Up_Photo_of_a_Dandelion.jpg\"\n",
    ")\n",
    "directory = \"output\"\n",
    "inp_file_name = \"A_Close_Up_Photo_of_a_Dandelion.jpg\"\n",
    "file_path = Path(directory)/Path(inp_file_name)\n",
    "# Download the image if it does not exist yet\n",
    "if not Path(inp_file_name).exists():\n",
    "    download_file(inp_img_url, inp_file_name, directory=directory)\n",
    "\n",
    "# Pre-process the image and get it ready for inference.\n",
    "input_image = pre_process_image(imagePath=file_path)\n",
    "print(f'input image shape: {input_image.shape}')\n",
    "print(f'input layer shape: {input_layer.shape}')\n",
    "\n",
    "res = compiled_model_pot([input_image])[output_layer]\n",
    "\n",
    "score = tf.nn.softmax(res[0])\n",
    "\n",
    "# Show the results\n",
    "image = Image.open(file_path)\n",
    "plt.imshow(image)\n",
    "print(\n",
    "    \"This image most likely belongs to {} with a {:.2f} percent confidence.\".format(\n",
    "        class_names[np.argmax(score)], 100 * np.max(score)\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compare Inference Speed\n",
    "\n",
    "Measure inference speed with the [OpenVINO Benchmark App](https://docs.openvino.ai/latest/openvino_inference_engine_tools_benchmark_tool_README.html). \n",
    "\n",
    "Benchmark App is a command line tool that measures raw inference performance for a specified OpenVINO IR model. Run `benchmark_app --help` to see a list of available parameters. By default, Benchmark App tests the performance of the model specified with the `-m` parameter with asynchronous inference on CPU, for one minute. Use the `-d` parameter to test performance on a different device, for example an Intel integrated Graphics (iGPU), and `-t` to set the number of seconds to run inference. See the [documentation](https://docs.openvino.ai/latest/openvino_inference_engine_tools_benchmark_tool_README.html) for more information. \n",
    "\n",
    "In this tutorial, we use a wrapper function from [Notebook Utils](https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/utils/notebook_utils.ipynb). It prints the `benchmark_app` command with the chosen parameters.\n",
    "\n",
    "In the next cells, inference speed will be measured for the original and quantized model on CPU. If an iGPU is available, inference speed will be measured for CPU+GPU as well. The number of seconds is set to 15.\n",
    "\n",
    "> NOTE: For the most accurate performance estimation, we recommended running `benchmark_app` in a terminal/command prompt after closing other applications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print the available devices on this system\n",
    "ie = Core()\n",
    "print(\"Device information:\")\n",
    "print(ie.get_property(\"CPU\", \"FULL_DEVICE_NAME\"))\n",
    "if \"GPU\" in ie.available_devices:\n",
    "    print(ie.get_property(\"GPU\", \"FULL_DEVICE_NAME\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-0eZaw1W5-wt",
    "tags": [],
    "test_replace": {
     "seconds=15": "seconds=3"
    }
   },
   "outputs": [],
   "source": [
    "# Original model - CPU\n",
    "benchmark_model(model_path=model_xml, device=\"CPU\", seconds=15, api='async')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-0eZaw1W5-wt",
    "tags": [],
    "test_replace": {
     "seconds=15": "seconds=3"
    }
   },
   "outputs": [],
   "source": [
    "# Quantized model - CPU\n",
    "benchmark_model(model_path=compressed_model_xml, device=\"CPU\", seconds=15, api='async')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-0eZaw1W5-wt"
   },
   "source": [
    "**Benchmark on MULTI:CPU,GPU**\n",
    "\n",
    "With a recent Intel CPU, the best performance can often be achieved by doing inference on both the CPU and the iGPU, with OpenVINO's [Multi Device Plugin](https://docs.openvino.ai/2021.4/openvino_docs_IE_DG_supported_plugins_MULTI.html). It takes a bit longer to load a model on GPU than on CPU, so this benchmark will take a bit longer to complete than the CPU benchmark, when run for the first time. Benchmark App supports caching, by specifying the `--cdir` parameter. In the cells below, the model will cached to the `model_cache` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-0eZaw1W5-wt",
    "tags": [],
    "test_replace": {
     "seconds=15": "seconds=3"
    }
   },
   "outputs": [],
   "source": [
    "# Original model - MULTI:CPU,GPU\n",
    "if \"GPU\" in ie.available_devices:\n",
    "    benchmark_model(model_path=model_xml, device=\"MULTI:CPU,GPU\", seconds=15, api='async')\n",
    "else:\n",
    "    print(\"A supported integrated GPU is not available on this system.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-0eZaw1W5-wt",
    "tags": [],
    "test_replace": {
     "seconds=15": "seconds=3"
    }
   },
   "outputs": [],
   "source": [
    "# Quantized model - MULTI:CPU,GPU\n",
    "if \"GPU\" in ie.available_devices:\n",
    "    benchmark_model(model_path=compressed_model_xml, device=\"MULTI:CPU,GPU\", seconds=15, api='async')\n",
    "else:\n",
    "    print(\"A supported integrated GPU is not available on this system.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print the available devices on this system\n",
    "print(\"Device information:\")\n",
    "print(ie.get_property(\"CPU\", \"FULL_DEVICE_NAME\"))\n",
    "if \"GPU\" in ie.available_devices:\n",
    "    print(ie.get_property(\"GPU\", \"FULL_DEVICE_NAME\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-0eZaw1W5-wt"
   },
   "source": [
    "**Original IR model - CPU**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-0eZaw1W5-wt",
    "tags": [],
    "test_replace": {
     "-t 15": "-t 3"
    }
   },
   "outputs": [],
   "source": [
    "benchmark_output = %sx benchmark_app -m $model_xml -t 15 -api async\n",
    "# Remove logging info from benchmark_app output and show only the results\n",
    "benchmark_result = [line for line in benchmark_output if not (line.startswith(r\"[\") or line.startswith(\"  \") or line==\"\")]\n",
    "print(\"\\n\".join(benchmark_result))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-0eZaw1W5-wt"
   },
   "source": [
    "**Quantized IR model - CPU**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-0eZaw1W5-wt",
    "tags": [],
    "test_replace": {
     "-t 15": "-t 3"
    }
   },
   "outputs": [],
   "source": [
    "benchmark_output = %sx benchmark_app -m $compressed_model_xml -t 15 -api async\n",
    "# Remove logging info from benchmark_app output and show only the results\n",
    "benchmark_result = [line for line in benchmark_output if not (line.startswith(r\"[\") or line.startswith(\"  \") or line==\"\")]\n",
    "print(\"\\n\".join(benchmark_result))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-0eZaw1W5-wt"
   },
   "source": [
    "**Original IR model - MULTI:CPU,GPU**\n",
    "\n",
    "With a recent Intel CPU, the best performance can often be achieved by doing inference on both the CPU and the iGPU, with OpenVINO's [Multi Device Plugin](https://docs.openvino.ai/latest/openvino_docs_OV_UG_Running_on_multiple_devices.html). It takes a bit longer to load a model on GPU than on CPU, so this benchmark will take a bit longer to complete than the CPU benchmark."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-0eZaw1W5-wt",
    "tags": [],
    "test_replace": {
     "-t 15": "-t 3"
    }
   },
   "outputs": [],
   "source": [
    "ie = Core()\n",
    "if \"GPU\" in ie.available_devices:\n",
    "    benchmark_output = %sx benchmark_app -m $model_xml -d MULTI:CPU,GPU -t 15 -api async\n",
    "    # Remove logging info from benchmark_app output and show only the results\n",
    "    benchmark_result = [line for line in benchmark_output if not (line.startswith(r\"[\") or line.startswith(\"  \") or line==\"\")]\n",
    "    print(\"\\n\".join(benchmark_result))\n",
    "else:\n",
    "    print(\"An integrated GPU is not available on this system.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-0eZaw1W5-wt"
   },
   "source": [
    "**Quantized IR model - MULTI:CPU,GPU**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-0eZaw1W5-wt",
    "tags": [],
    "test_replace": {
     "-t 15": "-t 3"
    }
   },
   "outputs": [],
   "source": [
    "ie = Core()\n",
    "if \"GPU\" in ie.available_devices:\n",
    "    benchmark_output = %sx benchmark_app -m $compressed_model_xml -d MULTI:CPU,GPU -t 15 -api async\n",
    "    # Remove logging info from benchmark_app output and show only the results\n",
    "    benchmark_result = [line for line in benchmark_output if not (line.startswith(r\"[\") or line.startswith(\"  \") or line==\"\")]\n",
    "    print(\"\\n\".join(benchmark_result))\n",
    "else:\n",
    "    print(\"An integrated GPU is not available on this system.\")"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "301-tensorflow-training-openvino-pot.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "interpreter": {
   "hash": "8e25c8ed6cc2cfe6c8620be5042bb64fac4c236f57496fb5eb68e9ea1795f1fe"
  },
  "kernelspec": {
   "display_name": "openvino_env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}