Lecture10 - OpenVINO Real Time Inference(MobileNet).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='https://www.anadronestarting.com/wp-content/uploads/intel-main_opt.png' width=50%>\n",
    "\n",
    "# 모바일넷을 이용한 이미지분류\n",
    "<font size=5><b>(Image Classification using Mobilenet)<b></font>\n",
    "\n",
    "<div align='right'>성  민  석<br>(Minsuk Sung)</div>\n",
    "\n",
    "<img src='https://gitcdn.xyz/cdn/Tony607/blog_statics/9ddd48eb623175fb734e63865ebe604351c64c61/images/openvino/openvino.png' width=50%>\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>강의목차<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#OpenVINO를-통해-실시간-추론\" data-toc-modified-id=\"OpenVINO를-통해-실시간-추론-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>OpenVINO를 통해 실시간 추론</a></span></li><li><span><a href=\"#Reference\" data-toc-modified-id=\"Reference-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Reference</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## OpenVINO를 통해 실시간 추론"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "# assert 'computer_vision_sdk' in os.environ['PYTHONPATH']\n",
    "\n",
    "from PIL import Image\n",
    "import numpy as np\n",
    "\n",
    "try:\n",
    "    from openvino import inference_engine as ie\n",
    "    from openvino.inference_engine import IENetwork, IEPlugin\n",
    "except Exception as e:\n",
    "    exception_type = type(e).__name__\n",
    "    print(\"The following error happened while importing Python API module:\\n[ {} ] {}\".format(exception_type, e))\n",
    "    sys.exit(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pre_process_image(image, img_height=224):\n",
    "    # Model input format\n",
    "    n, c, h, w = [1, 3, img_height, img_height]\n",
    "    processedImg = cv2.resize(image,(h,w),interpolation=cv2.INTER_AREA)\n",
    "\n",
    "    # Normalize to keep data between 0 - 1\n",
    "    processedImg = (np.array(processedImg) - 0) / 255.0\n",
    "\n",
    "    # Change data layout from HWC to CHW\n",
    "    processedImg = processedImg.transpose((2, 0, 1))\n",
    "    processedImg = processedImg.reshape((n, c, h, w))\n",
    "\n",
    "    return image, processedImg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plugin initialization for specified device and load extensions library if specified.\n",
    "plugin_dir = None\n",
    "model_xml = './bin/mobilenetv2.xml'\n",
    "model_bin = './bin/mobilenetv2.bin'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Devices: GPU (intel), CPU, MYRIAD\n",
    "# plugin = IEPlugin(\"MYRIAD\", plugin_dirs=plugi)\n",
    "plugin = IEPlugin(\"CPU\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/intel/imobilenet/myvenv/lib/python3.5/site-packages/ipykernel_launcher.py:2: DeprecationWarning: from_ir() method of IENetwork is deprecated. Please use IENetwork class constructor to create valid IENetwork instance\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "# Read IRn_dir\n",
    "net = IENetwork.from_ir(model=model_xml, weights=model_bin)\n",
    "assert len(net.inputs.keys()) == 1\n",
    "assert len(net.outputs) == 1\n",
    "input_blob = next(iter(net.inputs))\n",
    "out_blob = next(iter(net.outputs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load network to the plugin\n",
    "exec_net = plugin.load(network=net)\n",
    "del net"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "# Custom Labels\n",
    "# with open('./dropthebee.txt', 'r') as f:\n",
    "#     labels = f.readlines()\n",
    "    \n",
    "# labels = list(map(lambda x: x.replace('\\n',''),labels))\n",
    "\n",
    "with open('./bin/class20.pickle', 'rb') as f:\n",
    "    labels = pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/intel/imobilenet/myvenv/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/home/intel/imobilenet/myvenv/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/home/intel/imobilenet/myvenv/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/home/intel/imobilenet/myvenv/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/home/intel/imobilenet/myvenv/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/home/intel/imobilenet/myvenv/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n",
      "/home/intel/imobilenet/myvenv/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/home/intel/imobilenet/myvenv/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/home/intel/imobilenet/myvenv/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/home/intel/imobilenet/myvenv/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/home/intel/imobilenet/myvenv/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/home/intel/imobilenet/myvenv/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n"
     ]
    }
   ],
   "source": [
    "import cv2\n",
    "import matplotlib.pyplot as plt\n",
    "from tensorflow.keras.preprocessing.image import load_img\n",
    "\n",
    "capture = cv2.VideoCapture(0)\n",
    "capture.set(cv2.CAP_PROP_FRAME_WIDTH, 640)\n",
    "capture.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)\n",
    "\n",
    "while True:\n",
    "    ret, frame = capture.read()\n",
    "\n",
    "    image, processedImg = pre_process_image(frame)\n",
    "    res = exec_net.infer(inputs={input_blob: processedImg})\n",
    "    # Access the results and get the index of the highest confidence score\n",
    "    output_node_name = list(res.keys())[0]\n",
    "    res = res[output_node_name]\n",
    "\n",
    "    # Predicted class index.\n",
    "    idx = np.argsort(res[0])[-1]\n",
    "\n",
    "    prob = res[0][idx]*100\n",
    "\n",
    "    info = 'Predicted : None'\n",
    "    color = (0, 0, 0)\n",
    "\n",
    "    if prob > 60:\n",
    "        info = 'Predicted : {} ({:.2f}%)'.format(labels[idx].upper(), prob)\n",
    "        color = (0, 0, 255)\n",
    "\n",
    "    cv2.putText(frame, info, (10, 30), 0, 1, color, 2)\n",
    "\n",
    "    cv2.imshow(\"VideoFrame\", frame)\n",
    "\n",
    "    if cv2.waitKey(1) > 0:\n",
    "        break\n",
    "\n",
    "capture.release()\n",
    "cv2.destroyAllWindows()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Intel\n",
    "    - https://www.intel.co.kr/\n",
    "- Intel OpenVINO\n",
    "    - https://software.intel.com/en-us/openvino-toolkit\n",
    "- MNIST\n",
    "    - http://yann.lecun.com/exdb/mnist/\n",
    "- CIFAR10\n",
    "    - https://www.cs.toronto.edu/~kriz/cifar.html\n",
    "- ImageNet\n",
    "    - http://www.image-net.org\n",
    "- Tensorflow\n",
    "    - https://www.tensorflow.org/?hl=ko\n",
    "- Keras\n",
    "    - https://keras.io/\n",
    "    - https://tensorflow.blog/2019/03/06/tensorflow-2-0-keras-api-overview/\n",
    "    - https://tykimos.github.io/2017/02/22/Integrating_Keras_and_TensorFlow/\n",
    "    - https://tykimos.github.io/2017/03/08/CNN_Getting_Started/\n",
    "    - https://raw.githubusercontent.com/keras-team/keras-docs-ko/master/sources/why-use-keras.md\n",
    "- Keras to Caffe\n",
    "     - https://github.com/uhfband/keras2caffe\n",
    "     - http://www.deepvisionconsulting.com/from-keras-to-caffe/\n",
    "- Fully Connected Layer\n",
    "    - https://sonofgodcom.wordpress.com/2018/12/31/cnn%EC%9D%84-%EC%9D%B4%ED%95%B4%ED%95%B4%EB%B3%B4%EC%9E%90-fully-connected-layer%EB%8A%94-%EB%AD%94%EA%B0%80/\n",
    "- Convultional Nueral Network\n",
    "    - http://aikorea.org/cs231n/convolutional-networks/\n",
    "    - http://cs231n.stanford.edu/\n",
    "- CNN Models\n",
    "    - https://ratsgo.github.io/deep%20learning/2017/10/09/CNNs/\n",
    "\n",
    "- VOC2012\n",
    "    - https://blog.godatadriven.com/rod-keras-multi-label\n",
    "    - https://gist.github.com/rragundez/ae3a17428bfec631d1b35dcdc6296a85#file-multi-label_classification_with_keras_imagedatagenerator-ipynbhttps://fairyonice.github.io/Part_5_Object_Detection_with_Yolo_using_VOC_2012_data_training.html\n",
    "    - http://research.sualab.com/introduction/2017/11/29/image-recognition-overview-1.html"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "강의목차",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}