ShapeDetection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup environment\n",
    "\n",
    "Start by generating the protobuf sources and adding the required directories to Python PATH."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "cd  models/research\n",
    "# Generate .proto messages\n",
    "protoc object_detection/protos/*.proto --python_out=."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: PYTHONPATH=$PYTHONPATH:./models/research:./models/research/slim\n"
     ]
    }
   ],
   "source": [
    "# Add both research and research/slim to python path\n",
    "%env PYTHONPATH=$PYTHONPATH:./models/research:./models/research/slim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Real dataset\n",
    "\n",
    "Our dataset of real images is hosted in our lab's [webpage][vislab].\n",
    "We provide a convenient script for downloading it to your workspace.\n",
    "Running it should copy the set raw images and annotations to `workspace/data/raw` and the processed binaries to `workspace/data`. \n",
    "\n",
    "[vislab]: http://vislab.isr.ist.utl.pt/datasets/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sucessfully downloaded dataset!\n"
     ]
    }
   ],
   "source": [
    "! bash download_real_dataset.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Synthetic dataset\n",
    "\n",
    "You can generate you own synthetic datasets using [our Domain Randomization plugin for Gazebo][gap].\n",
    "We provide documentation on how to do so [here][TODO].\n",
    "After this step, the dataset should be split into `images` and `annotations` folder.\n",
    "The images themselves are split in folders each containing 100 Full-HD `.jpg` files.\n",
    "\n",
    "[gap]: https://github.com/jsbruglie/gap\n",
    "[TODO]: A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pre-processing\n",
    "\n",
    "The synthetic dataset has to undergo some processing, such as rescaling the images to 960x540 and condensing the `.XML` annotations into a single `.CSV` file.\n",
    "Assuming you have your synthetic dataset in `./workspace/data/raw` and want the output top be stored in `./workspace/data/proc` run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! python preproc.py \\\n",
    "    --img_ext jpg \\\n",
    "    --in_xml_dir workspace/data/raw/annotations \\\n",
    "    --in_img_dir workspace/data/raw/images \\\n",
    "    --out_csv workspace/data/proc/annotations.csv \\\n",
    "    --out_img_dir workspace/data/proc/images_resize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate TF records\n",
    "\n",
    "Both real and sytnthetic datasets must be converted to TF records so they can be used by the network.\n",
    "For each processed subdataset run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! python generate_tfrecord.py \\\n",
    "    --data_path workspace/data/ \\\n",
    "    --filename data \\\n",
    "    --csv_path workspace/data/proc/annotations.csv \\\n",
    "    --images_path workspace/data/proc/images_resize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For batch processing of several subdatasets we ran:\n",
    "\n",
    "```bash\n",
    "./run_preproc.sh woskspace/data/raw workspace/data/proc\n",
    "./generate_tf_records.sh workspace/data/proc\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download pre-trained SSD\n",
    "\n",
    "We have used Single Shot Detector [SSD][ssd] using [MobileNet][mobilenet] as the feature extractor (which was trained using [ImageNet][imagenet]) and pre-trained on [COCO][coco] dataset as our baseline.\n",
    "You can download and unpack a snapshot of the network in the form of TF `checkpoint` as follows:\n",
    "\n",
    "[coco]: http://cocodataset.org/#home\n",
    "[mobilenet]: https://arxiv.org/abs/1704.04861\n",
    "[imagenet]: https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf\n",
    "[ssd]: http://www.cs.unc.edu/%7Ewliu/papers/ssd.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ssd_mobilenet_v1_coco_11_06_2017/\n",
      "ssd_mobilenet_v1_coco_11_06_2017/model.ckpt.index\n",
      "ssd_mobilenet_v1_coco_11_06_2017/model.ckpt.meta\n",
      "ssd_mobilenet_v1_coco_11_06_2017/frozen_inference_graph.pb\n",
      "ssd_mobilenet_v1_coco_11_06_2017/model.ckpt.data-00000-of-00001\n",
      "ssd_mobilenet_v1_coco_11_06_2017/graph.pbtxt\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "wget http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_11_06_2017.tar.gz \\\n",
    "    -q -O ssd_mobilenet_v1_coco_11_06_2017.tar.gz\n",
    "tar xvzf ssd_mobilenet_v1_coco_11_06_2017.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train network\n",
    "\n",
    "We now provide an example on how to fine-tune the nework on our real image dataset.\n",
    "The procedure is similar for other datasets and run configurations.\n",
    "Provided you have the `workspace/data/train.record` and `workspace/data/eval.record` files (respectively training and validation partitions), the configuration of the desired run in `workspace/config/example.config` and wish the output of training to be stored in `workspace/train/example/` just run the following command\n",
    "(adjust the `num_clones` parameter to match the number of GPU devices in your machine; ours was 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! python models/research/object_detection/train.py \\\n",
    "    --logtostderr \\\n",
    "    --train_dir=workspace/train/example \\\n",
    "    --pipeline_config_path=workspace/config/example.config \\\n",
    "    --num_clones=2 \\\n",
    "    --ps_tasks=1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating performance on validation set\n",
    "\n",
    "Simultaneous with training, we ran an evaluation process on the CPU, to test **mAP** performance on validation set.\n",
    "Simply run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "# Make CPU the only visible CUDA device\n",
    "export CUDA_VISIBLE_DEVICES=3\n",
    "python models/research/object_detection/eval.py \\\n",
    "    --logtostderr \\\n",
    "    --eval_dir=workspace/eval/example/ \\\n",
    "    --pipeline_config_path=workspace/config/example.config \\\n",
    "    --checkpoint_dir=workspace/train/example/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Watch training in tensorboard\n",
    "\n",
    "To watch the current progress of the network training, such as loss function and **mAP** curves over time, you can use tensorboard by running:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! tensorboard \\\n",
    "    --logdir=training:workspace/train/,testing:workspace/eval/ \\\n",
    "    --port=6006 --host=localhost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating performance\n",
    "\n",
    "Now that we have trained the network, we want to use it for inference.\n",
    "For that, we need to first export the graph with frozen weights.\n",
    "\n",
    "#### Exporting inference graph\n",
    "\n",
    "Provided you have let the train process finished, you should have the checkpoint at the 100th iteration in `workspace/train/example/`.\n",
    "If so, export the frozen inference graph to `workspace/inference_graph/example` by running: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! python models/research/object_detection/export_inference_graph.py \\\n",
    "    --input_type image_tensor \\\n",
    "    --pipeline_config_path workspace/config/example.config \\\n",
    "    --trained_checkpoint_prefix workspace/train/example/model.ckpt-100 \\\n",
    "    --output_directory workspace/inference_graph/example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Infer detections\n",
    "\n",
    "To test the trained network on the real image test set you just need to have the corresponding `test.record` in `workspace/data` folder and run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "mkdir -p workspace/inference_results/example\n",
    "python -m object_detection/inference/infer_detections \\\n",
    "    --input_tfrecord_paths=workspace/data/test.record \\\n",
    "    --inference_graph=workspace/inference_graph/example/frozen_inference_graph.pb \\\n",
    "    --discard_image_pixels \\\n",
    "    --output_tfrecord_path=workspace/inference_results/example/detections.tfrecord"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Compute mAP and Precision-Recall curves\n",
    "\n",
    "With the generated inferences you can finally obtain the **AP** per class and mean, as well as the precision-recall curves per class.\n",
    "For this make sure you have a valid `example.pbtxt` file in `workspace/test_input_config/` and run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! python -m object_detection/metrics/offline_eval_map_corloc \\\n",
    "    --eval_dir=workspace/inference_results/example/ \\\n",
    "    --eval_config_path=workspace/config/test_eval_config.pbtxt \\\n",
    "    --input_config_path=workspace/test_input_config/example.pbtxt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Overlay bounding boxes on real image test set\n",
    "\n",
    "Finally, it will be usefull to look at the actual detections overlayed on the original test set images.\n",
    "For this, run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "mkdir -p workspace/overlay/example\n",
    "python overlay.py \\\n",
    "    --images_path=dataset/test/images/ \\\n",
    "    --save_path=workspace/overlay/example/ \\\n",
    "    --ckpt_path=workspace/inference_graph/example/frozen_inference_graph.pb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Automating the training procedures for several subdatasets\n",
    "\n",
    "For our ablation study we were required to train several networks, using different configurations.\n",
    "Our general approach is to prepare all required configuration files beforehand, and then run a script to automate the training and validation procedures, as well as exporting the inference graphs and calculating the required metrics.\n",
    "For this, we provide `run_network.sh`, which resembles our final setup:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! run_network.sh workspace"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}