diff --git a/README.md b/README.md
index 0d319a0..b4500e4 100644
--- a/README.md
+++ b/README.md
@@ -1,3 +1,8 @@
+## Contributing
+1. A python script `demo.py` that uses a pre-trained model to detect objects in a point cloud. (without downloading data | SUN RGB-D val set samples and pre-trained model provided in `demo` folder). Check `Run Demo`.
+2. A python script `fasterRCNN_detections.py` that uses a pre-trained Faster RCNN model trained on the Open Images V4 dataset to output 2D object detections in the format required by ImVoteNet. Check `FasterRCNN Detections` at the end of the file.
+3. Fixes to issues I ran into while training. Check `Fixes` at the end of the file.
+
# ImVoteNet
**Boosting 3D Object Detection in Point Clouds with Image Votes**
@@ -29,7 +34,7 @@ pip install tensorboardX --no-deps
```
Now we are ready to clone this repository:
```bash
-git clone git@github.com:facebookresearch/imvotenet.git
+git clone git@github.com:Sakshee5/imvotenet.git
cd imvotenet
```
The code depends on [PointNet++](http://arxiv.org/abs/1706.02413) as a backbone, which needs compilation:
@@ -38,6 +43,15 @@ cd pointnet2
python setup.py install
cd ..
```
+## Run Demo
+The pre-trained model with sample point clouds, RGB Images, Depth Maps, Camera Calib and the 2D bounding box detections are available in the demo folder.
+After completing installation, Run:
+```bash
+python demo.py
+```
+The demo uses a pre-trained model (on SUN RGB-D) to detect objects in a point cloud from an indoor room (from SUN RGB-D val set). You can use 3D visualization software such as the MeshLab to open the dumped file under `demo/results` to see the 3D detection output. Specifically, open `***_pc.ply` and `***_pred_confident_nms_bbox.ply` to see the input point cloud and predicted 3D bounding boxes. Incase you want to check the class labels of the detected objects, set `inference_switch = True` in the second last line of `demo.py`. You can check the `***_pred_map_cls.txt` to get the class labels.
+
+The ImVoteNet model needs the point cloud as well as the geometric, semantic and texture cues extracted from the RGB Image as input. `demo.py` creates a pipeline that inputs the RGB Image, Depth Map, Camera Calib and the 2D bounding box detections (Faster RCNN 2d object detection backbone output) to output all necessary inputs for ImVoteNet model in the right format. It further uses the pre-trained model to detect objects.
## Data
Please follow the steps listed [here](https://github.com/facebookresearch/votenet/blob/master/sunrgbd/README.md) to set up the SUN RGB-D dataset in the `sunrgbd` folder. The expected dataset structure under `sunrgbd` is:
@@ -85,3 +99,91 @@ For reference, ImVoteNet gives around 63 mAP@0.25.
## LICENSE
The code is released under the [MIT license](LICENSE).
+
+## FasterRCNN Detections
+The official ImVoteNet repository does not provide the pre-trained Faster RCNN model. Instead you are directly supposed to download the `.txt` files and use them for training.
+
+(Refer: For ImVoteNet, we provide 2D detection results from a pre-trained Faster R-CNN detector [here](https://dl.fbaipublicfiles.com/imvotenet/2d_bbox/sunrgbd_2d_bbox_50k_v1.tgz).)
+
+Currently the `demo.py` script directly uses RGB Images from the SUN RGB-D val dataset and thus the 2D bbox detections can directly be downloaded and used. But incase we want to run the demo script on a custom RGB Image, we need the 2D bbox detection in the right format to run the demo script.
+
+`fasterRCNN_detections.py` uses a pretrained Faster RCNN on Open Images V4 Dataset from the tensorflow-hub. It's been trained on 600 categories with ImageNet pre-trained Inception Resnet V2 as image feature extractor. The inference block added in the script makes sure that only objects of interest are detected and saved in a `.txt` file as required by ImvoteNet.
+
+Firtly install dependencies:
+```bash
+pip install tensorflow-gpu
+pip install tensorflow-hub
+
+```
+Run:
+```bash
+python fasterRCNN_detections.py
+```
+Check `demo/FasterRCNN_labels` to get the corresponding text file which can inturn be used with the `demo.py` script.
+
+#### Categories of interest from sun rgbd | possible category from the 600 categories of Open Images Dataset
+bed | Bed
+table | Table
+sofa | Sofa bed
+chair | Chair
+toilet | Toilet
+desk | Desk
+dresser | Filing cabinet
+night_stand | Nightstand
+bookshelf | Bookcase
+bathtub | Bathtub
+
+## Fixes
+3. Error while compiling PointNet2 with newer/higher CUDA version (like CUDA>=11.0)
+ FIX:
+ 1. Change all instances of AT_CHECK to TORCH_CHECK inside all the source files inside `pointnet2/_ext_src/src and pointnet2/_ext_src/include`. This is due to an API change in PyTorch.
+ 2. Change pointnet2/setup.py:
+ ```bash
+ # Copyright (c) Facebook, Inc. and its affiliates.
+ #
+ # This source code is licensed under the MIT license found in the
+ # LICENSE file in the root directory of this source tree.
+
+ from setuptools import setup
+ from torch.utils.cpp_extension import BuildExtension, CUDAExtension
+ import glob
+ import os
+
+ _ext_src_root = "_ext_src"
+ _ext_sources = glob.glob("{}/src/*.cpp".format(_ext_src_root)) + glob.glob(
+ "{}/src/*.cu".format(_ext_src_root)
+ )
+ _ext_headers = glob.glob("{}/include/*".format(_ext_src_root))
+
+ headers = "-I" + os.path.join(os.path.dirname(os.path.abspath(__file__)), '_ext_src', 'include')
+
+ setup(
+ name='pointnet2',
+ ext_modules=[
+ CUDAExtension(
+ name='pointnet2._ext',
+ sources=_ext_sources,
+ extra_compile_args={
+ "cxx": ["-O2", headers],
+ "nvcc": ["-O2", headers]
+ },
+ )
+ ],
+ cmdclass={
+ 'build_ext': BuildExtension
+ }
+ )
+ ```
+
+2. Error message before training: ImportError: No module named 'google'
+ FIX: Run
+ ```bash
+ pip install --upgrade google-api-python-client
+ ```
+
+3. Error message: AttributeError: ‘Fraction’ object has no attribute ‘gcd’
+ FIX:
+ A minor change is needed in the site-packages of your virtual environment.
+ Open `path to env/lib/python_3.8/site-packages/networkx/algorithms/dag.py`
+ Change `from fractions import gcd` to `import math`
+ Change the one gcd instance in the file from `gcd()` to `math.gcd()`
diff --git a/demo.py b/demo.py
new file mode 100644
index 0000000..8da11d1
--- /dev/null
+++ b/demo.py
@@ -0,0 +1,286 @@
+import os
+import sys
+import numpy as np
+import argparse
+import importlib
+import time
+from plyfile import PlyData
+import cv2
+import scipy.io as sio
+
+import torch
+import torch.nn as nn
+import torch.optim as optim
+
+BASE_DIR = os.path.dirname(os.path.abspath(__file__))
+ROOT_DIR = BASE_DIR
+sys.path.append(os.path.join(ROOT_DIR, 'utils'))
+sys.path.append(os.path.join(ROOT_DIR, 'models'))
+from ap_helper import parse_predictions
+
+NUM_CLS = 10 # sunrgbd number of classes
+MAX_NUM_2D_DET = 100 # maximum number of 2d boxes per image
+MAX_NUM_PIXEL = 530*730 # maximum number of pixels per image
+KEY_PREFIX_LIST = ['pc_img_']
+
+max_imvote_per_pixel = 3
+vote_dims = 1 + max_imvote_per_pixel*4
+
+type2class = {'bed': 0, 'table': 1, 'sofa': 2, 'chair': 3, 'toilet': 4, 'desk': 5, 'dresser': 6, 'night_stand': 7,
+ 'bookshelf': 8, 'bathtub': 9}
+class2type = {type2class[t]: t for t in type2class}
+num_point = 20000
+
+# choose any of the samples provided
+bbox_2d_path = os.path.join(BASE_DIR, 'demo/FasterRCNN_labels/000002.txt')
+calib_filepath = os.path.join(BASE_DIR, 'demo/calib/000002.txt')
+image_path = os.path.join(BASE_DIR, 'demo/images/000002.jpg')
+depth_filepath = os.path.join(BASE_DIR, 'demo/depth/000002.mat')
+
+
+def processData():
+ """
+ A pipeline to extract point cloud, geometric, semantic and texture cues for ImVoteNet training from Depth file,
+ calibration file, 2D RGB Image and corresponding 2D Faster RCNN object detection algorithm output.
+
+ :return:
+ """
+ def pre_load_2d_bboxes(bbox_2d_path):
+ print("pre-loading 2d boxes from: " + bbox_2d_path)
+
+ # Read 2D object detection boxes and scores
+ cls_id_list = []
+ cls_score_list = []
+ bbox_2d_list = []
+
+ for line in open(os.path.join(bbox_2d_path), 'r'):
+ det_info = line.rstrip().split(" ")
+ prob = float(det_info[-1])
+ # Filter out low-confidence 2D detections
+ if prob < 0.1:
+ continue
+ cls_id_list.append(type2class[det_info[0]])
+ cls_score_list.append(prob)
+ bbox_2d_list.append(np.array([float(det_info[i]) for i in range(4, 8)]).astype(np.int32))
+
+ return cls_id_list, cls_score_list, bbox_2d_list
+
+
+ def getCameraParameters(calib_filepath):
+ """
+ Calib .txt file looks as follows
+ 0.97959 0.012593 0.20061 0.012593 0.99223 -0.12377 -0.20061 0.12377 0.97182 -----> line 1 = rotation matrix
+ 529.5 0 0 0 529.5 0 365 265 1 ------> line 2 = camera intrinsic matrix
+
+ :return: 3x3 rotation matrix and 3x3 camera intrinsic matrix
+ """
+ lines = [line.rstrip() for line in open(calib_filepath)]
+ Rtilt = np.reshape(np.array([float(x) for x in lines[0].rstrip().split(' ')]), (3, 3), 'F')
+ Rtilt = np.expand_dims(Rtilt.astype(np.float32), 0)
+ K = np.reshape(np.array([float(x) for x in lines[1].rstrip().split(' ')]), (3, 3), 'F')
+ K = np.expand_dims(K.astype(np.float32), 0)
+
+ return Rtilt, K
+
+ def read_ply(ply_filepath):
+ """ read XYZ point cloud from filename PLY file """
+ plydata = PlyData.read(ply_filepath)
+ pc = plydata['vertex'].data
+ pc_array = np.array([[x, y, z] for x,y,z in pc])
+ return pc_array
+
+ def loadDepthMat(depth_filepath):
+ """
+ depth is represented as a .mat file.
+ :param depth_filepath: path to directory
+ :return: point cloud (x, y, z) values of each pixel
+ """
+ depth = sio.loadmat(depth_filepath)['instance']
+ # print(np.array(depth).shape) # -----> (N, 6)
+ pc = np.array(depth)[:, :3]
+
+ return pc
+
+ def random_sampling(pc, num_sample, replace=None, return_choices=False):
+ """ Input is NxC, output is num_samplexC
+ """
+ if replace is None: replace = (pc.shape[0]= max_imvote_per_pixel:
+ continue
+ full_img_votes[v, u, (1 + iidx * 4):(1 + iidx * 4 + 2)] = img_vote[v - v0, u - u0, :]
+ full_img_votes[v, u, (1 + iidx * 4 + 2)] = cls2d
+ full_img_votes[v, u, (1 + iidx * 4 + 3)] = i2d + 1
+ # add +1 here as we need a dummy feature for pixels outside all boxes
+ full_img_votes[v0:(v0 + h), u0:(u0 + w), 0] += 1
+
+ full_img_votes_1d = np.zeros((MAX_NUM_PIXEL * vote_dims), dtype=np.float32)
+ full_img_votes_1d[0:full_img_height * full_img_width * vote_dims] = full_img_votes.flatten()
+
+ full_img_votes_1d = np.expand_dims(full_img_votes_1d.astype(np.float32), 0)
+
+ return full_img_votes_1d
+
+ scale_ratio = np.array(1.0).astype(np.float32)
+ point_cloud = loadDepthMat(depth_filepath)
+ pc = preprocess_point_cloud(point_cloud)
+ Rtilt, K = getCameraParameters(calib_filepath)
+ cls_id_list, cls_score_list, bbox_2d_list = pre_load_2d_bboxes(bbox_2d_path)
+ cls_score_feats = getClsScoreFeats(cls_id_list, cls_score_list)
+ full_img = load_image(image_path)
+ full_img_1d, full_img_width = getTextureCues(full_img)
+ full_img_votes_1d = get_full_img_votes_1d(full_img, cls_id_list, bbox_2d_list)
+
+ return scale_ratio, pc, Rtilt, K, cls_score_feats, full_img_1d, full_img_width, full_img_votes_1d
+
+if __name__ == '__main__':
+
+ from sunrgbd_detection_dataset import DC # dataset config
+
+ # Set file paths and dataset config
+ demo_dir = os.path.join(BASE_DIR, 'demo')
+ sys.path.append(os.path.join(ROOT_DIR, 'sunrgbd'))
+ checkpoint_path = os.path.join(demo_dir, 'checkpoint.tar')
+
+ eval_config_dict = {'remove_empty_box': True, 'use_3d_nms': True, 'nms_iou': 0.25,
+ 'use_old_type_nms': False, 'cls_nms': False, 'per_class_proposal': False,
+ 'conf_thresh': 0.5, 'dataset_config': DC}
+
+ # Init the model and optimzier
+ MODEL = importlib.import_module('imvotenet') # import network module
+ device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
+
+ net = MODEL.ImVoteNet(input_feature_dim=1, num_proposal=256, vote_factor=1, sampling='vote_fps',
+ max_imvote_per_pixel=3, image_feature_dim=18,
+ num_class=DC.num_class, num_heading_bin=DC.num_heading_bin,
+ num_size_cluster=DC.num_size_cluster,
+ mean_size_arr=DC.mean_size_arr).to(device)
+ print('Constructed model.')
+
+ # Load checkpoint
+ optimizer = optim.Adam(net.parameters(), lr=0.001)
+ checkpoint = torch.load(checkpoint_path)
+ net.load_state_dict(checkpoint['model_state_dict'])
+ optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
+ epoch = checkpoint['epoch']
+ print("Loaded checkpoint %s (epoch: %d)" % (checkpoint_path, epoch))
+
+ # Load and preprocess inputs
+ net.eval() # set model to eval mode (for bn and dp)
+ scale_ratio, pc, Rtilt, K, cls_score_feats, full_img_1d, full_img_width, full_img_votes_1d = processData()
+
+ print('Input Data Loaded')
+
+ # Model inference
+ inputs = {'point_clouds': torch.from_numpy(pc).to(device)}
+ inputs.update({'scale': torch.from_numpy(scale_ratio).to(device),
+ 'calib_K': torch.from_numpy(K).to(device),
+ 'calib_Rtilt': torch.from_numpy(Rtilt).to(device),
+ 'cls_score_feats': torch.from_numpy(cls_score_feats).to(device),
+ 'full_img_votes_1d': torch.from_numpy(full_img_votes_1d).to(device),
+ 'full_img_1d': torch.from_numpy(full_img_1d).to(device),
+ 'full_img_width': torch.from_numpy(full_img_width).to(device),
+ })
+ tic = time.time()
+ with torch.no_grad():
+ end_points = net(inputs, joint_only=True)
+ toc = time.time()
+ print('Inference time: %f' % (toc - tic))
+
+ end_points['point_clouds'] = inputs['point_clouds']
+ end_points['scale'] = inputs['scale']
+ end_points['calib_K'] = inputs['calib_K']
+ end_points['calib_Rtilt'] = inputs['calib_Rtilt']
+ end_points['cls_score_feats'] = inputs['cls_score_feats']
+ end_points['full_img_votes_1d'] = inputs['full_img_votes_1d']
+ end_points['full_img_1d'] = inputs['full_img_1d']
+ end_points['full_img_width'] = inputs['full_img_width']
+ pred_map_cls = parse_predictions(end_points, eval_config_dict, KEY_PREFIX_LIST[0])
+ print('Finished detection. %d object detected.' % (len(pred_map_cls[0])))
+
+ dump_dir = os.path.join(demo_dir, 'results')
+ if not os.path.exists(dump_dir): os.mkdir(dump_dir)
+ MODEL.dump_results(end_points, dump_dir, DC, inference_switch=False, key_prefix='pc_img_')
+ print('Dumped detection results to folder %s' % (dump_dir))
\ No newline at end of file
diff --git a/demo/FasterRCNN_labels/000001.txt b/demo/FasterRCNN_labels/000001.txt
new file mode 100644
index 0000000..cd9ab37
--- /dev/null
+++ b/demo/FasterRCNN_labels/000001.txt
@@ -0,0 +1,100 @@
+dresser 0 0 -10 0.00 4.63 229.61 506.07 0.9671742
+bed 0 0 -10 325.94 197.41 674.12 522.54 0.7690236
+night_stand 0 0 -10 351.08 249.28 669.98 522.34 0.4608725
+night_stand 0 0 -10 574.68 274.59 675.14 335.50 0.2503404
+night_stand 0 0 -10 169.26 168.09 301.17 356.38 0.1819366
+dresser 0 0 -10 129.90 16.31 287.91 356.38 0.1383614
+dresser 0 0 -10 326.78 232.28 666.34 517.07 0.1048061
+dresser 0 0 -10 163.13 150.79 305.29 344.88 0.0900657
+dresser 0 0 -10 3.80 5.54 100.59 516.54 0.0817019
+night_stand 0 0 -10 586.22 269.81 712.06 399.23 0.0659305
+night_stand 0 0 -10 695.86 283.50 729.98 400.31 0.0485836
+table 0 0 -10 691.31 266.61 726.73 400.19 0.0300846
+dresser 0 0 -10 0.00 236.59 198.92 525.09 0.0285815
+night_stand 0 0 -10 536.23 253.23 663.51 335.17 0.0265132
+night_stand 0 0 -10 646.12 261.97 728.72 396.23 0.0254247
+bed 0 0 -10 376.57 155.37 586.56 283.71 0.0251486
+dresser 0 0 -10 337.41 160.90 641.98 415.38 0.0235936
+table 0 0 -10 172.86 168.53 300.49 352.31 0.0211648
+night_stand 0 0 -10 9.73 185.17 182.47 523.14 0.0184744
+sofa 0 0 -10 342.51 251.94 678.65 521.48 0.0174416
+bed 0 0 -10 337.44 154.80 637.57 386.09 0.0145650
+desk 0 0 -10 0.00 170.46 186.56 528.75 0.0111596
+night_stand 0 0 -10 389.08 219.22 599.69 352.49 0.0107695
+night_stand 0 0 -10 140.31 34.45 292.66 359.01 0.0106386
+night_stand 0 0 -10 23.66 6.78 238.60 441.25 0.0101270
+table 0 0 -10 579.26 269.50 669.43 319.72 0.0098805
+dresser 0 0 -10 384.86 162.48 577.13 277.41 0.0089098
+night_stand 0 0 -10 517.90 267.21 674.64 402.26 0.0088033
+table 0 0 -10 332.79 260.43 685.99 524.58 0.0087451
+table 0 0 -10 672.72 274.73 727.99 343.00 0.0085954
+dresser 0 0 -10 18.74 1.40 232.79 279.89 0.0084762
+dresser 0 0 -10 695.54 274.92 729.38 400.15 0.0083456
+bed 0 0 -10 2.11 63.15 203.93 530.00 0.0081682
+bed 0 0 -10 690.45 249.94 729.78 419.58 0.0077746
+table 0 0 -10 4.18 270.94 173.75 530.00 0.0065164
+chair 0 0 -10 685.83 261.95 729.78 388.94 0.0064098
+table 0 0 -10 662.30 252.19 728.58 382.03 0.0063842
+desk 0 0 -10 690.80 264.95 727.96 401.21 0.0061511
+sofa 0 0 -10 691.59 253.51 729.15 424.18 0.0059084
+table 0 0 -10 679.03 263.64 728.80 316.84 0.0053112
+dresser 0 0 -10 646.09 244.57 725.72 379.72 0.0041151
+bookshelf 0 0 -10 0.70 0.00 219.00 522.15 0.0039640
+chair 0 0 -10 333.46 257.68 687.54 523.01 0.0038350
+dresser 0 0 -10 26.34 0.00 410.22 432.79 0.0038129
+desk 0 0 -10 10.77 0.00 255.03 465.02 0.0034924
+chair 0 0 -10 0.00 207.89 217.19 527.30 0.0033958
+chair 0 0 -10 160.13 168.87 305.21 357.65 0.0031881
+night_stand 0 0 -10 571.95 241.98 672.69 314.44 0.0030922
+desk 0 0 -10 171.87 161.97 303.96 349.05 0.0030461
+bed 0 0 -10 169.86 151.31 302.63 356.72 0.0028647
+night_stand 0 0 -10 671.49 292.29 728.06 380.83 0.0027797
+night_stand 0 0 -10 714.38 277.69 730.00 411.96 0.0025933
+dresser 0 0 -10 157.76 40.79 241.83 345.19 0.0023496
+night_stand 0 0 -10 692.95 317.62 729.41 379.62 0.0023084
+chair 0 0 -10 249.30 157.39 312.12 243.23 0.0022627
+desk 0 0 -10 320.75 249.87 700.24 520.85 0.0022450
+chair 0 0 -10 263.71 138.26 316.77 304.21 0.0021180
+table 0 0 -10 648.15 246.53 727.66 328.78 0.0020180
+night_stand 0 0 -10 395.66 165.10 578.93 279.95 0.0019763
+table 0 0 -10 691.26 250.51 728.86 299.37 0.0019555
+night_stand 0 0 -10 349.04 175.79 639.17 406.46 0.0018587
+table 0 0 -10 708.74 275.42 729.63 415.19 0.0018364
+night_stand 0 0 -10 422.53 268.94 672.69 435.00 0.0017991
+table 0 0 -10 693.28 309.96 727.36 378.14 0.0017959
+bed 0 0 -10 0.00 0.00 352.13 499.84 0.0017427
+dresser 0 0 -10 688.34 197.43 729.23 395.57 0.0017250
+toilet 0 0 -10 690.28 265.76 727.77 398.83 0.0016906
+chair 0 0 -10 673.60 274.21 729.95 335.59 0.0016770
+dresser 0 0 -10 77.63 147.04 318.67 396.99 0.0016299
+table 0 0 -10 584.80 279.37 673.73 343.93 0.0015967
+chair 0 0 -10 276.23 166.41 316.79 247.45 0.0015934
+dresser 0 0 -10 241.74 153.24 308.42 303.26 0.0015828
+dresser 0 0 -10 583.23 220.52 728.69 366.64 0.0014523
+night_stand 0 0 -10 241.82 156.31 310.84 300.98 0.0014503
+bathtub 0 0 -10 690.26 263.93 727.66 394.66 0.0014140
+table 0 0 -10 729.53 253.24 729.99 411.41 0.0014067
+sofa 0 0 -10 387.06 155.49 587.18 277.79 0.0013994
+night_stand 0 0 -10 496.28 236.99 611.64 343.93 0.0013012
+bookshelf 0 0 -10 691.27 258.61 727.67 397.08 0.0012757
+dresser 0 0 -10 212.74 111.57 297.01 299.87 0.0012504
+bed 0 0 -10 456.26 160.63 705.05 406.83 0.0011776
+toilet 0 0 -10 332.41 264.09 677.78 526.14 0.0011581
+table 0 0 -10 237.08 158.03 306.61 306.78 0.0011235
+night_stand 0 0 -10 673.05 276.05 730.00 338.17 0.0011196
+night_stand 0 0 -10 637.29 282.62 729.20 470.02 0.0010951
+dresser 0 0 -10 584.94 267.61 708.60 398.21 0.0010399
+chair 0 0 -10 680.95 263.75 730.00 313.61 0.0010309
+chair 0 0 -10 333.16 152.04 644.07 428.21 0.0010036
+desk 0 0 -10 84.78 128.61 312.60 378.46 0.0010023
+table 0 0 -10 19.55 146.34 291.07 526.05 0.0009643
+night_stand 0 0 -10 595.88 272.60 694.84 318.33 0.0009457
+dresser 0 0 -10 714.47 269.98 729.78 412.97 0.0009117
+dresser 0 0 -10 116.78 182.27 224.91 366.22 0.0009071
+bed 0 0 -10 662.27 251.43 730.00 382.15 0.0009014
+dresser 0 0 -10 65.69 167.16 201.86 386.41 0.0008901
+night_stand 0 0 -10 543.40 201.31 704.90 369.41 0.0008888
+sofa 0 0 -10 664.18 252.88 730.00 378.39 0.0008821
+table 0 0 -10 585.18 264.83 715.12 381.44 0.0008692
+table 0 0 -10 641.81 273.63 717.12 318.19 0.0008667
+dresser 0 0 -10 124.16 30.28 212.52 350.38 0.0008626
diff --git a/demo/FasterRCNN_labels/000002.txt b/demo/FasterRCNN_labels/000002.txt
new file mode 100644
index 0000000..2506fda
--- /dev/null
+++ b/demo/FasterRCNN_labels/000002.txt
@@ -0,0 +1,100 @@
+bed 0 0 -10 0.00 120.49 471.22 530.00 0.9899822
+night_stand 0 0 -10 466.37 306.92 675.63 502.35 0.8677854
+night_stand 0 0 -10 2.61 148.38 72.09 251.87 0.6969147
+night_stand 0 0 -10 44.50 201.57 150.62 279.84 0.6011212
+dresser 0 0 -10 461.47 328.70 689.22 506.07 0.3985506
+night_stand 0 0 -10 9.45 193.37 158.25 313.39 0.3606137
+night_stand 0 0 -10 57.07 200.73 151.17 244.75 0.2942304
+sofa 0 0 -10 0.00 136.57 480.50 526.82 0.2374999
+night_stand 0 0 -10 11.17 178.17 135.49 256.76 0.1470702
+chair 0 0 -10 0.00 376.32 264.13 530.00 0.0393717
+night_stand 0 0 -10 4.58 129.03 128.84 342.96 0.0358145
+dresser 0 0 -10 0.81 132.54 74.10 249.18 0.0357704
+table 0 0 -10 477.02 319.02 671.73 511.27 0.0292029
+night_stand 0 0 -10 4.55 181.96 84.17 287.18 0.0224589
+night_stand 0 0 -10 1.98 176.69 64.44 233.20 0.0221353
+desk 0 0 -10 460.64 316.40 692.65 517.87 0.0220080
+night_stand 0 0 -10 8.41 125.19 108.46 251.00 0.0201211
+bed 0 0 -10 429.22 355.91 726.98 527.87 0.0185063
+bed 0 0 -10 0.00 178.25 167.84 384.35 0.0171684
+table 0 0 -10 2.26 146.75 81.35 244.60 0.0148819
+night_stand 0 0 -10 25.04 164.54 78.78 264.84 0.0120566
+dresser 0 0 -10 2.18 45.52 99.39 284.29 0.0118718
+sofa 0 0 -10 436.07 361.60 723.45 529.88 0.0104749
+night_stand 0 0 -10 13.98 228.17 140.28 289.41 0.0103945
+night_stand 0 0 -10 1.41 77.47 77.58 287.28 0.0083456
+bed 0 0 -10 1.12 195.62 250.09 523.04 0.0068518
+chair 0 0 -10 40.57 128.78 489.97 530.00 0.0068050
+bed 0 0 -10 0.00 73.26 209.40 424.01 0.0054967
+table 0 0 -10 0.23 222.69 475.25 530.00 0.0054419
+night_stand 0 0 -10 71.72 208.22 193.91 308.04 0.0052463
+table 0 0 -10 5.15 120.85 71.89 204.78 0.0047719
+dresser 0 0 -10 2.51 136.97 139.52 296.90 0.0045831
+desk 0 0 -10 0.69 124.40 77.11 247.12 0.0044912
+night_stand 0 0 -10 91.69 207.69 163.13 283.11 0.0041811
+chair 0 0 -10 12.09 279.70 406.92 527.74 0.0039414
+night_stand 0 0 -10 47.15 173.83 483.56 530.00 0.0039386
+bed 0 0 -10 1.99 357.69 299.51 529.42 0.0038626
+night_stand 0 0 -10 4.53 141.80 51.15 317.38 0.0035755
+night_stand 0 0 -10 35.75 183.85 93.19 272.81 0.0030481
+night_stand 0 0 -10 0.58 139.34 24.24 273.03 0.0030288
+dresser 0 0 -10 11.97 147.33 475.22 530.00 0.0029622
+night_stand 0 0 -10 5.65 136.96 78.47 202.58 0.0029175
+night_stand 0 0 -10 466.53 305.73 591.92 461.69 0.0027805
+table 0 0 -10 5.41 363.51 310.80 530.00 0.0026480
+night_stand 0 0 -10 11.32 167.70 54.52 213.72 0.0024617
+chair 0 0 -10 2.75 369.60 117.86 528.80 0.0022960
+night_stand 0 0 -10 0.64 110.38 50.97 231.62 0.0022278
+bed 0 0 -10 0.18 49.12 79.53 300.66 0.0020590
+bed 0 0 -10 0.00 121.15 93.50 484.76 0.0020510
+desk 0 0 -10 0.00 150.24 492.51 525.44 0.0020317
+bed 0 0 -10 188.03 103.69 698.05 530.00 0.0019509
+night_stand 0 0 -10 0.12 95.17 38.49 382.11 0.0019442
+table 0 0 -10 8.83 196.57 156.18 316.13 0.0019191
+dresser 0 0 -10 17.45 200.23 155.55 295.09 0.0018839
+night_stand 0 0 -10 116.37 213.28 211.10 296.56 0.0016193
+table 0 0 -10 47.10 197.79 148.73 247.48 0.0015911
+night_stand 0 0 -10 381.29 348.15 715.43 523.78 0.0015737
+table 0 0 -10 1.86 113.21 51.67 239.07 0.0015628
+dresser 0 0 -10 1.97 90.88 76.59 392.22 0.0015130
+bed 0 0 -10 0.00 131.89 146.14 308.38 0.0014833
+night_stand 0 0 -10 19.13 153.22 82.39 218.18 0.0014752
+bookshelf 0 0 -10 9.52 67.75 471.09 498.23 0.0014108
+bed 0 0 -10 16.75 201.00 166.94 303.35 0.0014089
+chair 0 0 -10 0.00 198.86 61.60 519.96 0.0013462
+table 0 0 -10 5.06 82.90 107.08 254.09 0.0013185
+dresser 0 0 -10 5.67 175.04 134.12 253.33 0.0012996
+dresser 0 0 -10 0.40 83.72 51.74 275.52 0.0012749
+bookshelf 0 0 -10 463.58 322.17 692.48 511.11 0.0012682
+dresser 0 0 -10 439.94 102.32 724.26 474.20 0.0012544
+table 0 0 -10 363.45 396.18 716.10 530.00 0.0012110
+bed 0 0 -10 1.11 146.13 47.20 503.16 0.0011724
+sofa 0 0 -10 189.38 121.39 700.62 530.00 0.0011389
+bed 0 0 -10 349.29 250.83 694.70 529.83 0.0011320
+table 0 0 -10 56.20 205.86 153.41 292.80 0.0010840
+chair 0 0 -10 143.23 353.57 397.73 530.00 0.0010635
+desk 0 0 -10 1.90 68.54 112.09 254.57 0.0010620
+night_stand 0 0 -10 0.34 145.46 86.19 460.98 0.0010575
+chair 0 0 -10 5.29 96.09 100.25 229.99 0.0010408
+bed 0 0 -10 19.53 46.36 459.08 282.07 0.0010165
+table 0 0 -10 1.94 107.83 56.45 178.65 0.0009709
+table 0 0 -10 6.44 400.83 161.24 530.00 0.0009640
+table 0 0 -10 0.99 89.23 41.60 203.96 0.0009325
+bed 0 0 -10 0.86 123.88 76.46 255.10 0.0009292
+sofa 0 0 -10 125.34 128.72 485.35 317.63 0.0008989
+table 0 0 -10 234.06 22.13 450.89 162.09 0.0008774
+chair 0 0 -10 0.63 9.91 81.05 221.67 0.0008550
+night_stand 0 0 -10 39.24 144.24 153.60 249.33 0.0007453
+chair 0 0 -10 464.37 308.14 699.93 515.04 0.0007267
+chair 0 0 -10 470.51 94.92 590.20 227.24 0.0007197
+dresser 0 0 -10 13.67 107.25 133.03 233.93 0.0007111
+bathtub 0 0 -10 459.45 305.26 704.38 517.44 0.0007088
+dresser 0 0 -10 48.33 205.17 149.14 249.25 0.0007021
+sofa 0 0 -10 2.47 114.88 86.15 213.98 0.0006836
+chair 0 0 -10 3.85 109.63 71.80 206.03 0.0006658
+night_stand 0 0 -10 3.84 147.41 15.74 237.27 0.0006526
+bathtub 0 0 -10 2.59 104.20 517.83 486.27 0.0006519
+dresser 0 0 -10 581.64 160.03 729.07 461.78 0.0006408
+desk 0 0 -10 0.00 144.72 140.61 305.96 0.0006347
+bookshelf 0 0 -10 488.36 103.53 666.82 252.57 0.0006192
+table 0 0 -10 1.05 143.28 22.58 272.82 0.0006187
diff --git a/demo/FasterRCNN_labels/000003.txt b/demo/FasterRCNN_labels/000003.txt
new file mode 100644
index 0000000..74f9f00
--- /dev/null
+++ b/demo/FasterRCNN_labels/000003.txt
@@ -0,0 +1,100 @@
+bed 0 0 -10 84.89 167.15 468.72 525.12 0.9952428
+night_stand 0 0 -10 455.40 309.21 551.32 411.49 0.9641371
+dresser 0 0 -10 47.87 213.47 231.82 335.18 0.7589999
+dresser 0 0 -10 544.18 286.44 715.26 434.96 0.5766941
+night_stand 0 0 -10 51.00 211.46 240.85 328.98 0.5533804
+dresser 0 0 -10 678.51 302.21 729.39 433.09 0.5294653
+dresser 0 0 -10 57.01 86.24 249.35 310.26 0.2084377
+night_stand 0 0 -10 543.35 287.92 708.94 446.28 0.2024483
+night_stand 0 0 -10 458.61 309.86 542.14 368.21 0.1511643
+dresser 0 0 -10 715.97 279.92 729.58 454.84 0.1455076
+dresser 0 0 -10 0.72 155.01 66.72 306.91 0.1092101
+dresser 0 0 -10 704.74 300.70 729.77 449.49 0.0842709
+night_stand 0 0 -10 690.09 297.14 730.00 450.58 0.0827914
+desk 0 0 -10 542.72 295.03 728.70 447.24 0.0816956
+dresser 0 0 -10 632.90 302.28 727.85 439.93 0.0776259
+night_stand 0 0 -10 463.74 293.27 655.10 426.80 0.0744234
+dresser 0 0 -10 124.66 99.86 242.89 270.47 0.0610872
+bed 0 0 -10 36.80 196.37 277.45 353.25 0.0343253
+bed 0 0 -10 529.47 280.49 727.81 453.98 0.0328916
+night_stand 0 0 -10 455.63 300.44 539.82 343.56 0.0241575
+table 0 0 -10 547.41 293.50 707.42 448.70 0.0231162
+bed 0 0 -10 53.04 76.08 255.28 360.82 0.0221206
+desk 0 0 -10 45.01 219.25 226.88 339.77 0.0202217
+dresser 0 0 -10 685.71 302.11 728.87 366.77 0.0195208
+night_stand 0 0 -10 0.76 146.57 67.84 316.21 0.0193852
+dresser 0 0 -10 463.32 304.36 557.95 415.84 0.0192442
+dresser 0 0 -10 697.29 327.59 728.85 416.76 0.0182020
+table 0 0 -10 47.47 222.57 222.77 336.20 0.0168611
+table 0 0 -10 1.62 153.07 59.35 329.54 0.0166102
+desk 0 0 -10 0.65 157.13 68.30 309.56 0.0165915
+night_stand 0 0 -10 545.00 281.84 631.55 412.19 0.0137479
+dresser 0 0 -10 580.86 267.57 686.39 382.20 0.0128085
+night_stand 0 0 -10 498.33 314.06 555.57 409.82 0.0121962
+dresser 0 0 -10 562.95 292.76 630.21 413.18 0.0119093
+night_stand 0 0 -10 716.17 289.60 729.82 451.85 0.0111223
+dresser 0 0 -10 721.67 294.43 729.47 428.96 0.0100282
+dresser 0 0 -10 718.59 333.92 728.25 435.22 0.0086671
+dresser 0 0 -10 710.70 338.71 729.25 420.59 0.0079500
+night_stand 0 0 -10 555.43 257.02 628.21 369.59 0.0078906
+night_stand 0 0 -10 659.39 329.87 730.00 431.99 0.0075559
+dresser 0 0 -10 661.09 304.85 716.70 417.77 0.0073430
+dresser 0 0 -10 664.60 338.88 728.73 403.87 0.0070114
+chair 0 0 -10 0.00 151.35 55.82 327.11 0.0069517
+night_stand 0 0 -10 447.89 304.76 513.85 354.82 0.0068676
+sofa 0 0 -10 83.41 229.83 498.05 530.00 0.0068555
+dresser 0 0 -10 556.10 257.62 627.38 369.61 0.0065345
+bed 0 0 -10 688.16 286.92 730.00 454.87 0.0065320
+table 0 0 -10 468.26 306.97 554.81 418.73 0.0064290
+night_stand 0 0 -10 485.08 299.81 593.26 414.03 0.0064152
+night_stand 0 0 -10 559.96 272.27 616.96 332.56 0.0063904
+table 0 0 -10 571.21 250.53 620.09 292.74 0.0062206
+dresser 0 0 -10 573.78 290.15 620.30 369.52 0.0056945
+dresser 0 0 -10 82.52 157.75 480.26 520.08 0.0051989
+night_stand 0 0 -10 581.94 269.86 624.16 356.97 0.0044403
+table 0 0 -10 1.00 174.54 22.71 340.00 0.0040861
+dresser 0 0 -10 457.72 292.01 652.75 425.10 0.0040758
+night_stand 0 0 -10 62.96 94.72 247.24 305.13 0.0040478
+dresser 0 0 -10 454.08 272.03 542.25 393.80 0.0040153
+desk 0 0 -10 689.58 303.16 729.60 456.88 0.0039485
+dresser 0 0 -10 615.84 263.19 705.29 361.32 0.0038466
+night_stand 0 0 -10 99.91 230.46 476.04 530.00 0.0036867
+night_stand 0 0 -10 481.95 303.35 566.29 359.54 0.0036863
+dresser 0 0 -10 725.00 292.90 729.82 416.28 0.0035398
+dresser 0 0 -10 76.28 156.15 231.71 293.09 0.0035050
+dresser 0 0 -10 659.07 298.61 723.06 362.53 0.0032321
+dresser 0 0 -10 637.03 301.87 691.60 409.24 0.0031889
+dresser 0 0 -10 574.65 276.29 634.97 343.92 0.0031767
+chair 0 0 -10 571.50 185.02 639.57 260.40 0.0030462
+dresser 0 0 -10 3.99 156.18 30.53 332.09 0.0030361
+table 0 0 -10 334.19 100.68 491.84 212.73 0.0030203
+table 0 0 -10 689.89 306.53 729.05 459.71 0.0030100
+dresser 0 0 -10 576.49 169.07 730.00 407.53 0.0029546
+dresser 0 0 -10 310.19 103.44 505.97 287.60 0.0029033
+table 0 0 -10 564.65 189.85 639.07 255.92 0.0027595
+chair 0 0 -10 43.17 213.31 207.91 340.82 0.0027036
+table 0 0 -10 83.42 252.76 486.01 530.00 0.0026743
+night_stand 0 0 -10 340.14 363.77 477.33 527.89 0.0026283
+chair 0 0 -10 331.42 96.29 457.36 205.08 0.0025833
+bed 0 0 -10 463.30 300.20 558.68 421.37 0.0025583
+chair 0 0 -10 90.15 170.94 486.90 528.33 0.0025543
+desk 0 0 -10 464.44 301.69 558.45 417.45 0.0024806
+night_stand 0 0 -10 163.14 222.71 237.78 279.99 0.0023566
+night_stand 0 0 -10 452.06 278.18 557.06 356.37 0.0022941
+dresser 0 0 -10 7.68 123.93 135.29 301.46 0.0022436
+chair 0 0 -10 537.89 274.39 692.60 437.13 0.0022244
+chair 0 0 -10 556.99 267.92 617.41 335.24 0.0022171
+bookshelf 0 0 -10 62.16 79.51 246.83 312.90 0.0022010
+night_stand 0 0 -10 65.58 211.25 272.50 438.70 0.0021692
+chair 0 0 -10 675.29 291.15 730.00 432.73 0.0021467
+dresser 0 0 -10 575.35 189.56 645.39 267.42 0.0021301
+dresser 0 0 -10 53.95 207.93 166.34 315.33 0.0021280
+night_stand 0 0 -10 617.32 301.03 729.36 463.05 0.0021005
+dresser 0 0 -10 599.02 276.24 637.16 362.23 0.0020880
+night_stand 0 0 -10 718.75 338.23 728.52 434.79 0.0020755
+night_stand 0 0 -10 686.52 300.70 729.15 366.60 0.0019261
+table 0 0 -10 568.29 225.22 627.65 295.93 0.0018681
+night_stand 0 0 -10 575.91 250.78 667.23 347.20 0.0018560
+bed 0 0 -10 0.00 144.78 68.11 315.08 0.0018240
+bookshelf 0 0 -10 575.68 187.92 644.30 268.84 0.0018238
+dresser 0 0 -10 576.84 251.14 666.68 347.09 0.0017708
diff --git a/demo/FasterRCNN_labels/000004.txt b/demo/FasterRCNN_labels/000004.txt
new file mode 100644
index 0000000..cd318ae
--- /dev/null
+++ b/demo/FasterRCNN_labels/000004.txt
@@ -0,0 +1,100 @@
+bed 0 0 -10 112.13 87.89 589.25 524.76 0.9889286
+night_stand 0 0 -10 51.85 196.24 177.39 309.23 0.9588282
+table 0 0 -10 142.70 217.15 603.89 530.00 0.1537533
+night_stand 0 0 -10 166.14 271.91 620.53 530.00 0.0760628
+table 0 0 -10 55.94 201.35 166.67 304.40 0.0422957
+dresser 0 0 -10 185.41 103.02 413.93 218.89 0.0351818
+dresser 0 0 -10 47.12 201.89 170.02 313.16 0.0350747
+table 0 0 -10 528.35 180.13 588.56 237.47 0.0299322
+desk 0 0 -10 44.33 201.95 171.54 311.86 0.0238756
+table 0 0 -10 521.42 178.51 600.98 268.53 0.0236182
+desk 0 0 -10 520.90 167.80 595.20 242.25 0.0232244
+desk 0 0 -10 120.98 209.35 607.94 530.00 0.0225495
+sofa 0 0 -10 682.44 200.69 728.40 312.82 0.0191000
+table 0 0 -10 524.05 165.02 593.08 219.75 0.0150612
+dresser 0 0 -10 135.46 268.30 623.04 530.00 0.0118353
+sofa 0 0 -10 114.65 93.90 594.41 527.26 0.0111465
+bed 0 0 -10 153.15 274.64 634.19 530.00 0.0077129
+table 0 0 -10 528.15 176.46 591.21 206.35 0.0073763
+chair 0 0 -10 145.20 95.96 613.04 530.00 0.0070725
+bed 0 0 -10 45.33 195.07 171.81 314.78 0.0064634
+chair 0 0 -10 672.02 192.90 729.50 300.39 0.0057663
+table 0 0 -10 557.54 187.99 592.25 268.33 0.0054983
+table 0 0 -10 538.06 186.31 588.26 210.67 0.0049863
+bed 0 0 -10 173.79 31.35 568.12 301.14 0.0047635
+table 0 0 -10 678.86 203.22 727.62 298.44 0.0046870
+table 0 0 -10 518.10 183.37 549.35 237.58 0.0042183
+table 0 0 -10 578.61 180.37 592.28 270.74 0.0038931
+table 0 0 -10 163.97 396.23 617.09 525.35 0.0038182
+desk 0 0 -10 524.55 176.25 590.43 210.36 0.0037775
+chair 0 0 -10 517.20 177.80 547.93 238.77 0.0037572
+dresser 0 0 -10 122.12 97.65 534.24 427.75 0.0032294
+table 0 0 -10 574.87 183.75 598.86 283.81 0.0030261
+table 0 0 -10 479.29 180.55 587.06 242.55 0.0029998
+sofa 0 0 -10 156.38 281.55 631.98 530.00 0.0029662
+dresser 0 0 -10 574.79 173.31 598.44 271.09 0.0027777
+table 0 0 -10 52.80 203.72 178.27 379.40 0.0025228
+night_stand 0 0 -10 574.68 180.56 599.55 270.01 0.0025183
+desk 0 0 -10 578.45 177.79 592.57 271.46 0.0024593
+sofa 0 0 -10 54.40 200.19 169.63 302.81 0.0023983
+desk 0 0 -10 140.74 357.73 632.02 528.84 0.0023793
+chair 0 0 -10 300.73 0.83 415.45 118.58 0.0023754
+chair 0 0 -10 571.58 170.54 600.29 267.96 0.0022792
+sofa 0 0 -10 522.72 168.16 593.29 231.30 0.0022592
+bed 0 0 -10 112.13 115.64 414.75 437.74 0.0021883
+dresser 0 0 -10 680.55 194.99 728.53 304.72 0.0021749
+dresser 0 0 -10 199.70 9.91 413.73 202.06 0.0021531
+sofa 0 0 -10 515.15 185.74 556.45 246.56 0.0021174
+chair 0 0 -10 36.42 197.11 164.40 311.62 0.0020486
+bed 0 0 -10 681.60 197.70 728.80 313.75 0.0019523
+table 0 0 -10 525.57 163.74 594.10 191.35 0.0019128
+night_stand 0 0 -10 680.86 197.97 728.85 308.32 0.0018603
+night_stand 0 0 -10 178.79 399.96 627.80 527.25 0.0018243
+chair 0 0 -10 523.15 169.66 574.74 236.26 0.0017822
+bookshelf 0 0 -10 133.80 93.05 528.20 425.64 0.0015993
+night_stand 0 0 -10 138.35 198.17 179.07 258.02 0.0015264
+night_stand 0 0 -10 117.59 114.64 569.71 506.80 0.0014741
+bed 0 0 -10 521.08 165.16 594.29 231.62 0.0014433
+bed 0 0 -10 57.82 108.09 211.85 310.64 0.0013841
+desk 0 0 -10 544.78 182.19 597.06 268.74 0.0013351
+night_stand 0 0 -10 558.96 188.88 594.26 269.01 0.0013000
+chair 0 0 -10 567.05 129.29 611.47 269.24 0.0012696
+dresser 0 0 -10 0.06 60.24 28.24 313.94 0.0012633
+desk 0 0 -10 525.64 138.05 583.10 219.10 0.0012370
+table 0 0 -10 58.14 192.41 182.04 223.49 0.0012028
+night_stand 0 0 -10 58.29 192.01 186.00 223.78 0.0011687
+table 0 0 -10 562.95 169.29 611.37 280.92 0.0011661
+bookshelf 0 0 -10 47.07 199.54 171.24 313.12 0.0011507
+chair 0 0 -10 499.54 161.25 550.31 248.21 0.0011468
+night_stand 0 0 -10 50.48 193.96 149.16 263.54 0.0010976
+dresser 0 0 -10 585.70 172.41 599.45 280.40 0.0010503
+toilet 0 0 -10 41.62 202.99 167.48 316.80 0.0010450
+chair 0 0 -10 461.18 1.90 533.96 69.99 0.0010359
+table 0 0 -10 556.61 172.48 600.77 239.38 0.0009840
+desk 0 0 -10 574.58 180.23 599.47 283.89 0.0009795
+bathtub 0 0 -10 145.52 285.53 609.58 528.78 0.0009729
+night_stand 0 0 -10 300.95 0.00 409.01 108.24 0.0009492
+desk 0 0 -10 517.49 180.85 551.06 239.57 0.0009300
+sofa 0 0 -10 524.33 175.55 589.36 209.71 0.0009208
+table 0 0 -10 93.12 107.99 575.27 441.40 0.0008780
+bed 0 0 -10 346.06 112.49 585.36 329.59 0.0008666
+dresser 0 0 -10 520.67 165.02 595.54 239.99 0.0008405
+bathtub 0 0 -10 35.01 206.63 171.09 315.41 0.0008277
+dresser 0 0 -10 370.21 52.00 556.18 234.53 0.0008238
+night_stand 0 0 -10 111.70 198.33 188.57 246.83 0.0008101
+night_stand 0 0 -10 126.79 301.85 207.67 449.35 0.0007972
+chair 0 0 -10 520.40 117.45 582.00 175.36 0.0007972
+bed 0 0 -10 475.03 176.66 592.59 243.99 0.0007962
+desk 0 0 -10 85.83 97.88 582.45 447.33 0.0007869
+desk 0 0 -10 679.66 196.45 729.43 303.19 0.0007535
+table 0 0 -10 537.08 175.67 701.46 273.44 0.0007395
+chair 0 0 -10 203.67 2.04 335.26 96.49 0.0007392
+sofa 0 0 -10 636.29 195.54 723.83 307.53 0.0007385
+bed 0 0 -10 547.54 173.15 598.43 242.09 0.0007356
+desk 0 0 -10 526.98 99.66 602.72 235.80 0.0007344
+table 0 0 -10 573.46 189.63 586.23 262.55 0.0007129
+bed 0 0 -10 565.95 174.39 604.04 259.64 0.0007010
+table 0 0 -10 297.85 1.59 405.20 109.49 0.0007000
+table 0 0 -10 629.41 183.02 710.70 307.78 0.0006749
+desk 0 0 -10 524.81 162.99 593.91 199.55 0.0006713
+dresser 0 0 -10 198.01 26.32 551.97 278.84 0.0006538
diff --git a/demo/FasterRCNN_labels/000005.txt b/demo/FasterRCNN_labels/000005.txt
new file mode 100644
index 0000000..6f39031
--- /dev/null
+++ b/demo/FasterRCNN_labels/000005.txt
@@ -0,0 +1,100 @@
+bed 0 0 -10 74.46 161.19 655.18 529.01 0.9930301
+night_stand 0 0 -10 307.95 253.90 406.98 333.71 0.9116214
+night_stand 0 0 -10 78.67 211.75 167.34 267.40 0.8895400
+dresser 0 0 -10 217.50 204.89 338.26 272.76 0.7789017
+night_stand 0 0 -10 599.61 294.51 730.00 521.02 0.7670963
+bed 0 0 -10 0.00 180.33 167.07 364.41 0.6118156
+dresser 0 0 -10 602.82 294.29 730.00 507.44 0.4892671
+night_stand 0 0 -10 59.18 213.01 172.87 324.05 0.3439279
+night_stand 0 0 -10 252.22 263.47 429.98 336.41 0.2388627
+night_stand 0 0 -10 313.69 249.78 400.56 274.47 0.1844822
+night_stand 0 0 -10 7.87 218.95 168.70 355.72 0.1803956
+bed 0 0 -10 0.00 192.08 123.53 292.38 0.1673693
+bed 0 0 -10 0.00 124.02 131.54 322.73 0.1489016
+desk 0 0 -10 233.29 205.17 339.02 268.55 0.1445911
+dresser 0 0 -10 208.65 201.47 281.26 272.65 0.1049450
+dresser 0 0 -10 0.69 127.83 77.90 218.39 0.1000058
+bed 0 0 -10 0.00 130.69 261.16 392.02 0.0775105
+sofa 0 0 -10 114.08 181.48 674.31 511.21 0.0754615
+table 0 0 -10 303.27 254.71 396.68 346.07 0.0692337
+night_stand 0 0 -10 2.17 121.84 78.63 217.26 0.0484133
+night_stand 0 0 -10 223.38 205.29 342.42 273.16 0.0474547
+night_stand 0 0 -10 199.30 203.83 390.00 336.05 0.0394364
+table 0 0 -10 4.90 233.39 138.50 300.65 0.0300534
+dresser 0 0 -10 79.62 213.73 169.02 269.10 0.0266357
+dresser 0 0 -10 1.45 113.48 135.12 305.88 0.0257915
+dresser 0 0 -10 1.42 195.94 161.26 380.18 0.0243817
+sofa 0 0 -10 3.86 207.29 157.26 386.18 0.0230836
+table 0 0 -10 4.18 213.21 162.27 371.29 0.0228784
+night_stand 0 0 -10 114.15 273.30 670.49 530.00 0.0222727
+dresser 0 0 -10 615.61 195.05 727.16 403.88 0.0216420
+desk 0 0 -10 0.63 120.32 79.14 217.40 0.0213965
+bed 0 0 -10 27.51 200.20 358.93 381.50 0.0208348
+dresser 0 0 -10 227.09 150.33 337.72 263.34 0.0207125
+dresser 0 0 -10 48.92 255.63 672.10 530.00 0.0190928
+night_stand 0 0 -10 272.60 220.71 424.71 356.17 0.0185771
+dresser 0 0 -10 645.29 101.36 729.52 338.97 0.0169754
+night_stand 0 0 -10 8.11 210.17 156.54 280.74 0.0162615
+bed 0 0 -10 38.51 211.70 173.92 318.71 0.0158024
+dresser 0 0 -10 683.22 301.04 729.34 523.46 0.0157379
+table 0 0 -10 216.84 204.18 334.22 269.44 0.0145469
+desk 0 0 -10 604.83 287.69 730.00 530.00 0.0133835
+bookshelf 0 0 -10 0.82 123.77 78.44 220.95 0.0125560
+dresser 0 0 -10 207.41 202.81 336.18 336.27 0.0122133
+night_stand 0 0 -10 318.56 237.12 411.15 269.33 0.0118800
+dresser 0 0 -10 60.51 213.34 171.51 326.53 0.0116822
+dresser 0 0 -10 681.49 115.52 725.13 310.26 0.0113607
+night_stand 0 0 -10 318.90 249.52 428.83 287.71 0.0104074
+dresser 0 0 -10 252.59 200.58 310.22 263.65 0.0103685
+table 0 0 -10 5.76 211.58 172.93 283.02 0.0098571
+table 0 0 -10 248.06 266.66 425.32 336.10 0.0097284
+table 0 0 -10 3.34 125.18 76.71 216.89 0.0094840
+bed 0 0 -10 205.50 179.07 552.63 450.11 0.0093813
+sofa 0 0 -10 417.49 181.43 659.67 275.83 0.0088945
+dresser 0 0 -10 305.62 254.78 395.78 345.57 0.0084592
+bed 0 0 -10 589.44 246.66 730.00 530.00 0.0083581
+night_stand 0 0 -10 211.85 199.67 280.92 269.44 0.0082138
+night_stand 0 0 -10 331.04 253.56 381.84 273.45 0.0077677
+sofa 0 0 -10 359.25 188.55 678.47 483.09 0.0075638
+chair 0 0 -10 0.46 117.84 78.54 213.59 0.0073066
+bed 0 0 -10 0.00 116.69 91.67 264.12 0.0072958
+dresser 0 0 -10 704.28 325.02 728.94 508.56 0.0072917
+desk 0 0 -10 303.33 254.87 398.09 346.77 0.0071821
+table 0 0 -10 65.30 205.68 165.23 300.82 0.0060006
+desk 0 0 -10 7.62 213.83 163.26 320.16 0.0057858
+dresser 0 0 -10 574.39 213.49 708.03 498.51 0.0057135
+dresser 0 0 -10 154.16 194.13 256.29 271.52 0.0057010
+chair 0 0 -10 659.04 103.61 729.52 230.18 0.0054716
+dresser 0 0 -10 154.43 125.53 334.84 269.25 0.0050611
+dresser 0 0 -10 171.96 199.19 375.91 288.43 0.0048371
+bed 0 0 -10 294.34 255.78 438.41 339.25 0.0048234
+chair 0 0 -10 302.53 250.81 397.12 344.48 0.0048216
+table 0 0 -10 314.05 249.87 399.65 273.89 0.0047883
+bed 0 0 -10 0.00 84.60 433.06 411.83 0.0046707
+dresser 0 0 -10 645.01 198.43 726.21 304.85 0.0046671
+chair 0 0 -10 217.23 201.59 319.90 268.18 0.0045847
+bed 0 0 -10 211.25 197.63 338.83 270.84 0.0045323
+desk 0 0 -10 208.42 199.87 340.97 302.14 0.0044855
+dresser 0 0 -10 675.41 228.46 725.17 416.18 0.0043830
+sofa 0 0 -10 3.60 122.01 78.55 216.66 0.0043619
+dresser 0 0 -10 252.66 264.85 427.76 336.15 0.0042804
+bookshelf 0 0 -10 2.09 99.29 90.30 268.06 0.0041900
+dresser 0 0 -10 717.85 294.89 729.72 514.43 0.0041356
+bed 0 0 -10 65.61 239.32 430.71 515.92 0.0040306
+dresser 0 0 -10 32.12 206.25 159.94 289.33 0.0039678
+chair 0 0 -10 2.14 124.07 132.06 318.31 0.0039380
+night_stand 0 0 -10 4.45 128.57 134.10 302.65 0.0037985
+sofa 0 0 -10 2.49 204.28 167.40 280.73 0.0037281
+chair 0 0 -10 189.42 196.75 306.38 337.33 0.0034810
+toilet 0 0 -10 302.74 255.20 394.42 351.65 0.0034125
+night_stand 0 0 -10 8.41 224.57 128.73 262.57 0.0032220
+table 0 0 -10 609.40 301.67 727.68 530.00 0.0031929
+table 0 0 -10 313.97 250.40 435.68 326.27 0.0031802
+table 0 0 -10 76.85 213.03 161.91 268.20 0.0031740
+night_stand 0 0 -10 307.89 238.91 391.42 304.79 0.0031300
+bookshelf 0 0 -10 689.89 110.62 727.49 260.70 0.0030572
+night_stand 0 0 -10 8.30 247.14 158.31 301.84 0.0030477
+night_stand 0 0 -10 169.86 128.97 229.23 190.27 0.0030271
+chair 0 0 -10 12.75 202.70 153.58 312.42 0.0029309
+sofa 0 0 -10 618.69 193.05 729.67 322.25 0.0029175
+bookshelf 0 0 -10 263.11 122.92 330.10 210.98 0.0028865
diff --git a/demo/calib/000001.txt b/demo/calib/000001.txt
new file mode 100644
index 0000000..e55dd30
--- /dev/null
+++ b/demo/calib/000001.txt
@@ -0,0 +1,2 @@
+0.97959 0.012593 0.20061 0.012593 0.99223 -0.12377 -0.20061 0.12377 0.97182
+529.5 0 0 0 529.5 0 365 265 1
diff --git a/demo/calib/000002.txt b/demo/calib/000002.txt
new file mode 100644
index 0000000..039edd6
--- /dev/null
+++ b/demo/calib/000002.txt
@@ -0,0 +1,2 @@
+0.97944 0.011827 0.20136 0.011827 0.99319 -0.11586 -0.20136 0.11586 0.97264
+529.5 0 0 0 529.5 0 365 265 1
diff --git a/demo/calib/000003.txt b/demo/calib/000003.txt
new file mode 100644
index 0000000..570bd1f
--- /dev/null
+++ b/demo/calib/000003.txt
@@ -0,0 +1,2 @@
+0.9902 0.005576 0.13957 0.005576 0.99683 -0.079386 -0.13957 0.079386 0.98703
+529.5 0 0 0 529.5 0 365 265 1
diff --git a/demo/calib/000004.txt b/demo/calib/000004.txt
new file mode 100644
index 0000000..0b47cca
--- /dev/null
+++ b/demo/calib/000004.txt
@@ -0,0 +1,2 @@
+0.99999 0.000326 0.003104 0.000326 0.97816 -0.20783 -0.003104 0.20783 0.97816
+529.5 0 0 0 529.5 0 365 265 1
diff --git a/demo/calib/000005.txt b/demo/calib/000005.txt
new file mode 100644
index 0000000..c9a381d
--- /dev/null
+++ b/demo/calib/000005.txt
@@ -0,0 +1,2 @@
+0.99345 0.004112 0.11422 0.004112 0.99742 -0.071675 -0.11422 0.071675 0.99087
+529.5 0 0 0 529.5 0 365 265 1
diff --git a/demo/checkpoint.tar b/demo/checkpoint.tar
new file mode 100644
index 0000000..ea86c46
Binary files /dev/null and b/demo/checkpoint.tar differ
diff --git a/demo/depth/000001.mat b/demo/depth/000001.mat
new file mode 100644
index 0000000..a37e0d5
Binary files /dev/null and b/demo/depth/000001.mat differ
diff --git a/demo/depth/000002.mat b/demo/depth/000002.mat
new file mode 100644
index 0000000..2d0bc51
Binary files /dev/null and b/demo/depth/000002.mat differ
diff --git a/demo/depth/000003.mat b/demo/depth/000003.mat
new file mode 100644
index 0000000..15ebdd3
Binary files /dev/null and b/demo/depth/000003.mat differ
diff --git a/demo/depth/000004.mat b/demo/depth/000004.mat
new file mode 100644
index 0000000..4e336bf
Binary files /dev/null and b/demo/depth/000004.mat differ
diff --git a/demo/depth/000005.mat b/demo/depth/000005.mat
new file mode 100644
index 0000000..9239856
Binary files /dev/null and b/demo/depth/000005.mat differ
diff --git a/demo/images/000001.jpg b/demo/images/000001.jpg
new file mode 100644
index 0000000..d93ac79
Binary files /dev/null and b/demo/images/000001.jpg differ
diff --git a/demo/images/000002.jpg b/demo/images/000002.jpg
new file mode 100644
index 0000000..ee0a61e
Binary files /dev/null and b/demo/images/000002.jpg differ
diff --git a/demo/images/000003.jpg b/demo/images/000003.jpg
new file mode 100644
index 0000000..51cfedc
Binary files /dev/null and b/demo/images/000003.jpg differ
diff --git a/demo/images/000004.jpg b/demo/images/000004.jpg
new file mode 100644
index 0000000..81c8270
Binary files /dev/null and b/demo/images/000004.jpg differ
diff --git a/demo/images/000005.jpg b/demo/images/000005.jpg
new file mode 100644
index 0000000..d91e3e7
Binary files /dev/null and b/demo/images/000005.jpg differ
diff --git a/fasterRCNN_detections.py b/fasterRCNN_detections.py
new file mode 100644
index 0000000..c7cea69
--- /dev/null
+++ b/fasterRCNN_detections.py
@@ -0,0 +1,224 @@
+
+"""Faster R-CNN
+
+Pretrained Faster RCNN on Open Images V4 Dataset with 600 categories.
+Object detection model trained on Open Images V4 with ImageNet pre-trained Inception Resnet V2 as image feature extractor.
+
+Categories of interest from sun rgbd | possible category of Open Images Dataset
+- bed | Bed
+- table | Table
+- sofa | Sofa bed
+- chair | Chair
+- toilet | Toilet
+- desk | Desk
+- dresser | Filing cabinet
+- night_stand | Nightstand
+- bookshelf | Bookcase
+- bathtub | Bathtub
+"""
+
+import os
+# For running inference on the TF-Hub module.
+import tensorflow as tf
+import tensorflow_hub as hub
+
+# For downloading the image.
+import matplotlib.pyplot as plt
+import tempfile
+from six.moves.urllib.request import urlopen
+from six import BytesIO
+
+# For drawing onto the image.
+import numpy as np
+from PIL import Image
+from PIL import ImageColor
+from PIL import ImageDraw
+from PIL import ImageFont
+from PIL import ImageOps
+
+# For filtering out objects of interest from 600 categories
+from collections import defaultdict
+
+# For measuring the inference time.
+import time
+
+# Print Tensorflow version
+print(tf.__version__)
+
+# Check available GPU devices.
+print(tf.test.is_built_with_cuda())
+print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))
+
+
+def display_image(image):
+ fig = plt.figure(figsize=(10, 8))
+ plt.grid(False)
+ plt.imshow(image)
+
+
+def download_and_resize_image(url, new_width=256, new_height=256, display=False):
+ """
+ To test on sample images from the internet
+ """
+ _, filename = tempfile.mkstemp(suffix=".jpg")
+ response = urlopen(url)
+ image_data = response.read()
+ image_data = BytesIO(image_data)
+ pil_image = Image.open(image_data)
+ pil_image = ImageOps.fit(pil_image, (new_width, new_height), Image.ANTIALIAS)
+ pil_image_rgb = pil_image.convert("RGB")
+ pil_image_rgb.save(filename, format="JPEG", quality=90)
+ print("Image downloaded to %s." % filename)
+ if display:
+ display_image(pil_image)
+ return filename
+
+
+def draw_bounding_box_on_image(image, ymin, xmin, ymax, xmax, color, font, thickness=4, display_str_list=()):
+ """
+ Adds a bounding box to an image.
+ """
+ draw = ImageDraw.Draw(image)
+ im_width, im_height = image.size
+
+ # since bbox coordinates are normalized between 0 to 1
+ (left, right, top, bottom) = (xmin * im_width, xmax * im_width, ymin * im_height, ymax * im_height)
+ draw.line([(left, top), (left, bottom), (right, bottom), (right, top), (left, top)], width=thickness, fill=color)
+
+ # If the total height of the display strings added to the top of the bounding
+ # box exceeds the top of the image, stack the strings below the bounding box
+ # instead of above.
+ display_str_heights = [font.getsize(ds)[1] for ds in display_str_list]
+
+ # Each display_str has a top and bottom margin of 0.05x.
+ total_display_str_height = (1 + 2 * 0.05) * sum(display_str_heights)
+
+ if top > total_display_str_height:
+ text_bottom = top
+ else:
+ text_bottom = top + total_display_str_height
+
+ # Reverse list and print from bottom to top.
+ for display_str in display_str_list[::-1]:
+ text_width, text_height = font.getsize(display_str)
+ margin = np.ceil(0.05 * text_height)
+ draw.rectangle([(left, text_bottom - text_height - 2 * margin), (left + text_width, text_bottom)], fill=color)
+ draw.text((left + margin, text_bottom - text_height - margin), display_str, fill="black", font=font)
+ text_bottom -= text_height - 2 * margin
+
+
+def draw_boxes(image, boxes, class_names, scores, max_boxes=10, min_score=0.15):
+ """
+ Overlay labeled boxes on an image with formatted scores and label names.
+ """
+ colors = list(ImageColor.colormap.values())
+
+ try:
+ font = ImageFont.truetype("/usr/share/fonts/truetype/liberation/LiberationSansNarrow-Regular.ttf", 25)
+ except IOError:
+ print("Font not found, using default font.")
+ font = ImageFont.load_default()
+
+ for i in range(min(boxes.shape[0], max_boxes)):
+ if scores[i] >= min_score:
+ ymin, xmin, ymax, xmax = tuple(boxes[i])
+ display_str = "{}: {}%".format(class_names[i].decode("ascii"), int(100 * scores[i]))
+ color = colors[hash(class_names[i]) % len(colors)]
+ image_pil = Image.fromarray(np.uint8(image)).convert("RGB")
+ draw_bounding_box_on_image(image_pil, ymin, xmin, ymax, xmax, color, font, display_str_list=[display_str])
+ np.copyto(image, np.array(image_pil))
+ return image
+
+
+def load_img(path):
+ img = tf.io.read_file(path)
+ img = tf.image.decode_jpeg(img, channels=3)
+ return img
+
+# Main Function which uses pretrained detector from tensorflow hub and inputs the image path and dump directory file path
+def run_detector(detector, path, filePath):
+ img = load_img(path)
+
+ image = img.numpy()
+ image = Image.fromarray(np.uint8(image)).convert("RGB")
+ im_width, im_height = image.size
+
+ converted_img = tf.image.convert_image_dtype(img, tf.float32)[tf.newaxis, ...]
+ start_time = time.time()
+ result = detector(converted_img)
+ end_time = time.time()
+
+ result = {key: value.numpy() for key, value in result.items()}
+
+ # new dictionary which filters out objects of interest as mentioned at the start
+ filter = defaultdict(list)
+ for i in range(len(result["detection_class_entities"])):
+ if result["detection_class_entities"][i] == b"Bed" or \
+ result["detection_class_entities"][i] == b"Kitchen & dining room table" or \
+ result["detection_class_entities"][i] == b"Table" or \
+ result["detection_class_entities"][i] == b"Sofa bed" or \
+ result["detection_class_entities"][i] == b"Chair" or \
+ result["detection_class_entities"][i] == b"Toilet" or \
+ result["detection_class_entities"][i] == b"Filing cabinet" or \
+ result["detection_class_entities"][i] == b"Desk" or \
+ result["detection_class_entities"][i] == b"Nightstand" or \
+ result["detection_class_entities"][i] == b"Bookcase" or \
+ result["detection_class_entities"][i] == b"Bathtub":
+
+ filter["detection_class_entities"].append(result["detection_class_entities"][i])
+ filter["detection_boxes"].append(result["detection_boxes"][i])
+ filter["detection_scores"].append(result["detection_scores"][i])
+
+ # print(filter["detection_class_entities"])
+ # print(filter["detection_boxes"])
+ # print(filter["detection_scores"])
+
+ print("Found %d objects." % len(result["detection_scores"]))
+ print("Inference time: ", end_time - start_time)
+
+ # code to save all detected objects in a local text file (as per ImVoteNet requirements)
+ currentFile = open(filePath, mode='w')
+ for i in range(len(filter["detection_class_entities"])):
+ xmin = filter["detection_boxes"][i][0] * im_width
+ xmax = filter["detection_boxes"][i][2] * im_width
+ ymin = filter["detection_boxes"][i][1] * im_height
+ ymax = filter["detection_boxes"][i][3] * im_height
+ if str(filter["detection_class_entities"][i].decode("ascii")) == 'Bed':
+ className = 'bed'
+ elif str(filter["detection_class_entities"][i].decode("ascii")) == 'Table':
+ className = 'table'
+ elif str(filter["detection_class_entities"][i].decode("ascii")) == 'Sofa bed':
+ className = 'sofa'
+ elif str(filter["detection_class_entities"][i].decode("ascii")) == 'Chair':
+ className = 'chair'
+ elif str(filter["detection_class_entities"][i].decode("ascii")) == 'Toilet':
+ className = 'toilet'
+ elif str(filter["detection_class_entities"][i].decode("ascii")) == 'Desk':
+ className = 'desk'
+ elif str(filter["detection_class_entities"][i].decode("ascii")) == 'Filing Cabinet':
+ className = 'dresser'
+ elif str(filter["detection_class_entities"][i].decode("ascii")) == 'Nightstand':
+ className = 'night_stand'
+ elif str(filter["detection_class_entities"][i].decode("ascii")) == 'Bookcase':
+ className = 'bookshelf'
+ elif str(filter["detection_class_entities"][i].decode("ascii")) == 'Bathtub':
+ className = 'bathtub'
+ currentFile.write(
+ className + ' ' + '0' + ' ' + '0' + ' ' + '-10' + ' ' + str(xmin) + ' ' + str(ymin) + ' ' + str(
+ xmax) + ' ' + str(ymax) + ' ' + str(filter["detection_scores"][i]) + '\n')
+ currentFile.close()
+
+ image_with_boxes = draw_boxes(img.numpy(), np.array(filter["detection_boxes"]),
+ np.array(filter["detection_class_entities"]), np.array(filter["detection_scores"]))
+ display_image(image_with_boxes)
+
+
+module_handle = "https://tfhub.dev/google/faster_rcnn/openimages_v4/inception_resnet_v2/1"
+detector = hub.load(module_handle).signatures['default']
+
+BASE_DIR = os.path.dirname(os.path.abspath(__file__))
+ROOT_DIR = BASE_DIR
+img_path = os.path.join(BASE_DIR, 'demo/image/000001.jpg')
+# path at which resulting text file needs to be dumped
+filePath = os.path.join(BASE_DIR, 'demo/FasterRCNN_labels/textfile.txt')
+run_detector(detector, img_path, filePath)
\ No newline at end of file