v3

LICENSE

@@ -0,0 +1,25 @@
+BSD 2-Clause License
+
+Copyright (c) 2020, Chenfeng Xu
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README.md

@@ -0,0 +1,160 @@
+# SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentation.
+By Chenfeng Xu, Bichen Wu, Zining Wang, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka.
+
+This repository contains a Pytorch implementation of SqueezeSegV3, a state-of-the-art model for LiDAR segmentation. The framework of our SqueezeSegV3 can be found below:
+
+<p align="center">
+    <img src="./figure/framework.png"/ width="750">
+</p>
+
+
+Selected quantitative results of different approaches on the SemanticKITTI dataset (* means KNN post-processing):
+
+|   Method       | mIoU | car | bicycle| motorcycle | truck | person | bicyclist | motorcyclist | road |
+| ---------------|------|-----|--------|------------|-------|--------|-----------|--------------|------|
+| SqueezeSeg     | 29.5 |68.8 |16.0    |4.1         |3.3    |12.9    |13.1       |0.9           |85.4  |
+|SqueezeSegV2    | 39.7 | 81.8|18.5    | 17.9       |13.4   |20.1    |25.1       |3.9           |88.6  |
+| RangeNet21     | 47.4 |85.4 |26.2    |26.5        |18.6   |31.8    |33.6       |4.0           |91.4  |
+| RangeNet53     | 49.9 |86.4 |24.5    |32.7        | 25.5  |36.2    |33.6       |4.7           |**91.8**|
+|SqueezeSegV3-21 |48.8  |84.6 |31.5    |32.4        | 11.3  |39.4    |36.1       |**21.3**      | 90.8 |
+|SqueezeSegV3-53 |52.9  |87.4 |35.2    |33.7        |29.0   | 41.8   |39.1       | 20.1         | **91.8**|
+|SqueezeSegV3-21*|51.6  |89.4 |33.7    |34.9        |11.3   |42.6    |44.9       |21.2          |90.8|
+|SqueezeSegV3-53*|**55.9**|**92.5**|**38.7**|**36.5**|**29.6**|**45.6**|**46.2** | 20.1         | 91.7 |
+
+Visualization results of SqueezeSegV3:
+<p align="center">
+    <img src="./figure/sample3.gif" width="425" height="450" />
+    <img src="./figure/sample4.gif" width="425" height="450" />
+</p>
+
+
+For more details, please refer to our paper: [SqueezeSegV3](https://arxiv.org/abs/2004.01803). The work is a follow-up work to [SqueezeSeg](https://github.com/BichenWuUCB/SqueezeSeg), [SqueezeSegV2](https://github.com/xuanyuzhou98/SqueezeSegV2) and [LATTE](https://github.com/bernwang/latte). If you find this work useful for your research, please consider citing:
+
+```
+@article{xu2020squeezesegv3,
+  title={SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentation},
+  author={Xu, Chenfeng and Wu, Bichen and Wang, Zining and Zhan, Wei and Vajda, Peter and Keutzer, Kurt and Tomizuka, Masayoshi},
+  journal={arXiv preprint arXiv:2004.01803},
+  year={2020}
+}
+```
+
+Related works:
+
+```
+ @inproceedings{wu2018squeezesegv2,
+   title={SqueezeSegV2: Improved Model Structure and Unsupervised Domain Adaptation
+   for Road-Object Segmentation from a LiDAR Point Cloud},
+   author={Wu, Bichen and Zhou, Xuanyu and Zhao, Sicheng and Yue, Xiangyu and Keutzer, Kurt},
+   booktitle={ICRA},
+   year={2019},
+ }
+ 
+@inproceedings{wu2017squeezeseg,
+   title={Squeezeseg: Convolutional neural nets with recurrent crf for real-time road-object segmentation from 3d lidar point cloud},
+   author={Wu, Bichen and Wan, Alvin and Yue, Xiangyu and Keutzer, Kurt},
+   booktitle={ICRA}, 
+   year={2018}
+ }
+ 
+@inproceedings{wang2019latte,
+  title={LATTE: accelerating lidar point cloud annotation via sensor fusion, one-click annotation, and tracking},
+  author={Wang, Bernie and Wu, Virginia and Wu, Bichen and Keutzer, Kurt},
+  booktitle={2019 IEEE Intelligent Transportation Systems Conference (ITSC)},
+  pages={265--272},
+  year={2019},
+  organization={IEEE}
+}
+```
+
+## License
+**SqueezeSegV3** is released under the BSD license (See [LICENSE](https://github.com/chenfengxu714/SqueezeSegV3/blob/master/LICENSE) for details).
+
+## Installation
+
+The instructions are tested on Ubuntu 16.04 with python 3.6 and Pytorch 1.1.0 with GPU support.
+
+* Clone the SqueezeSeg3 repository:
+
+
+```shell
+git clone https://github.com/chenfengxu714/SqueezeSegV3.git
+```
+
+* Use pip to install required Python packages:
+
+
+```shell
+pip install -r requirements.txt
+```
+* The SemanticKITTI dataset can be download [here](http://semantic-kitti.org/dataset.html).
+
+
+## Pre-trained Models
+
+The pre-trained SqueezezSegV3-21 and SqueezeSegV3-53 are avaliable at [Google Drive](https://drive.google.com/drive/folders/1oIZXnMxQPaEINlI11V3kn_kXdSTfUgm6?usp=sharing), you can directly download the two files.
+
+
+## Demo
+
+We provide a demo script:
+```shell
+cd ./src/tasks/semantic/
+python demo.py -m /path/to/model
+```
+You can find the prediction .label files and projected map in ./src/sample_output file, an example is shown below:
+<p align="center">
+    <img src="./figure/000000.png"/ width="800">
+</p>
+
+
+## Inference
+To infer the predictions for the entire dataset:
+
+```shell
+cd ./src/tasks/semantic/
+python infer.py -d /path/to/dataset/ -l /path/for/predictions -m /path/to/model
+```
+
+To visualize the prediction for the sequence point cloud:
+```shell
+python visualize.py -d /path/to/dataset/ -p /path/to/predictions/ -s SQ_Number
+```
+
+
+## Training
+
+```shell
+cd ./src/tasks/semantic/
+```
+
+To train a network (from scratch):
+
+```shell
+python train.py -d /path/to/dataset -ac /config/arch/CHOICE.yaml -l /path/to/log
+```
+
+
+To train a network (from pretrained model):
+
+```shell
+python train.py -d /path/to/dataset -ac /config/arch/CHOICE.yaml -l /path/to/log -p /path/to/pretrained
+```
+
+We can monitor the training process using tensorboard.
+```shell
+tensorboard --logdir /file_path/
+```
+
+## Evaluation
+
+```shell
+python evaluate_iou.py -d /path/to/dataset -p /path/to/predictions/ --split valid
+```
+
+
+## Credits
+We referred to RangeNet++ ([Paper](http://www.ipb.uni-bonn.de/wp-content/papercite-data/pdf/milioto2019iros.pdf), [Code](https://github.com/PRBonn/lidar-bonnetal)) during our development. We thank the authors of RangeNet++ for open-sourcing their code.
+
+
+

requirements.txt

@@ -0,0 +1,12 @@
+# first apt install python3-tk
+numpy==1.14.0
+torchvision==0.2.2.post3
+matplotlib==2.2.3
+tensorflow==1.13.1
+scipy==0.19.1
+torch==1.1.0
+vispy==0.5.3
+opencv_python==4.1.0.25
+opencv_contrib_python==4.1.0.25
+Pillow==6.1.0
+PyYAML==5.1.1

src/backbones/SAC.py

@@ -0,0 +1,178 @@
+# This file was modified from https://github.com/BobLiu20/YOLOv3_PyTorch
+# It needed to be modified in order to accomodate for different strides in the
+from __future__ import division
+import torch
+import torch.nn as nn
+from collections import OrderedDict
+import torch.nn.functional as F
+
+
+class SACBlock(nn.Module):
+  def __init__(self, inplanes, expand1x1_planes, bn_d = 0.1):
+
+    super(SACBlock, self).__init__()
+    self.inplanes = inplanes
+    self.bn_d = bn_d
+
+    self.attention_x = nn.Sequential(
+            nn.Conv2d(3, 9 * self.inplanes, kernel_size = 7, padding = 3),
+            nn.BatchNorm2d(9 * self.inplanes, momentum = 0.1),
+            )
+
+    self.position_mlp_2 = nn.Sequential(
+            nn.Conv2d(9 * self.inplanes, self.inplanes, kernel_size = 1),
+            nn.BatchNorm2d(self.inplanes, momentum = 0.1),
+            nn.ReLU(inplace = True),
+            nn.Conv2d(self.inplanes, self.inplanes, kernel_size = 3, padding = 1),
+            nn.BatchNorm2d(self.inplanes, momentum = 0.1),
+            nn.ReLU(inplace = True),
+            )
+
+  def forward(self, input):
+    xyz = input[0]
+    new_xyz= input[1]
+    feature = input[2]
+    N,C,H,W = feature.size()
+
+    new_feature = F.unfold(feature, kernel_size = 3, padding = 1).view(N, -1, H, W)
+    attention = F.sigmoid(self.attention_x(new_xyz))
+    new_feature = new_feature * attention
+    new_feature = self.position_mlp_2(new_feature)
+    fuse_feature = new_feature + feature
+
+    return xyz, new_xyz, fuse_feature
+
+# ******************************************************************************
+
+# number of layers per model
+model_blocks = {
+    21: [1, 1, 2, 2, 1],
+    53: [1, 2, 8, 8, 4],
+}
+
+
+class Backbone(nn.Module):
+  """
+     Class for DarknetSeg. Subclasses PyTorch's own "nn" module
+  """
+
+  def __init__(self, params):
+    super(Backbone, self).__init__()
+    self.use_range = params["input_depth"]["range"]
+    self.use_xyz = params["input_depth"]["xyz"]
+    self.use_remission = params["input_depth"]["remission"]
+    self.drop_prob = params["dropout"]
+    self.bn_d = params["bn_d"]
+    self.OS = params["OS"]
+    self.layers = params["extra"]["layers"]
+    print("Using squeezesegv3" + str(self.layers) + " Backbone")
+    self.input_depth = 0
+    self.input_idxs = []
+    if self.use_range:
+      self.input_depth += 1
+      self.input_idxs.append(0)
+    if self.use_xyz:
+      self.input_depth += 3
+      self.input_idxs.extend([1, 2, 3])
+    if self.use_remission:
+      self.input_depth += 1
+      self.input_idxs.append(4)
+    print("Depth of backbone input = ", self.input_depth)
+
+    self.strides = [2, 2, 2, 1, 1]
+
+    current_os = 1
+    for s in self.strides:
+      current_os *= s
+    print("Original OS: ", current_os)
+
+    if self.OS > current_os:
+      print("Can't do OS, ", self.OS,
+            " because it is bigger than original ", current_os)
+    else:
+
+      for i, stride in enumerate(reversed(self.strides), 0):
+        if int(current_os) != self.OS:
+          if stride == 2:
+            current_os /= 2
+            self.strides[-1 - i] = 1
+          if int(current_os) == self.OS:
+            break
+      print("New OS: ", int(current_os))
+      print("Strides: ", self.strides)
+
+    assert self.layers in model_blocks.keys()
+
+
+    self.blocks = model_blocks[self.layers]
+
+    self.conv1 = nn.Conv2d(self.input_depth, 32, kernel_size=3,
+                           stride=1, padding=1, bias=False)
+    self.bn1 = nn.BatchNorm2d(32, momentum=self.bn_d)
+    self.relu1 = nn.LeakyReLU(0.1)
+
+    self.enc1 = self._make_enc_layer(SACBlock, [32, 64], self.blocks[0],
+                                     stride=self.strides[0], DS=True, bn_d=self.bn_d)
+    self.enc2 = self._make_enc_layer(SACBlock, [64, 128], self.blocks[1],
+                                     stride=self.strides[1], DS=True, bn_d=self.bn_d)
+    self.enc3 = self._make_enc_layer(SACBlock, [128, 256], self.blocks[2],
+                                     stride=self.strides[2], DS=True, bn_d=self.bn_d)
+    self.enc4 = self._make_enc_layer(SACBlock, [256, 256], self.blocks[3],
+                                     stride=self.strides[3], DS=False, bn_d=self.bn_d)
+    self.enc5 = self._make_enc_layer(SACBlock, [256, 256], self.blocks[4],
+                                     stride=self.strides[4], DS=False, bn_d=self.bn_d)
+
+    self.dropout = nn.Dropout2d(self.drop_prob)
+
+    self.last_channels = 256
+
+  def _make_enc_layer(self, block, planes, blocks, stride, DS, bn_d=0.1):
+    layers = []
+
+    inplanes = planes[0]
+    for i in range(0, blocks):
+      layers.append(("residual_{}".format(i),
+                     block(inplanes, planes,bn_d)))
+    if DS==True:
+        layers.append(("conv", nn.Conv2d(planes[0], planes[1],
+                                     kernel_size=3,
+                                     stride=[1, stride], dilation=1,
+                                     padding=1, bias=False)))
+        layers.append(("bn", nn.BatchNorm2d(planes[1], momentum=bn_d)))
+        layers.append(("relu", nn.LeakyReLU(0.1)))
+
+    return nn.Sequential(OrderedDict(layers))
+
+  def run_layer(self, xyz, feature, layer, skips, os, flag=True):
+    new_xyz = xyz    
+    if flag == True:
+      xyz, new_xyz, y = layer[:-3]([xyz, new_xyz, feature])
+      y = layer[-3:](y)
+      xyz = F.upsample_bilinear(xyz, size=[xyz.size()[2], xyz.size()[3]//2])
+    else:
+      xyz,new_xyz,y = layer([xyz, new_xyz, feature])
+    if y.shape[2] < feature.shape[2] or y.shape[3] < feature.shape[3]:
+      skips[os] = feature.detach()
+      os *= 2
+    feature = self.dropout(y)
+    return xyz, feature, skips, os
+
+  def forward(self, feature):
+    skips = {}
+    os = 1
+    xyz = feature[:,1:4,:,:]
+    feature = self.relu1(self.bn1(self.conv1(feature)))
+
+    xyz,feature, skips, os = self.run_layer(xyz,feature, self.enc1, skips, os)
+    xyz,feature, skips, os = self.run_layer(xyz,feature, self.enc2, skips, os)
+    xyz,feature, skips, os = self.run_layer(xyz,feature, self.enc3, skips, os)
+    xyz,feature, skips, os = self.run_layer(xyz,feature, self.enc4, skips, os, flag=False)
+    xyz,feature, skips, os = self.run_layer(xyz,feature, self.enc5, skips, os, flag=False)
+
+    return feature, skips
+
+  def get_last_depth(self):
+    return self.last_channels
+
+  def get_input_depth(self):
+    return self.input_depth