StreamPETR with 3dppe Extension

Introduction

This repository is an implementation of StreamPETR with 3dppe.

---

Getting Started

1. Prepare nuScenes dataset and generate 2D annotations and temporal information for training & evaluation. (see streamPETR)

2. Conda env

conda create -n xxx python=3.8 -y
conda activate xxx
pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

pip install flash-attn==0.2.2  # (Tesla v100 is not compatible)

pip install mmcv-full==1.6.0
pip install mmdet==2.28.2
pip install mmsegmentation==0.30.0
git clone https://github.com/open-mmlab/mmdetection3d.git
cd mmdetection3d
git checkout v1.0.0rc6 
pip install -v -e .

Note : make sure that numba 0.53.0 numpy 1.23.5
(if not, reinstall numba==0.53.0).

Catalogue: tree -d -L 1

.
├── ckpts
├── data
├── mmdetection3d
├── projects
└── tools

3. Train & Infer

tools/dist_train.sh [-config] [-num_gpus]

tools/dist_test.sh [-config] [-model] [-num_gpus] --eval bbox

---

Results on NuScenes Val Set

Model	Setting	Pretrain	Lr Schd	Training Time	NDS	mAP	Config	Download
StreamPETR	V2-99-900q-800x320	FCOS3D	24ep	13h	57.1	48.3	config	model/log
Stream3dppe	V2-99-900q-800x320	FCOS3D	24ep	16h	58.45/58.45	49.95/50.04	config	（model1,model2)/(log1,log2)
Stream3dppe_gt_detph	V2-99-900q-800x320	FCOS3D	24ep	22h	61.7	55.3	config	model/log
StreamPETR	V2-99-900q-1600x640	FCOS3D	24ep
Stream3DPPE	V2-99-900q-1600x640	FCOS3D	24ep

Note : Stream3dppe is trained on 4 x RTX 3090 with bs4 ,while Stream3dppe_gt_detph is trained on 4 x RTX 2080Ti with bs2 .

More result please refer to https://github.com/drilistbox/3DPPE.

---

Acknowledgement

Many thanks to the authors of PETR and StreamPETR.

---

Citation

If you find this project useful for your research, please consider citing:

@article{shu20233DPPE,
  title={3DPPE: 3D Point Positional Encoding for Multi-Camera 3D Object Detection Transformers},
  author={Shu, Changyong and Deng, Jiajun and Yu, Fisher and Liu, Yifan},
  journal={arXiv preprint arXiv:2211.14710},
  year={2023}
}