SceneGraphFusion

Authors: Shun-Cheng Wu, Johanna Wald, Keisuke Tateno, Nassir Navab and Federico Tombari

This is the SceneGraphFusion c++ framework. For the network part of the code, check here.

Prerequisites

git, cmake

apt update
apt install git
# cmake
sudo apt install python3-pip
pip3 install cmake
# create ssh key and associate to your account in order to clone this project.
# * Generate new ssh key [link](https://docs.github.com/en/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent)
# * Adding a new ssh key to your account [link](https://docs.github.com/en/github/authenticating-to-github/adding-a-new-ssh-key-to-your-github-account)

OpenCV, Eigen

# Eigen3 3.3.90  
# OpenCV 4.0.0  
# This project will try to install them locally if they were not found.

Assimp

loading meshes for online rendered view generation.

apt install libassimp-dev

Optional dependencies

With GUI:

# OpenGL & Glfw3
apt install freeglut3-dev libglfw3-dev

With graph prediction:

# Onnxruntime
# See https://github.com/microsoft/onnxruntime for more information
git clone --recursive -b v1.8.2 https://github.com/microsoft/onnxruntime
cd onnxruntime
# For Linux
./build.sh --config RelWithDebInfo --build_shared_lib --parallel
cd build/Linux/RelWithDebInfo
sudo make install

BUILD

build the basic segmentation system

git clone {this_repo}
cd SceneGraphFusion
git submodule init
git submodule update
mkdir build
cd build
cmake ..
make

build with GUI or graph prediction, pass these options in cmake:

cmake -DBUILD_GRAPHPRED=ON -DBUILD_GUI=ON ..

Run

./exe_GraphSLAM --pth_in path/to/3RScan/squence/
# or with GUI
./exe_GraphSLAM_GUI --pth_in path/to/3RScan/squence/
# to see usage and options 
./exe_GraphSLAM --help
# to run with graph prediction (need to build with graph predict)
./exe_GraphSLAM_GUI --pth_in path/to/3RScan/squence/ --pth_model /path/to/the/traced/model/folder/

We provide data loader for 3RScan dataset and ScanNet. In default, we use rendered view for both dataset. The rendered view will be generated during on time.

you can download the traced model of our pretrained network here. The model is trained with 20 NYUv2 object classes used in ScanNet benchmark, and with 8 support types of predicates.

Example:

To run our system with a test sequence, run bash prepare_example_data.sh. It will download an example sequence provided from 3RScan repository. Then run

cd bin
./exe_GraphSLAM --pth_in ../data/3RScan/4acaebcc-6c10-2a2a-858b-29c7e4fb410d/sequence/

Or using the full sequence from either ScanNet or 3RScan

# For 3RScan:
./exe_GraphSLAM --pth_in [path/to/3RScan]/[sequence_id]/sequence/
# For ScanNet
./exe_GraphSLAM --pth_in [path/to/scannet]/[scan_id]/[scan_id].sens
# Run with graph prediction
./exe_GraphSLAM --pth_in ./path/to/data --pth_model /path/to/model

Run with example sequence

If you would like to try out the code without downloading the 3RScan & 3DSSG datset, you can clone the repository of 3RScan which consist of two example sequences.

git clone [email protected]:WaldJohannaU/3RScan.git
cd 3RSan
bash setup.sh

Then cd to the exe_GraphSLAM executable directory and do

./exe_GraphSLAM --pth_in /3RScan/data/3RScan/4acaebcc-6c10-2a2a-858b-29c7e4fb410d/sequence/
# with graph_pred 
./exe_GraphSLAM --pth_in /3RScan/data/3RScan/4acaebcc-6c10-2a2a-858b-29c7e4fb410d/sequence/ --pth_model /CVPR21_traced/traced/

License

Paper

If you find the code useful please consider citing our paper:

@inproceedings{Wu2021,
    title = {{SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences}},
    author = {Shun-Cheng Wu and Johanna Wald and Keisuke Tateno and Nassir Navab and Federico Tombari},
    booktitle = {Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    year = {2021}
}

Acknowledgements

This work is supported by the German Research Foundation (DFG, project number 407378162) and the Bavarian State Ministry of Education, Science and the Arts in the framework of the Centre Digitisation Bavaria (ZD.B).

Troubleshooting

In some platforms the CMake config for Assimp is called assimpConfig.cmake and in others AssimpConfig.cmake. This mismatch can lead to CMake not finding the library despite being available in the system. If this is the case, edit cmake/UseAssimp.cmake to look like this:

find_package(assimp QUIET)
OPTION(WITH_ASSIMP "Build with Assimp support?" ${assimp_FOUND})

IF(WITH_ASSIMP)
    MESSAGE(STATUS "WITH Assimp")
    find_package(assimp REQUIRED)
    INCLUDE_DIRECTORIES(${assimp_INCLUDE_DIRS})
ENDIF()

See assimp/assimp#3455, microsoft/vcpkg#14256 for more details