GGSD

Official PyTorch codes for "Open Vocabulary 3D Scene Understanding via Geometry Guided Self-Distillation", ECCV2024

Note！！

Please note that the code has not been fully organized yet. The current code cannot be executed directly. I will remove this note once I have finished organizing it.

Installation

Our code is based on OpenScene, and you can install the environment according to OpenScene or the following commands.

Start by cloning the repo:

git clone https://github.com/Wang-pengfei/GGSD.git
cd GGSD

First of all, you have to make sure that you have all dependencies in place. The simplest way to do so, is to use anaconda.

You can create an anaconda environment called GGSD as below. For linux, you need to install libopenexr-dev before creating the environment.

sudo apt-get install libopenexr-dev # for linux
conda create -n GGSD python=3.8
conda activate GGSD

Step 1: install PyTorch (we tested on 1.7.1, but the following versions should also work):

pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

Step 2: install MinkowskiNet:

sudo apt install build-essential python3-dev libopenblas-dev

If you do not have sudo right, try the following:

conda install openblas-devel -c anaconda

And now install MinkowskiNet:

pip install -U git+https://github.com/NVIDIA/MinkowskiEngine -v --no-deps \
                           --install-option="--force_cuda" \
                           --install-option="--blas=openblas"

If it is still giving you error, please refer to their official installation page.

Step 3: install all the remaining dependencies:

pip install -r requirements.txt

Step 4 (optional): if you need to run multi-view feature fusion with OpenSeg (especially for your own dataset), remember to install:

pip install tensorflow

Data Preparation

We provide the pre-processed 3D&2D data and multi-view fused features for the following datasets:

• ScanNet
• Matterport3D
• nuScenes
• Replica

Pre-processed 3D&2D Data

One can download the pre-processed datasets by running the script below, and following the command line instruction to download the corresponding datasets:

bash scripts/download_dataset.sh

The script will download and unpack data into the folder data/. One can also download the dataset somewhere else, but link to the corresponding folder with the symbolic link:

ln -s /PATH/TO/DOWNLOADED/FOLDER data

Note: 2D processed datasets (e.g. scannet_2d) are only needed if you want to do multi-view feature fusion on your own. If so, please follow the instruction for multi-view fusion.

Multi-view Fused Features

You can run the following to directly download provided fused features:

bash scripts/download_fused_features.sh

Pre-processed superpoint Data

will released

Run

When you have installed the environment and obtained the processed 3D data and multi-view fused features, you are ready to run our OpenScene disilled/ensemble model for 3D semantic segmentation, or distill your own model from scratch.

Distillation

• Start distilling: sh run/distill.sh EXP_NAME CONFIG.yaml

Evaluation

# Run 3D distilled model
sh run/eval.sh out/replica_openseg config/replica/ours_openseg_pretrained.yaml distill

Acknowledgement

We build our code on top of the OpenScene repository.