Skip to content

Latest commit

 

History

History

open_vocabulary_segmentation

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
README.md
README.md
 
 

README.md

Installation

Requirements

  • Linux or macOS with Python ≥ 3.8
  • CUDA>=11.7, lower CUDA versions may result in not successfully built on detectron2
  • Mask2Former

This document provides a simiple use of FC-CLIP on COCONut dataset.

Example virtualenv environment setup for kMaX-DeepLab

pip3 install virtualenv
python3 -m virtualenv fc-clip --python=python3
source fc-clip/bin/activate

# recommened pytorch version, others may not work
pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2

# get fc-clip repo and set up environment
git clone https://github.com/bytedance/fc-clip.git
pip install -r requirements.txt

# it is recommend to use our provided local detectron2.zip to set up detectron2
unzip detectron2.zip
cd detectron2
pip install -e .

# panotic api
pip install git+https://github.com/cocodataset/panopticapi.git

# install the multi-scale deformable conv
cd fcclip/modeling/pixel_decoder/ops
pip install -e . # it is recommended to use pip install instead of sh make.sh which does not work any more.

Inference Demo with Pre-trained Models

cd demo/
python demo.py \
  --input YOUR_IMG_1.jpg YOUR_IMG_2.jpg \
  [--other-options]
  --opts MODEL.WEIGHTS YOUR_MODEL_PATH

The configs are made for training, therefore we need to specify MODEL.WEIGHTS to a model from model zoo for evaluation. This command will run the inference and show visualizations in an OpenCV window.

More details refer to the official repo of fc-clip.

Training & Evaluation in Command Line

To train a model with "train_net.py", first setup the corresponding datasets following datasets/README.md, then run:

python train_net.py --num-gpus 8 \
  --config-file configs/coco/panoptic-segmentation/fcclip/fcclip_convnext_large_eval_ade20k.yaml

To evaluate a model's performance, use

python train_net.py \
  --config-file configs/coco/panoptic-segmentation/fcclip/fcclip_convnext_large_eval_ade20k.yaml \
  --eval-only MODEL.WEIGHTS /path/to/checkpoint_file

Model zoo

ADE20K-150 A-847 PC-459 PC-59 PAS-21
method backbone training set PQ AP_mask mIoU mIoU mIoU mIoU mIoU model
FC-CLIP ConvNeXt-Large COCO 26.8 16.8 34.1 14.8 18.2 58.4 81.8 download
COCONut-S 27.3 17.3 33.8 15.3 20.4 57.5 82.1 download
COCONut-B 27.4 17.4 33.7 15.5 20.1 58.5 82.0 download