OpenOOD-VLM: Benchmarking Generalized OOD Detection with Vision-Language Models

This repository release the official codes of our works for out-of-distribution detection with Vision-Language Models. For reference, we also reproduce some representative methods of OOD detection.

🔥 News

• 21 Nov, 2024: AdaNeg is accepted by NeurIPS 2024, you can easily reproduce our results with Bash

• 23 Aug, 2024: LAPT is accepted by ECCV 2024, you can easily reproduce our results with Bash and Checkpoint

⚙️ Dependencies and Installation

Follow OpenOOD to set up the environment, or use our provided instructions below.

pip install git+https://github.com/YBZH/OpenOOD-VLM

📷 Datasets

We also follow OpenOOD to manage the training and testing datasets. If you only use our evaluator, the benchmarks for evaluation will be automatically downloaded by the evaluator (again check out this tutorial). If you would like to also use OpenOOD-VLM for training, you can get all data with our downloading script. Note that ImageNet-1K training images should be downloaded from its official website.

Besides datasets used in OpenOOD, we also provide evaluation on some popular OOD datasets iNaturalist, SUN, Places, and Texture curated by Huang et al. 2021. Please follow instruction from the this repository to download the subsampled datasets where semantically overlapped classes with ImageNet-1k are removed.

Our codebase accesses the datasets from ./data/ and pretrained models from ./results/checkpoints/ by default.

├── ...
├── data
│   ├── benchmark_imglist
│   ├── images_classic
│   └── images_largescale
├── openood
├── results
│   ├── checkpoints
│   └── ...
├── scripts
├── main.py
├── ...

Supported Datasets for Out-of-Distribution Detection

• BIMCV (A COVID X-Ray Dataset)

  Near-OOD: CT-SCAN, X-Ray-Bone;
  Far-OOD: MNIST, CIFAR-10, Texture, Tiny-ImageNet;
  

• MNIST

  Near-OOD: NotMNIST, FashionMNIST;
  Far-OOD: Texture, CIFAR-10, TinyImageNet, Places365;
  

• CIFAR-10

  Near-OOD: CIFAR-100, TinyImageNet;
  Far-OOD: MNIST, SVHN, Texture, Places365;
  

• CIFAR-100

  Near-OOD: CIFAR-10, TinyImageNet;
  Far-OOD: MNIST, SVHN, Texture, Places365;
  

• ImageNet-200

  Near-OOD: SSB-hard, NINCO;
  Far-OOD: iNaturalist, Texture, OpenImage-O;
  Covariate-Shifted ID: ImageNet-C, ImageNet-R, ImageNet-v2;

• ImageNet-1K

  Near-OOD: SSB-hard, NINCO;
  Far-OOD: iNaturalist, Texture, OpenImage-O;
  Covariate-Shifted ID: ImageNet-C, ImageNet-R, ImageNet-v2;

• ImageNet-1K Traditional Four Datasets

  Far-OOD: iNaturalist, SUN, Places, Texture;
  Covariate-Shifted ID: ImageNet-C, ImageNet-R, ImageNet-v2;

🚀 Training/Testing

We provide training and evaluation scripts for all the methods we support in scripts folder.

• [NeurIPS Official] AdaNeg

sh scripts/ood/adaneg/imagenet.sh
sh scripts/ood/adaneg/plus_LAPT.sh # Apply AdaNeg to the pre-trained LAPT prompt

• [ECCV2024 Official] LAPT with pre-trained Checkpoint

sh scripts/ood/lapt/test.sh

• [NIPS2022 Reproduced] Zero-shot MCM

sh scripts/ood/mcm/official.sh

• [ICLR2024 Reproduced] Zero-shot NegLabel

sh scripts/ood/neglabel/official.sh

📖 Citation

If you find our repository useful for your research, please consider citing our paper:

@article{zhang2024lapt,
  title={LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models},
  author={Zhang, Yabin and Zhu, Wenjie and He, Chenhang and Zhang, Lei},
  journal={Proceedings of the european conference on computer vision (ECCV)},
  year={2024}
}

@article{zhang2024adaneg,
  title={AdaNeg: Adaptive Negative Proxy Guided OOD Detection with Vision-Language Models},
  author={Zhang, Yabin and Zhang, Lei},
  journal={Conference on Neural Information Processing Systems},
  year={2024}
}

🙏 Acknowledgement

• OpenOOD, an extensible codebase with Vision Models only.