Learning to Find Good Correspondences (CVPR 2018)

This repository is a reference implementation for K. Yi*, E. Trulls*, Y. Ono, V. Lepetit, M. Salzmann, and P. Fua, "Learning to Find Good Correspondences", CVPR 2018 (* equal contributions). If you use this code in your research, please cite the paper.

Installation

This code base is based on Python3. For more details on the required libraries, see requirements.txt. You can also easily prepare this by doing

pip install -r requirements.txt

Preparing data

Download the brown_bm sequence and st_peters sequence and extract them in the datasets directory. For example, for st_peters you should have a directory looking like ./datasets/st_peters_square/train. Also download reichstag dataset for testing, which is quite small!

Once the datasets are downloaded, run dump_data.py to prepare the datasets. The following commands should be run.

./dump_data.py --data_tr=st_peters --data_va=st_peters --data_te=st_peters
./dump_data.py --data_tr=brown_bm_3_05 --data_va=brown_bm_3_05 --data_te=brown_bm_3_05
./dump_data.py --data_tr=reichstag --data_va=reichstag --data_te=reichstag

Hang on tight, this would take a while.

Training

While we also provide our trained models, you can also easily train your own models. Simply run:

./main.py --run_mode=train

See config.py for more options in running the software. Try it yourself. Nearly all parameters that we changed in the paper should be there.

For designating datasets, modify data_tr, data_va, and data_te to your liking. Also you can simply do something like st_peters.brown_bm_3_05 to train with the combined datasets.

The default place to store the results is ./logs. To change this, use res_dir to set the base directory, log_dir for the suffix for the training configurations. test_log_dir is used to if you want to change the suffix for storing results. For example, log_dir can store the training configuration, and test_log_dir can store which training configuration is used on which testing dataset.

Testing

Again, testing is quite simple. After training is done, run:

./main.py --run_mode=test

Or if you simply want to test the pretrained model on your dataset, you can:

./main.py --run_mode=test --res_dir="./" --log_dir="models" --test_log_dir="results" --data_tr="st_peters.brown_bm_3_05" --data_va="reichstag" --data_te="reichstag" 

# Or use following command to save an additional numpy structured output (Thanks to GrumpyZhou)
./main.py --run_mode=test_simple --res_dir="./" --log_dir="models" --test_log_dir="results"  --data_va="reichstag" --data_te="reichstag"

This time, it should only take about 30 seconds since reichstag is tiny, but depending on the dataset, this might take a while (roughly seven minutes on st_peters). This is because you are testing on the entire test dataset. One thing to note is that, as written on the paper, most the computation is done in the CPU, and GPU is not really necessary for testing.

Also, as of now, there is no stand-alone testing code for a single image pair. The current code-base also evaluates testing on the validation dataset as well, since the only validation performed while training is with the eight-point algorithm. We welcome contributions for untying this mess we have.

Once run, you'll find a TensorBoard log files and some txt files that contain the results of the expriments.

Notes on implementation

For the dataset generation, we used OpenCV SIFT with fast-math flag on. We've noticed that when using opencv-contrib-python package from pip, you get different results, slightly different from the paper.