Codes and datasets of WWW2020 Paper Selective Weak Supervision for Neural Information Retrieval. Paper
Model | Method | ClueWeb09 NDCG@20 | ClueWeb09 ERR@20 | Robust04 NDCG@20 | Robust04 ERR@20 | ClueWeb12 NDCG@20 | ClueWeb12 ERR@20 |
---|---|---|---|---|---|---|---|
Conv-KNRM | No Weak Supervision | 0.2873 | 0.1597 | 0.4267 | 0.1168 | 0.1123 | 0.0915 |
ReInfoSelect | 0.3096 | 0.1611 | 0.4423 | 0.1202 | 0.1225 | 0.1044 | |
TK | No Weak Supervision | 0.3003 | 0.1577 | 0.4273 | 0.1163 | 0.1192 | 0.0991 |
ReInfoSelect | 0.3103 | 0.1626 | 0.4320 | 0.1183 | 0.1297 | 0.1043 | |
EDRM | No Weak Supervision | 0.2922 | 0.1642 | 0.4263 | 0.1158 | 0.1119 | 0.0910 |
ReInfoSelect | 0.3163 | 0.1794 | 0.4396 | 0.1208 | 0.1215 | 0.0980 | |
BERT | No Weak Supervision | 0.2999 | 0.1631 | 0.4258 | 0.1163 | 0.1190 | 0.0963 |
ReInfoSelect | 0.3261 | 0.1669 | 0.4500 | 0.1220 | 0.1276 | 0.0997 |
More results are available in results.
Data can be downloaded from Datasets.
Datasets | Queries/Anchors | Query/Anchor-Doc Pairs | Released Files |
---|---|---|---|
Weak Supervision | 100K | 6.75M | Anchors, A-D Relations |
ClueWeb09-B | 200 | 47.1K | Queries, Q-D Relations, SDM scores |
Robust04 | 249 | 311K | Queries, Q-D Relations, SDM scores |
ClueWeb12-B13 | 100 | 28.9K | Queries, Q-D Relations, SDM scores |
As we cannot release the document contents, the document IDs are used instead.
python == 3.7
torch >= 1.0.0
To run ReInfoSelect, please install all requirements.
pip install -r requirements.txt
cd docker
docker build -t reinfoselect_official:v0.1 .
wget http://nlp.stanford.edu/data/glove.6B.zip -P ./data
unzip ./data/glove.6B.zip -d ./data
wget https://msmarco.blob.core.windows.net/msmarcoranking/triples.train.small.tar.gz -P ./data
tar -zxvf triples.train.small.tar.gz -C ./data
First, please prepare your data in recommended format. The MS MARCO or ClueWeb09 corpus may be too large for your machine, you can change -max_input to set max input numbers of training instances.
For cknrm training, set -model to cknrm, for bert training, set to bert, the batch_size for bert may need to set smaller.
sh train.sh
Then, concatenate the neural features with retrieval (SDM or BM25) score and run Coor-Ascent using RankLib.
sh coor_ascent.sh
For inference:
sh inference.sh
For ensemble:
sh ensemble.sh
Please cite our paper if you find it helpful.
@inproceedings{Zhang2020SelectiveWS,
title = {Selective Weak Supervision for Neural Information Retrieval},
author = {Kaitao Zhang and Chenyan Xiong and Zhenghao Liu and Zhiyuan Liu},
booktitle = {{WWW} '20: The Web Conference 2020},
year = {2020}
}
If you have any questions, suggestions and bug reports, please email [email protected].