diff --git a/README.md b/README.md index d2ccd9130..4c7919d98 100644 --- a/README.md +++ b/README.md @@ -271,7 +271,10 @@ See individual pages for details! The experiments described below are not associated with rigorous end-to-end regression testing and thus provide a lower standard of reproducibility. For the most part, manual copying and pasting of commands into a shell is required to reproduce our results. -### MS MARCO (V1) +
+MS MARCO V1 + +### MS MARCO V1 + Reproducing [BM25 baselines for MS MARCO Passage Ranking](docs/experiments-msmarco-passage.md) + Reproducing [BM25 baselines for MS MARCO Document Ranking](docs/experiments-msmarco-doc.md) @@ -280,16 +283,28 @@ For the most part, manual copying and pasting of commands into a shell is requir + Reproducing [docTTTTTquery results](docs/experiments-docTTTTTquery.md) (MS MARCO Passage and Document Ranking) + Notes about reproduction issues with [MS MARCO Document Ranking w/ docTTTTTquery](docs/experiments-msmarco-doc-doc2query-details.md) -### MS MARCO (V2) +
+
+MS MARCO V2 + +### MS MARCO V2 + Reproducing [BM25 baselines on the MS MARCO V2 Collections](docs/experiments-msmarco-v2.md) +
+
+TREC-COVID and CORD-19 + ### TREC-COVID and CORD-19 + [Indexing AI2's COVID-19 Open Research Dataset](docs/experiments-cord19.md) + [Baselines for the TREC-COVID Challenge](docs/experiments-covid.md) + [Baselines for the TREC-COVID Challenge using doc2query](docs/experiments-covid-doc2query.md) +
+
+Other Experiments and Features + ### Other Experiments and Features + [Working with the 20 Newsgroups Dataset](docs/experiments-20newsgroups.md) @@ -302,6 +317,8 @@ For the most part, manual copying and pasting of commands into a shell is requir + Runbook for [ECIR 2019 paper on cross-collection relevance feedback](docs/runbook-ecir2019-ccrf.md) + Support for [approximate nearest-neighbor search](docs/approximate-nearestneighbor.md) on dense vectors with inverted indexes +
+ ## 🙋 How Can I Contribute? If you've found Anserini to be helpful, we have a simple request for you to contribute back. diff --git a/docs/regressions/regressions-dl19-passage-cos-dpr-distil.md b/docs/regressions/regressions-dl19-passage-cos-dpr-distil.md index 5ccbad093..085dc3150 100644 --- a/docs/regressions/regressions-dl19-passage-cos-dpr-distil.md +++ b/docs/regressions/regressions-dl19-passage-cos-dpr-distil.md @@ -99,13 +99,13 @@ With the above commands, you should be able to reproduce the following results: | **AP@1000** | **cosDPR-distil**| |:-------------------------------------------------------------------------------------------------------------|-----------| -| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.460 | +| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.458 | | **nDCG@10** | **cosDPR-distil**| -| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.722 | +| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.717 | | **R@100** | **cosDPR-distil**| -| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.609 | +| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.605 | | **R@1000** | **cosDPR-distil**| -| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.807 | +| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.805 | Note that due to the non-deterministic nature of HNSW indexing, results may differ slightly between each experimental run. Nevertheless, scores are generally stable to the third digit after the decimal point. diff --git a/docs/regressions/regressions-dl20-passage-cos-dpr-distil.md b/docs/regressions/regressions-dl20-passage-cos-dpr-distil.md index d7e5a6423..846a98812 100644 --- a/docs/regressions/regressions-dl20-passage-cos-dpr-distil.md +++ b/docs/regressions/regressions-dl20-passage-cos-dpr-distil.md @@ -103,9 +103,9 @@ With the above commands, you should be able to reproduce the following results: | **nDCG@10** | **cosDPR-distil**| | [DL20 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.701 | | **R@100** | **cosDPR-distil**| -| [DL20 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.714 | +| [DL20 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.712 | | **R@1000** | **cosDPR-distil**| -| [DL20 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.844 | +| [DL20 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.843 | Note that due to the non-deterministic nature of HNSW indexing, results may differ slightly between each experimental run. Nevertheless, scores are generally stable to the third digit after the decimal point. diff --git a/docs/regressions/regressions-msmarco-passage-cos-dpr-distil.md b/docs/regressions/regressions-msmarco-passage-cos-dpr-distil.md index 0208be17f..5bb001128 100644 --- a/docs/regressions/regressions-msmarco-passage-cos-dpr-distil.md +++ b/docs/regressions/regressions-msmarco-passage-cos-dpr-distil.md @@ -95,13 +95,13 @@ With the above commands, you should be able to reproduce the following results: | **AP@1000** | **cosDPR-distil**| |:-------------------------------------------------------------------------------------------------------------|-----------| -| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking) | 0.392 | +| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking) | 0.393 | | **RR@10** | **cosDPR-distil**| -| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking) | 0.387 | +| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking) | 0.388 | | **R@100** | **cosDPR-distil**| -| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking) | 0.900 | +| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking) | 0.903 | | **R@1000** | **cosDPR-distil**| -| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking) | 0.970 | +| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking) | 0.974 | Note that due to the non-deterministic nature of HNSW indexing, results may differ slightly between each experimental run. Nevertheless, scores are generally stable to the third digit after the decimal point.