Add regressions for OpenAI-ada2 embeddings on MS MARCO passage (#2235)

With Lucene 9.8 upgrade, the 1536d embeddings work now!
castorini · Nov 4, 2023 · d76bb4c · d76bb4c
1 parent 0dc8f8a
commit d76bb4c
Show file tree

Hide file tree

Showing 15 changed files with 877 additions and 14 deletions.
diff --git a/README.md b/README.md
@@ -86,7 +86,8 @@ See individual pages for details!
 | SPLADE++ CoCondenser-SelfDistil             |          [✓](docs/regressions/regressions-msmarco-passage-splade-pp-sd.md)           |          [✓](docs/regressions/regressions-dl19-passage-splade-pp-sd.md)           |          [✓](docs/regressions/regressions-dl20-passage-splade-pp-sd.md)           |
 | SPLADE++ CoCondenser-SelfDistil (ONNX)      |        [✓](docs/regressions/regressions-msmarco-passage-splade-pp-sd-onnx.md)        |        [✓](docs/regressions/regressions-dl19-passage-splade-pp-sd-onnx.md)        |        [✓](docs/regressions/regressions-dl20-passage-splade-pp-sd-onnx.md)        |
 | **Learned Dense**                           |                                                                                      |                                                                                   |                                                                                   |
-| cosDPR-distil                               |         [✓](docs/regressions/regressions-msmarco-passage-cos-dpr-distil.md)          |         [✓](docs/regressions/regressions-dl19-passage-cos-dpr-distil.md)          |         [✓](docs/regressions/regressions-dl20-passage-cos-dpr-distil.md)          |  
+| cosDPR-distil                               |         [✓](docs/regressions/regressions-msmarco-passage-cos-dpr-distil.md)          |         [✓](docs/regressions/regressions-dl19-passage-cos-dpr-distil.md)          |         [✓](docs/regressions/regressions-dl20-passage-cos-dpr-distil.md)          |
+| OpenAI-ada2                                 |           [✓](docs/regressions/regressions-msmarco-passage-openai-ada2.md)           |           [✓](docs/regressions/regressions-dl19-passage-openai-ada2.md)           |           [✓](docs/regressions/regressions-dl20-passage-openai-ada2.md)           |
 
 ### Available Corpora for Download
 
@@ -102,6 +103,7 @@ See individual pages for details!
 | [SPLADE++ CoCondenser-EnsembleDistil](https://rgw.cs.uwaterloo.ca/pyserini/data/msmarco-passage-splade-pp-ed.tar)                         | 4.2 GB | `e489133bdc54ee1e7c62a32aa582bc77` |
 | [SPLADE++ CoCondenser-SelfDistil](https://rgw.cs.uwaterloo.ca/pyserini/data/msmarco-passage-splade-pp-sd.tar)                             | 4.8 GB | `cb7e264222f2bf2221dd2c9d28190be1` |
 | [cosDPR-distil](https://rgw.cs.uwaterloo.ca/pyserini/data/msmarco-passage-cos-dpr-distil.tar)                                             |  57 GB | `e20ffbc8b5e7f760af31298aefeaebbd` |
+| [OpenAI-ada2](https://rgw.cs.uwaterloo.ca/pyserini/data/msmarco-passage-openai-ada2.tar)                                                  | 109 GB | `a4d843d522ff3a3af7edbee789a63402` |
 
 </details>
 <details>

diff --git a/docs/experiments-msmarco-passage-openai-ada2.md b/docs/experiments-msmarco-passage-openai-ada2.md
@@ -1,4 +1,4 @@
-# Anserini: OpenAI-ada2 Embeddings for MS MARCO Passage Ranking
+# Anserini: OpenAI-ada2 Embeddings for MS MARCO Passage
 
 This guide explains how to reproduce experiments with OpenAI-ada2 emebddings on the [MS MARCO passage ranking task](https://github.com/microsoft/MSMARCO-Passage-Ranking).
 In these experiments, we are using pre-encoded queries (i.e., cached results of query embeddings).

diff --git a/docs/regressions/regressions-dl19-passage-openai-ada2.md b/docs/regressions/regressions-dl19-passage-openai-ada2.md
@@ -0,0 +1,119 @@
+# Anserini Regressions: TREC 2019 Deep Learning Track (Passage)
+
+**Model**: OpenAI-ada2 embeddings (using pre-encoded queries) with HNSW indexes
+
+This page describes regression experiments, integrated into Anserini's regression testing framework, using OpenAI-ada2 embeddings on the [TREC 2019 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper:
+
+> Jimmy Lin, Ronak Pradeep, Tommaso Teofili, and Jasper Xian. [Vector Search with OpenAI Embeddings: Lucene Is All You Need.](https://arxiv.org/abs/2308.14963) _arXiv:2308.14963_, 2023.
+
+In these experiments, we are using pre-encoded queries (i.e., cached results of query encoding).
+
+Note that the NIST relevance judgments provide far more relevant passages per topic, unlike the "sparse" judgments provided by Microsoft (these are sometimes called "dense" judgments to emphasize this contrast).
+For additional instructions on working with MS MARCO passage collection, refer to [this page](experiments-msmarco-passage.md).
+
+The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/dl19-passage-openai-ada2.yaml).
+Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/dl19-passage-openai-ada2.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation.
+
+From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end:
+
+```bash
+python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-openai-ada2
+```
+
+We make available a version of the MS MARCO Passage Corpus that has already been encoded with the OpenAI-ada2 embedding model.
+
+From any machine, the following command will download the corpus and perform the complete regression, end to end:
+
+```bash
+python src/main/python/run_regression.py --download --index --verify --search --regression dl19-passage-openai-ada2
+```
+
+The `run_regression.py` script automates the following steps, but if you want to perform each step manually, simply copy/paste from the commands below and you'll obtain the same regression results.
+
+## Corpus Download
+
+Download the corpus and unpack into `collections/`:
+
+```bash
+wget https://rgw.cs.uwaterloo.ca/pyserini/data/msmarco-passage-openai-ada2.tar -P collections/
+tar xvf collections/msmarco-passage-openai-ada2.tar -C collections/
+```
+
+To confirm, `msmarco-passage-openai-ada2.tar` is 109 GB and has MD5 checksum `a4d843d522ff3a3af7edbee789a63402`.
+With the corpus downloaded, the following command will perform the remaining steps below:
+
+```bash
+python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-openai-ada2 \
+  --corpus-path collections/msmarco-passage-openai-ada2
+```
+
+## Indexing
+
+Sample indexing command, building HNSW indexes:
+
+```bash
+target/appassembler/bin/IndexHnswDenseVectors \
+  -collection JsonDenseVectorCollection \
+  -input /path/to/msmarco-passage-openai-ada2 \
+  -index indexes/lucene-hnsw.msmarco-passage-openai-ada2/ \
+  -generator LuceneDenseVectorDocumentGenerator \
+  -threads 16 -M 16 -efC 100 -memorybuffer 65536 \
+  >& logs/log.msmarco-passage-openai-ada2 &
+```
+
+The path `/path/to/msmarco-passage-openai-ada2/` should point to the corpus downloaded above.
+
+Upon completion, we should have an index with 8,841,823 documents.
+
+<!-- For additional details, see explanation of [common indexing options](common-indexing-options.md). -->
+
+## Retrieval
+
+Topics and qrels are stored [here](https://github.com/castorini/anserini-tools/tree/master/topics-and-qrels), which is linked to the Anserini repo as a submodule.
+The regression experiments here evaluate on the 43 topics for which NIST has provided judgments as part of the TREC 2019 Deep Learning Track.
+The original data can be found [here](https://trec.nist.gov/data/deep2019.html).
+
+After indexing has completed, you should be able to perform retrieval as follows:
+
+```bash
+target/appassembler/bin/SearchHnswDenseVectors \
+  -index indexes/lucene-hnsw.msmarco-passage-openai-ada2/ \
+  -topics tools/topics-and-qrels/topics.dl19-passage.openai-ada2.jsonl.gz \
+  -topicreader JsonIntVector \
+  -output runs/run.msmarco-passage-openai-ada2.openai-ada2.topics.dl19-passage.openai-ada2.jsonl.txt \
+  -querygenerator VectorQueryGenerator -topicfield vector -threads 16 -hits 1000 -efSearch 1000 &
+```
+
+Evaluation can be performed using `trec_eval`:
+
+```bash
+tools/eval/trec_eval.9.0.4/trec_eval -m map -c -l 2 tools/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage-openai-ada2.openai-ada2.topics.dl19-passage.openai-ada2.jsonl.txt
+tools/eval/trec_eval.9.0.4/trec_eval -m ndcg_cut.10 -c tools/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage-openai-ada2.openai-ada2.topics.dl19-passage.openai-ada2.jsonl.txt
+tools/eval/trec_eval.9.0.4/trec_eval -m recall.100 -c -l 2 tools/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage-openai-ada2.openai-ada2.topics.dl19-passage.openai-ada2.jsonl.txt
+tools/eval/trec_eval.9.0.4/trec_eval -m recall.1000 -c -l 2 tools/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage-openai-ada2.openai-ada2.topics.dl19-passage.openai-ada2.jsonl.txt
+```
+
+## Effectiveness
+
+With the above commands, you should be able to reproduce the following results:
+
+| **AP@1000**                                                                                                  | **OpenAI-ada2**|
+|:-------------------------------------------------------------------------------------------------------------|-----------|
+| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html)                                                   | 0.479     |
+| **nDCG@10**                                                                                                  | **OpenAI-ada2**|
+| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html)                                                   | 0.704     |
+| **R@100**                                                                                                    | **OpenAI-ada2**|
+| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html)                                                   | 0.624     |
+| **R@1000**                                                                                                   | **OpenAI-ada2**|
+| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html)                                                   | 0.857     |
+
+Note that due to the non-deterministic nature of HNSW indexing, results may differ slightly between each experimental run.
+Nevertheless, scores are generally stable to the third digit after the decimal point.
+
+Also note that retrieval metrics are computed to depth 1000 hits per query (as opposed to 100 hits per query for document ranking).
+Also, for computing nDCG, remember that we keep qrels of _all_ relevance grades, whereas for other metrics (e.g., AP), relevance grade 1 is considered not relevant (i.e., use the `-l 2` option in `trec_eval`).
+The experimental results reported here are directly comparable to the results reported in the [track overview paper](https://arxiv.org/abs/2003.07820).
+
+## Reproduction Log[*](reproducibility.md)
+
+To add to this reproduction log, modify [this template](../../src/main/resources/docgen/templates/dl19-passage-openai-ada2.template) and run `bin/build.sh` to rebuild the documentation.
diff --git a/docs/regressions/regressions-dl20-passage-openai-ada2.md b/docs/regressions/regressions-dl20-passage-openai-ada2.md
@@ -0,0 +1,119 @@
+# Anserini Regressions: TREC 2020 Deep Learning Track (Passage)
+
+**Model**: OpenAI-ada2 embeddings (using pre-encoded queries) with HNSW indexes
+
+This page describes regression experiments, integrated into Anserini's regression testing framework, using OpenAI-ada2 embeddings on the [TREC 2020 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper:
+
+> Jimmy Lin, Ronak Pradeep, Tommaso Teofili, and Jasper Xian. [Vector Search with OpenAI Embeddings: Lucene Is All You Need.](https://arxiv.org/abs/2308.14963) _arXiv:2308.14963_, 2023.
+
+In these experiments, we are using pre-encoded queries (i.e., cached results of query encoding).
+
+Note that the NIST relevance judgments provide far more relevant passages per topic, unlike the "sparse" judgments provided by Microsoft (these are sometimes called "dense" judgments to emphasize this contrast).
+For additional instructions on working with MS MARCO passage collection, refer to [this page](experiments-msmarco-passage.md).
+
+The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/dl20-passage-openai-ada2.yaml).
+Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/dl20-passage-openai-ada2.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation.
+
+From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end:
+
+```bash
+python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-openai-ada2
+```
+
+We make available a version of the MS MARCO Passage Corpus that has already been encoded with the OpenAI-ada2 embedding model.
+
+From any machine, the following command will download the corpus and perform the complete regression, end to end:
+
+```bash
+python src/main/python/run_regression.py --download --index --verify --search --regression dl20-passage-openai-ada2
+```
+
+The `run_regression.py` script automates the following steps, but if you want to perform each step manually, simply copy/paste from the commands below and you'll obtain the same regression results.
+
+## Corpus Download
+
+Download the corpus and unpack into `collections/`:
+
+```bash
+wget https://rgw.cs.uwaterloo.ca/pyserini/data/msmarco-passage-openai-ada2.tar -P collections/
+tar xvf collections/msmarco-passage-openai-ada2.tar -C collections/
+```
+
+To confirm, `msmarco-passage-openai-ada2.tar` is 109 GB and has MD5 checksum `a4d843d522ff3a3af7edbee789a63402`.
+With the corpus downloaded, the following command will perform the remaining steps below:
+
+```bash
+python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-openai-ada2 \
+  --corpus-path collections/msmarco-passage-openai-ada2
+```
+
+## Indexing
+
+Sample indexing command, building HNSW indexes:
+
+```bash
+target/appassembler/bin/IndexHnswDenseVectors \
+  -collection JsonDenseVectorCollection \
+  -input /path/to/msmarco-passage-openai-ada2 \
+  -index indexes/lucene-hnsw.msmarco-passage-openai-ada2/ \
+  -generator LuceneDenseVectorDocumentGenerator \
+  -threads 16 -M 16 -efC 100 -memorybuffer 65536 \
+  >& logs/log.msmarco-passage-openai-ada2 &
+```
+
+The path `/path/to/msmarco-passage-openai-ada2/` should point to the corpus downloaded above.
+
+Upon completion, we should have an index with 8,841,823 documents.
+
+<!-- For additional details, see explanation of [common indexing options](common-indexing-options.md). -->
+
+## Retrieval
+
+Topics and qrels are stored [here](https://github.com/castorini/anserini-tools/tree/master/topics-and-qrels), which is linked to the Anserini repo as a submodule.
+The regression experiments here evaluate on the 54 topics for which NIST has provided judgments as part of the TREC 2020 Deep Learning Track.
+The original data can be found [here](https://trec.nist.gov/data/deep2020.html).
+
+After indexing has completed, you should be able to perform retrieval as follows:
+
+```bash
+target/appassembler/bin/SearchHnswDenseVectors \
+  -index indexes/lucene-hnsw.msmarco-passage-openai-ada2/ \
+  -topics tools/topics-and-qrels/topics.dl20-passage.openai-ada2.jsonl.gz \
+  -topicreader JsonIntVector \
+  -output runs/run.msmarco-passage-openai-ada2.openai-ada2.topics.dl20-passage.openai-ada2.jsonl.txt \
+  -querygenerator VectorQueryGenerator -topicfield vector -threads 16 -hits 1000 -efSearch 1000 &
+```
+
+Evaluation can be performed using `trec_eval`:
+
+```bash
+tools/eval/trec_eval.9.0.4/trec_eval -m map -c -l 2 tools/topics-and-qrels/qrels.dl20-passage.txt runs/run.msmarco-passage-openai-ada2.openai-ada2.topics.dl20-passage.openai-ada2.jsonl.txt
+tools/eval/trec_eval.9.0.4/trec_eval -m ndcg_cut.10 -c tools/topics-and-qrels/qrels.dl20-passage.txt runs/run.msmarco-passage-openai-ada2.openai-ada2.topics.dl20-passage.openai-ada2.jsonl.txt
+tools/eval/trec_eval.9.0.4/trec_eval -m recall.100 -c -l 2 tools/topics-and-qrels/qrels.dl20-passage.txt runs/run.msmarco-passage-openai-ada2.openai-ada2.topics.dl20-passage.openai-ada2.jsonl.txt
+tools/eval/trec_eval.9.0.4/trec_eval -m recall.1000 -c -l 2 tools/topics-and-qrels/qrels.dl20-passage.txt runs/run.msmarco-passage-openai-ada2.openai-ada2.topics.dl20-passage.openai-ada2.jsonl.txt
+```
+
+## Effectiveness
+
+With the above commands, you should be able to reproduce the following results:
+
+| **AP@1000**                                                                                                  | **OpenAI-ada2**|
+|:-------------------------------------------------------------------------------------------------------------|-----------|
+| [DL20 (Passage)](https://trec.nist.gov/data/deep2020.html)                                                   | 0.477     |
+| **nDCG@10**                                                                                                  | **OpenAI-ada2**|
+| [DL20 (Passage)](https://trec.nist.gov/data/deep2020.html)                                                   | 0.676     |
+| **R@100**                                                                                                    | **OpenAI-ada2**|
+| [DL20 (Passage)](https://trec.nist.gov/data/deep2020.html)                                                   | 0.723     |
+| **R@1000**                                                                                                   | **OpenAI-ada2**|
+| [DL20 (Passage)](https://trec.nist.gov/data/deep2020.html)                                                   | 0.867     |
+
+Note that due to the non-deterministic nature of HNSW indexing, results may differ slightly between each experimental run.
+Nevertheless, scores are generally stable to the third digit after the decimal point.
+
+Also note that retrieval metrics are computed to depth 1000 hits per query (as opposed to 100 hits per query for document ranking).
+Also, for computing nDCG, remember that we keep qrels of _all_ relevance grades, whereas for other metrics (e.g., AP), relevance grade 1 is considered not relevant (i.e., use the `-l 2` option in `trec_eval`).
+The experimental results reported here are directly comparable to the results reported in the [track overview paper](https://arxiv.org/abs/2003.07820).
+
+## Reproduction Log[*](reproducibility.md)
+
+To add to this reproduction log, modify [this template](../../src/main/resources/docgen/templates/dl20-passage-openai-ada2.template) and run `bin/build.sh` to rebuild the documentation.