Major difference in query likelihood retrieval performance when querying a pyserini index vs an indri index

[noamaon]

Hi, I was comparing the performance of pyserini to the Indri toolkit and I got significant differences in performance. I compared NDCG@10, NDCG@5, MAP@50 and P@5 for a sample of 30 queries from the 2019 TREC retrieval challenge.
The corpus I was indexing is msmarco-passage (the first version). For both indices krovetz stemming was applied and the same mu parameter was set when retrieving. I even checked by hand that the text of the indexed passages is identical for both indices. Is this a known issue?
Help will be much appreciated.

[lintool]

Yes, it is not a surprise that different retrieval engines produce slightly different results. How big are your differences in the various metrics?

[noamaon]

I must be missing something, but none of the examples there have a stemmer/tokenizer argument. Apologies for askig again, when I indexed the corpus I simply provided a "-stemmer " argument, is there a similar argument when querying?

[lintool]

On the indexing end, look here: https://github.com/castorini/anserini/blob/master/src/main/java/io/anserini/index/IndexCollection.java

In Lucene stemmers are called Analyzers.

Or you could just follow our guides and use our defaults... and not try to do anything "special" like messing with the stemmer, BM25 instead of QL, etc.

[noamaon]

I am using different parameters (QL, krovetz stemmer) for consistency purposes w.r.t other work from our lab. Is there anything stemming related on the querying end, or should I use the analyzer manually before performing the search? Additionally, even when using the default paramerers (default index stemmer, bm25) there are similar differences in performance when compared to Indri

[lintool]

I would try to reproduce our scores exactly with default parameters (default index stemmer, bm25), then try to change one thing at a time... bm25 to QL, the stemmer, etc. and you'll find out where you went wrong.

[noamaon]

Hi, I found the issue - I think the search method automatically applies default stemming to queries unless stated otherwise, and regardless of the stemmer used for the corpus. Once I manually applied krovetz stemming to the queries and searched over the krovetz stemmed index, with the 'pretokenized' argument, everything worked as expected. Thank you very much for the quick replies, and for maintaining such a useful library!

[lintool]

Happy to help. Can you please close the loop by posting the updated results of Indri vs. Pyserini?

[noamaon]

Of course :)

[lintool]

Okay, this looks as expected! Glad we resolved the issues.