pyserini.search.lucene recall result not good as LuceneSearcher.search

[better629]

I have index the corpus using pyserini.index --collection JsonCollection --generator DefaultLuceneDocumentGenerator --storePositions --storeDocvectors --storeRaw.

And do a seach using both pyserini.search.lucene and LuceneSearcher.search, then check the result with the ground-truth documents.

The result is recall of pyserini.search.lucene is smaller than LuceneSearcher.search. But according to the https://github.com/castorini/pyserini/blob/master/docs/usage-index.md#building-a-bm25-index-direct-java-implementation, it seems that the result should be the same. Anyone know why?

[lintool]

hi @better629 - how are the other metrics beyond recall? have you examined the output ranked list and manually compared?

[lintool]

Yes, different parameters seems like the likely culprit...

[better629]

I use default params with both methods. So, how to keep the two results be same ?

[lintool]

Why not explicitly set the parameters so they are identical, as opposed to relying on the defaults?

[better629]

seems the two implement code is different, so the params maybe different.
Do you have any idea which parameters to set in order to get same results?

[lintool]

k1 and b are the relevant parameters for BM25:

• pyserini.search.lucene: https://github.com/castorini/pyserini/blob/master/pyserini/search/lucene/__main__.py#L87
• LuceneSearcher.search: https://github.com/castorini/pyserini/blob/master/pyserini/search/lucene/_searcher.py#L345
• Defaults for LuceneSearcher (which wraps SimpleSearcher in Anserini) are defined in https://github.com/castorini/anserini/blob/master/src/main/java/io/anserini/search/SearchCollection.java#L265

[better629]

@lintool I found the reason, due to the query string contain \n \r, so has a escaping like from \n to \\n and write to tsv file(if using \n, write to tsv failed). And then, the text from query_iterator is a little different from raw string. This cause the recall result difference.

After input raw string as a param into search.lucene, both got same results.

[lintool]

Glad you resolved the issue!