In Splade example for MS Marco evaluation why index 8.8M train passages and evaluate wiht 6980 queries from dev ?

[PrithivirajDamodaran]

I am referring to the docs page titled "Pyserini: SPLADEv2 on MS MARCO V1 Passage Ranking"

Please advice.

[lintool]

Sorry, not sure if I quite understand the question - this is the standard evaluation methodology for MS MARCO...

[lintool]

This is MS MARCO v1. The hgf dataset was added much later. I've never heard of v1.1 and v2.1 until now.

Refer to the official website: https://microsoft.github.io/msmarco/

[PrithivirajDamodaran]

Understood and I have no further questions, thanks for responding.

https://github.com/microsoft/MSMARCO-Passage-Ranking

The download scripts are referring to files with v2.1, so looks like why you are calling v1 is now v2.1 and reflecting in HF as v2.1

Just an fyi I looked at the official website and the repo offices repo:

[lintool]

No, I don't think so - MS MARCO is a collection of different datasets. What you are referring to is the QA dataset. For ranking, "msmarcoranking" is that people use.

[PrithivirajDamodaran]

If you are so sure then HF should be notified about this version discrepancy. May not be your priority, but I will talk to them and respond here for the benefit of the community.

[PrithivirajDamodaran]

Thank you and I would like to take this moment to appreciate all the work with pyserini and related packages - maintaining this is a massive undertaking. Props to you and your team.

Also I read your whacky weights paper :) very interesting.

Happy NY.