Unclear how to interpret the relevance document

[lkurlandski]

Could use some documentation about what athome4.qrel.sample actually contains.

[nims11]

The qrel file is the standard qrel format used by TREC: https://trec.nist.gov/data/qrels_eng/

[lkurlandski]

Okay I thought it was something along those lines, however, not every document in athome4_sample.tgz is represented in the athome4.qrel.sample document. For example, 000018, 000022, 000055, 000093, and 000094 are in athome4_sample.tgz but not in athome4.qrel.sample.

I'm guessing this is because "Documents not occurring in the qrels file were not judged by the human assessor and are assumed to be irrelevant in the evaluations used in TREC". Just want to make sure I'm not missing something obvious.

Nice repo, and thank you for your help.

[nims11]

Qrels are ground truth files which usually contain documents for which a judgment was made. So if a document was not judged, it will not show up in the file. Depending on the use case, it is common to assume unjudged documents as not relevant.

…

On Mon, 20 Sep 2021, 13:48 Luke-Kurlandski-TCNJ, ***@***.***> wrote: Okay I thought it was something along those lines, however, not every document in athome4_sample.tgz is represented in the athome4.qrel.sample document. For example, 000018, 000022, 000055, 000093, and 000094 are in athome4_sample.tgz but not in athome4.qrel.sample. Thank you for your help. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJYT55EGV5H5U36VANFPZ3UC4UR3ANCNFSM5EJOM24Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[lkurlandski]

Great thank you!

[Kaotic3]

I see this is all very old now - but a couple of things to note.

Firstly, the QREL document is not conformant to the QREL standard - it has 2 in the Relevancy column, and I can't really see why that would be.

Secondly, the QREL document is inaccurate. Here is a demonstration of that, I include a few surrounding documents so you can see the 2 as well.

401 0 088668 2
401 0 008416 1
401 0 019276 2
401 0 020205 2

401 0 008416 1 - this has a 1, indicating relevancy to the Topic 401 - which is the Olympic Bid topic.

The document is about a student using campus websites inappropriately - but also discusses the individual holding an "Olympics sports day" for kids in the community - this is not a bid for the Olympics.

It isn't discussing the bid for the Olympics, it is literally nothing to do with that at all.

But it is marked 1. It should be marked 0.

I only put this here, so that people like myself who stumble across it - know that it may be useful for somethings, but it is not a reliable dataset which you can benchmark against.