Question about the results

[peterwan1]

Dear author:
Firstly, thanks for this great work.
When reading the paper, I notice that the baseline results reported in Favicomp are very different from those reported in the original papers, like COMPACT. In COMPACT, Figure 4 shows the result of COMPACT and raw with Contriever as retriever, llama3-8b as reader, topk=5, and the result of COMPACT and raw are very different from those in Favicomp. Is there any difference in the experiment settings that I've missed? Do you know what causes the differences?

[JungDongwon]

Hi @peterwan1,

Thanks for your attention to our paper! In the Figure 4 of the COMPACT paper, the reader they used for the experiment is GPT-3.5-Turbo, which is a much stronger reader than what we used (Llama3-8B-Instruct). Let me know if you have more questions.