Locally reproducible HF-Leaderboard evals

[eldarkurtic]

Hi folks,

I am trying to run HF-Leaderboard (v2) evals locally, and according to the blog https://huggingface.co/spaces/open-llm-leaderboard/blog the scores are normalized and random prediction accuracy is subtracted for all tasks where this is applicable.
Is this by any chance supported in lm-evaluation-harness, and if not, do you have any pointers where to look in order to add this behaviour?

[baberabb]

Hi! Their documentation links to this branch

[eldarkurtic]

I have already tried, but the reported scores are not matching the ones in https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard , and also their branch crashes with --apply_chat_template + vllm backend.
At the same time, this works just fine on main here, which is why I hoped to add these aggregation scores so the leaderboard is fully reproducible locally with lm-evaluation-harness.

[maziyarpanahi]

I also failed to use their repo/branch via vllm. The results are very off!