[EVAL]: Add more African Benchmarks

[dadelani]

Evaluation short description

• Why is this evaluation interesting?
  This focuses on 16 African languages, evaluated on three knowledge QA and reasoning tasks such as AfriMMLU, AfriMGSM and AfriXNLI, human translated from MMLU, MGSM and XNLI respectively.

• How used is it in the community?

Evaluation metadata: IrokoBench

Provide all available

• Paper url: https://arxiv.org/abs/2406.03368
• Github url:
• Dataset url: https://huggingface.co/collections/masakhane/irokobench-665a21b6d4714ed3f81af3b1

Evaluation metadata: Uhura

Provide all available

• Paper url:
• Github url:
• Dataset url: https://huggingface.co/datasets/ebayes/uhura-arc-easy-clean , https://huggingface.co/datasets/ebayes/uhura-truthfulqa-clean

Evaluation metadata: SIB-200

Provide all available

• Paper url: https://aclanthology.org/2024.eacl-long.14/
• Github url: https://github.com/dadelani/sib-200
• Dataset url: https://huggingface.co/datasets/Davlan/sib200

@NathanHB

[clefourrier]

Hi @dadelani , thanks for your issue!

This sounds like a great idea, do you want to add them yourself to the library and open a PR? We've got a guide here for adding tasks and one here specific to multilingual evaluations :)

[hynky1999]

Hi,
Iroko bench is already implemented

See: the tasks.py
Currently only the swahili subset is available, because we don't have correct translations for anchor words, that are required during the evaluation (e.g. answer/question).

If you are a native speaker in any of the languages we would love your help in improving that!
See the guide how to add a translations:

[dadelani]

Great, thank you @clefourrier and @hynky1999 .

For the anchor words, maybe you can use google translate of the keywords. I can provide for Yoruba below:

Question: Ìbéèrè
Answer: Ìdáhùn

[NathanHB]

Great ! do you think you can open a PR for the evals and the keywords ? We will not use google translate as it can be unreliable.

As Clementine said, the documentation to add a translation is here:
https://github.com/huggingface/lighteval/wiki/Contributing-to-multilingual-evaluations