You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Thanks a lot for this package! I've been running the bbh benchmarks. Logging the model responses, I'm seeing something weird: The data seems to indicate that the model is outputting lower scores for the correct answers. I'm wondering if there's something I'm missing (e.g. is there a negative sign put into the scores for some reason?). For example, for the phi-3.5 model, under doc_id 0 the label is False and the model response is:
This implies that the logits are higher for True, but the acc_norm column seems to mark the model response as correct. Am I misreading the scores that come with the model responses? I've tried creating a discussion thread here, but figured I'd just ask the team directly.
Thanks,
Linh
The text was updated successfully, but these errors were encountered:
Hi team,
Thanks a lot for this package! I've been running the bbh benchmarks. Logging the model responses, I'm seeing something weird: The data seems to indicate that the model is outputting lower scores for the correct answers. I'm wondering if there's something I'm missing (e.g. is there a negative sign put into the scores for some reason?). For example, for the phi-3.5 model, under doc_id 0 the label is False and the model response is:
This implies that the logits are higher for True, but the acc_norm column seems to mark the model response as correct. Am I misreading the scores that come with the model responses? I've tried creating a discussion thread here, but figured I'd just ask the team directly.
Thanks,
Linh
The text was updated successfully, but these errors were encountered: