Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gptjudge empty response handling #34

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

gptjudge empty response handling #34

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c086d22
catch empty response
snova-nidhih Oct 28, 2024
b1c8e92
remove extras
snova-nidhih Oct 28, 2024
34fa0c0
isort
snova-nidhih Oct 28, 2024
8005c3d
black lint
snova-nidhih Oct 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions lmms_eval/api/metrics.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
# the code is adapted from https://github.com/EleutherAI/lm-evaluation-harness
import logging
import math
import os
import random
import re
import string
import time
from collections.abc import Iterable
from typing import List

import numpy as np
import sacrebleu
import os
import time
from transformers import AutoTokenizer

from lmms_eval.api.registry import register_aggregation, register_metric
Expand Down Expand Up @@ -405,7 +405,10 @@ def gpt4judge(references, predictions, query): # This is a passthrough function
eval_logger.error(f"All 5 attempts failed. Last error message: {str(e)}.\nResponse: {str(error_msg)}")
response = ""

score = int(extract_number_from_brackets(response))
if response is None: # Rare case of gpt returning empty response
score = 0
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we do a try+except wrapping extract_number_from_brackets, and also add logging whenever this is caught?

else:
score = int(extract_number_from_brackets(response))
responses.append(response)
values.append(score)

Expand All @@ -417,9 +420,10 @@ def sambajudge(references, predictions, query): # This is a passthrough functio
"""https://github.com/QwenLM/Qwen-VL/blob/master/eval_mm/infographicsvqa_eval.py"""
values = []
responses = []
import requests
import json

import requests

NUM_SECONDS_TO_SLEEP = 30
from openai import OpenAI

Expand Down
Loading