Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] GroundTruthAgreement is not working with Litellm #1655

Open
Arslan-Soomro opened this issue Nov 18, 2024 · 0 comments
Open

[BUG] GroundTruthAgreement is not working with Litellm #1655

Arslan-Soomro opened this issue Nov 18, 2024 · 0 comments
Assignees
Labels
bug Something isn't working

Comments

@Arslan-Soomro
Copy link

Bug Description
I am using ground truth agreement feedback function, and the when I print the result for that feedback, it is none. all other feedbacks work okay though.
image

To Reproduce

    from trulens.providers.litellm import LiteLLM
    from trulens.core import TruSession, Feedback
    from trulens.feedback import GroundTruthAgreement
    from trulens.apps.llamaindex import TruLlama

    trulens_provider = Litellm(model)
    qa_dataset = [{"query": "Who are you?", "response": "I am an ai Chatbot"}]
    f_groundtruth = Feedback(
        GroundTruthAgreement(
            ground_truth=qa_dataset, 
            provider=trulens_provider
        ).agreement_measure, name="Answer Correctness"
    ).on_input_output()
   
    tru_recorder = TruLlama(
        query_engine,
        app_name="Naive RAG Eval Llama",
        app_version="1.0",
        feedbacks=[f_groundtruth],
    )

    with tru_recorder as recording:
            query_engine.query(qa_dataset[0]["query"]);
    
    rec = recording.get();
    for feedback, feedback_result in rec.wait_for_feedback_results().items():
            print(f"{feedback.name}: {feedback_result.result}")

Expected behavior
groundtruth feedback should provide the score for answer correctness in the result.

Environment:

  • OS: MacOS
  • Python Version: 3.12
  • TruLens version: 1.2.7
@Arslan-Soomro Arslan-Soomro added the bug Something isn't working label Nov 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants