Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add example for Llama3 vllm server #404

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

cameron-chen
Copy link

Hello, I am creating this PR to share the example of evaluating by local model using API call (vllm server).

I find this approach can be quite useful when:

  • server the local annotator on the cluster so that other nodes only need to call API for evaluating.
  • the user wants to use "weighted"-style annotator similar with weighted_alpaca_eval_gpt4_turbo.

Please let me know if this looks good. I am happy to add more detailed instructions.


To use the API, add the client config local_configs.yaml and activate it:

default:
    - api_key: "token-abc123"
      base_url: "http://localhost:8000/v1"

@YannDubs
Copy link
Collaborator

LGTM, can you add a README.md in src/alpaca_eval/evaluators_configs/weighted_alpaca_eval_vllm_llama3_70b/ that explains the setup and goal? thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants