Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: vLLM metrics optimization #379

Merged
merged 1 commit into from
Sep 21, 2024

Conversation

yinggeh
Copy link
Contributor

@yinggeh yinggeh commented Sep 18, 2024

What does the PR do?

Optimize vLLM by reporting metrics in a separate thread.

Checklist

  • PR title reflects the change and is of format <commit_type>: <Title>
  • Changes are described in the pull request.
  • Related issues are referenced.
  • Populated github labels field
  • Added test plan and verified test passes.
  • Verified that the PR passes existing CI.
  • Verified copyright is correct on all changed files.
  • Added succinct git squash message before merging ref.
  • All template sections are filled out.
  • Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

  • perf

Related PRs:

triton-inference-server/vllm_backend#66

Where should the reviewer start?

n/a

Test plan:

L0_backend_vllm/metrics_test

  • CI Pipeline ID:
    18506139

Caveats:

n/a

Background

n/a

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

n/a

@yinggeh yinggeh self-assigned this Sep 18, 2024
@yinggeh yinggeh added the enhancement New feature or request label Sep 18, 2024
@yinggeh yinggeh requested review from kthui and indrajit96 September 18, 2024 17:48
@yinggeh yinggeh merged commit 35a1c1f into main Sep 21, 2024
3 checks passed
@yinggeh yinggeh deleted the yinggeh-DLIS-7271-vllm-metrics-optimization branch December 30, 2024 21:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Development

Successfully merging this pull request may close these issues.

2 participants