Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Include garbage collect for non streaming and improve triggering logic #52

Merged
merged 3 commits into from
Jul 26, 2024

Conversation

kthui
Copy link
Contributor

@kthui kthui commented Jul 25, 2024

What does the PR do?

Fix:

  • Garbage collect when the request count drops to zero.
  • Include garbage collect for non-streaming case.
  • Fix improper placement of decrement_ongoing_request_count = True.
  • Fix missing lora_request argument into llm_engine.add_request().

for #46 PR.

Checklist

  • PR title reflects the change and is of format <commit_type>: <Title>
  • Changes are described in the pull request.
  • Related issues are referenced.
  • Populated github labels field
  • Added test plan and verified test passes.
  • Verified that the PR passes existing CI.
  • Verified copyright is correct on all changed files.
  • Added succinct git squash message before merging ref.
  • All template sections are filled out.
  • Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

  • build
  • ci
  • docs
  • feat
  • fix
  • perf
  • refactor
  • revert
  • style
  • test

Related PRs:

#46

Where should the reviewer start?

N/A

Test plan:

This PR is to fix some issues on the performance improvement #46 PR, so any issues should be covered by existing tests.

  • CI Pipeline ID: 16926866

N/A

Caveats:

N/A

Background

N/A

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

N/A

@kthui kthui marked this pull request as ready for review July 25, 2024 21:24
@kthui kthui requested a review from Tabrizian July 25, 2024 21:24
@kthui kthui merged commit c54dfef into imant-response-send Jul 26, 2024
3 checks passed
@kthui kthui deleted the jacky-res-sender-del-non-stream branch July 26, 2024 18:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants