perf: Check for cancellation on response thread

[kthui]

What does the PR do?

Cancellation check is to be performed during the response loop on the response thread. The status of the cancellation will be written back a object shared between the response loop and generate loop on a particular request. After sending a response, the cancellation status will be checked and the generate loop will cancel the generation if the request is cancelled. Combined with the release of GIL while checking for cancellation, this maximizes the vLLM generation time on the GIL, which has seen noticeable performance improvement.

Checklist

• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging ref.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

• build
• ci
• docs
• feat
• fix
• perf
• refactor
• revert
• style
• test

Related PRs:

triton-inference-server/python_backend#372

Where should the reviewer start?

Start at once a cancellation is issued while responses are being actively generated, does the generate loop capture the cancellation signal? Is the final flag sent (both streaming/non-streaming) after cancelling? Is the response sender deleted and garbage collected after cancelling?

Test plan:

This is a performance improvement, so any issue should be covered by existing test cases.

• CI Pipeline ID: 17061062

Caveats:

N/A

Background

Need to improve the performance on vLLM backend.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

N/A

[oandreeva-nv]

LGTM! Please, let @Tabrizian to look at it as well

[Tabrizian]

One minor comment, otherwise looks good.