Skip to content

Commit

Permalink
Add sample codeblocks for request cancellation (#415)
Browse files Browse the repository at this point in the history
* Add sample codeblocks for request cancellation

* Update README.md

Co-authored-by: Ryan McCormick <[email protected]>

---------

Co-authored-by: Ryan McCormick <[email protected]>
  • Loading branch information
tanmayv25 and rmccorm4 committed Oct 11, 2023
1 parent 46f5fd4 commit cc8ff52
Showing 1 changed file with 18 additions and 1 deletion.
19 changes: 18 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -528,9 +528,26 @@ examples demonstrate how to infer with AsyncIO.
Starting from r23.10, triton python gRPC client can issue cancellation
to inflight requests. This can be done by calling `cancel()` on the
CallContext object returned by `async_infer()` API.

```python
ctx = client.async_infer(...)
ctx.cancel()
```

For streaming requests, `cancel_requests=True` can be sent to
`stop_stream()` API to terminate all the inflight requests
sent via this stream. See more details about these APIs in
sent via this stream.

```python
client.start_stream()
for _ in range(10):
client.async_stream_infer(...)

# Cancels all pending requests on stream closure rather than blocking until requests complete
client.stop_stream(cancel_requests=True)
```

See more details about these APIs in
[grpc/\_client.py](src/python/library/tritonclient/grpc/_client.py).

See [request_cancellation](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/request_cancellation.md)
Expand Down

0 comments on commit cc8ff52

Please sign in to comment.