Why I still perform inference while triton server is not ready?

[ptran1203]

Description
According to this link . If triton model is not ready, I shouldn't able to send inference request to server, But when I tried to send Inference request despite of readiness is False (grpcclient.InferenceServerClient.is_server_ready() return False), the server still response me the prediction.

Triton Information
What version of Triton are you using?

22.08

Are you using the Triton container or did you build it yourself?
I use Triton container: nvcr.io/nvidia/tritonserver:22.08-py3

To Reproduce

• I use model-control-mode=poll
• Assume my model is loaded and Server is performing inference
• I make a change on my model repository config (i.e: version 1 to version 2)
• When I call grpcclient.InferenceServerClient.is_server_ready(), it returns False

Expected behavior
When server readiness is False, I shouldn't be able to send inference requests

Actual behavior
The server still response me the prediction

[dyastremsky]

In polling mode, model reloads should not result in loss of availability. You can read more here. The documentation also discusses how all new requests will be routed to the new model, assuming loading succeeds.

As far as the is_server_ready() flags, that is unexpected. The model shouldn't be become "not ready" during reloading. However, polling can be non-atomic in loading/unloading, which is why we recommend EXPLICIT mode for running in production. That gives you greater control over model behavior.

Can you try running the index API to see what models are listed as ready or not ready? You may also be able to look at the verbose logs for context. This will help make sure that what's being marked as not ready is definitely the model being reloaded.

I'd recommend these steps:

1. Start Triton
2. Check that server is ready
3. Change model version
4. Check index to see what models ready/not ready
5. Check whether server is ready

CC: @GuanLuo

[ptran1203]

Thank you for your response, I will try your suggestion!