Nested Api Url

[talhairfanbentley]

@bentoml.artifacts([PytorchModelArtifact('detectron2d1')]) # detectron2 draft 1
class PytorchModelService(bentoml.BentoService):
    @bentoml.api(input=ImageInput(), api_name='v1/model/predict', batch=False)
    def predict(self, imgs):
        # img = imgs[0] # in case of batch=True
        with torch.no_grad():
            # image = image[:, :, ::-1]
            img = imgs[:, :, ::-1]
            inputs = preprocess(img)
            predictions = self.artifacts.detectron2d1([inputs])[0]
            # print(predictions)
            # for k, v in predictions['instances']._fields.items():
            #     print(k)
            reponse = {
                "pred_boxes" : predictions['instances'].pred_boxes.tensor.tolist(),
                "scores" : predictions['instances'].scores.tolist(),
                "pred_classes" : predictions['instances'].pred_classes.tolist(),
                # "pred_masks" : predictions['instances'].pred_masks.tolist(),
            }
            return (reponse)

So in above code, by default the url for my function is http://< host >/predict
is there any way to make it look like http://< host >/v1/model/predict
is this possible in current build?

[yubozhao]

@talhairfanbentley Right now the API name is restricted to only contains alphanumeric and underscores. The API name you mentioned, v1/model/predict, is not possible with the current version of BentoML. It should be pretty straight forward for BentoML to change and work with the more complex path like you mentioned.

Is this a blocker for you? Are you working with any frameworks that have a strict format for the endpoint URL?

[talhairfanbentley]

Yea so , I'm trying to deploy it on kubeflow/kfserving (Knative) and when I make a InferenceService (crd) it creates a path (virtual service) for it by default but I have to edit it and make it /predict (supported by bentoml) rather than /v1/models/$MODEL_NAME:predict (default uri created by kfserving in virtual service).

[yubozhao]

Got it.

For KfServing, we are going to support their prediction V2 API that is on their roadmap for q3: https://github.com/kubeflow/kfserving/blob/master/ROADMAP.md. That way you won't need to change the endpoints and still can take advantage of the KFserving offers.

For the current KFserving API, we recommend to bypass the top level InferenceService provide by KFserving, here is our guide to deploy to Kfserving https://docs.bentoml.org/en/latest/deployment/kfserving.html

If you want to deploy it as a Knative service, you can find related documentation/guide here: https://docs.bentoml.org/en/latest/deployment/knative.html

Hope this helps and happy to answer any questions or issues.

[talhairfanbentley]

Sounds good at the moment, thanks.