Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Errors in grpc_server.cc while installing onnxruntime backend #6989

Closed
Aniket-20 opened this issue Mar 15, 2024 · 11 comments
Closed

Errors in grpc_server.cc while installing onnxruntime backend #6989

Aniket-20 opened this issue Mar 15, 2024 · 11 comments
Labels
question Further information is requested

Comments

@Aniket-20
Copy link

/home/aniket/server/src/grpc_server.cc: In lambda function:
/home/aniket/server/src/grpc_server.cc:826:24: error: narrowing conversion of ‘(int)byte_size’ from ‘int’ to ‘google::protobuf::stringpiece_internal::StringPiece::size_type’ {aka ‘long unsigned int’} [-Werror=narrowing]
826 | {buffer, (int)byte_size}, response->mutable_config());
| ^~~~~~~~~~~~~~
/home/aniket/server/src/grpc_server.cc: In instantiation of ‘TRITONSERVER_Error* triton::server::{anonymous}::InferResponseCompleteCommon(TRITONSERVER_Server*, TRITONSERVER_InferenceResponse*, inference::ModelInferResponse&, const triton::server::{anonymous}::AllocPayload&) [with ResponseType = inference::ModelInferResponse]’:
/home/aniket/server/src/grpc_server.cc:3800:69: required from here
/home/aniket/server/src/grpc_server.cc:3353:5: error: enumeration value ‘TRITONSERVER_PARAMETER_DOUBLE’ not handled in switch [-Werror=switch]
3353 | switch (type) {
| ^~~~~~
/home/aniket/server/src/grpc_server.cc: In instantiation of ‘TRITONSERVER_Error* triton::server::{anonymous}::InferResponseCompleteCommon(TRITONSERVER_Server*, TRITONSERVER_InferenceResponse*, inference::ModelInferResponse&, const triton::server::{anonymous}::AllocPayload&) [with ResponseType = inference::ModelStreamInferResponse]’:
/home/aniket/server/src/grpc_server.cc:4400:77: required from here
/home/aniket/server/src/grpc_server.cc:3353:5: error: enumeration value ‘TRITONSERVER_PARAMETER_DOUBLE’ not handled in switch [-Werror=switch]
cc1plus: all warnings being treated as errors
make[5]: *** [CMakeFiles/grpc-endpoint-library.dir/build.make:76: CMakeFiles/grpc-endpoint-library.dir/grpc_server.cc.o] Error 1
make[4]: *** [CMakeFiles/Makefile2:440: CMakeFiles/grpc-endpoint-library.dir/all] Error 2
make[3]: *** [Makefile:136: all] Error 2
make[2]: *** [CMakeFiles/triton-server.dir/build.make:86: triton-server/src/triton-server-stamp/triton-server-build] Error 2
make[1]: *** [CMakeFiles/Makefile2:193: CMakeFiles/triton-server.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

Triton Information
nvcr.io/nvidia/tritonserver:24.02-py3
Version:24.02

Are you using the Triton container or did you build it yourself?
Using docker container

Using cmake to install onnxruntime backend:-
$ mkdir build
$ cd build
$ cmake -DCMAKE_INSTALL_PREFIX:PATH=pwd/install -DTRITON_BUILD_ONNXRUNTIME_VERSION=1.17.1 -DTRITON_BUILD_CONTAINER_VERSION=24.02..
$ make install

Kindly assist in identifying the root causes of these compilation errors and suggesting appropriate solutions for successful server build and execution.

Thank you for your attention to this matter.

I'm ready to provide further guidance as needed. Please share any additional details or questions you may have.en.
@Aniket-20 Aniket-20 changed the title Errors in grpc_server.cc wheile building onnxruntime backend for nvidia triton server. Errors in grpc_server.cc while installing onnxruntime backend Mar 15, 2024
@lkomali
Copy link
Contributor

lkomali commented Mar 15, 2024

ONNX Runtime backend is not included with 24.02 release due to incompatibility issues. However iGPU and Windows build assets shipped with ONNX Runtime backend.

@Aniket-20
Copy link
Author

Aniket-20 commented Mar 15, 2024 via email

@lkomali
Copy link
Contributor

lkomali commented Mar 15, 2024 •
edited
Loading

@Aniket-20 Triton 24.01 comes with onnx runtime backend. You can pull the 24.01 image and run a docker container and you should be able to deploy onnx models without building it. Here's a tutorial explaining how to deploy an onnx model - https://github.com/triton-inference-server/tutorials/blob/main/Quick_Deploy/ONNX/README.md
If you are modifying the onnx backend and want to build it with your changes, please provide more details on what changes you are trying to make and if possible the code changes you made.

@Aniket-20
Copy link
Author

Aniket-20 commented Mar 15, 2024 via email

@lkomali lkomali added the question Further information is requested label Mar 15, 2024
@Aniket-20
Copy link
Author

Aniket-20 commented Mar 18, 2024 via email

@lkomali
Copy link
Contributor

lkomali commented Mar 18, 2024

Please look at this Support Matrix guide - https://docs.nvidia.com/deeplearning/frameworks/support-matrix/index.html

@Aniket-20
Copy link
Author

Aniket-20 commented Mar 19, 2024 via email

@lkomali
Copy link
Contributor

lkomali commented Mar 19, 2024 

E0319 17:48:18.714876 193 model_repository_manager.cc:1325] Poll failed for
model directory 'reports': Invalid model name: Could not determine backend
for model 'reports' with no backend in model configuration. Expected model
name of the form 'model.<backend_name>'.

It looks like an error with the model repository you are providing. Can you share the structure of your model repository as well as the config.pbtxt files?
You can follow this guide on how to structure your model_repository and creating config files - https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/model_configuration.html
You can have a look at the backends section in this guide to modify config.pbtxt according to the backend you are using - https://github.com/triton-inference-server/backend/blob/main/README.md#backends

@Aniket-20
Copy link
Author

Aniket-20 commented Mar 19, 2024 via email

@Aniket-20
Copy link
Author

Aniket-20 commented Mar 20, 2024 via email

@oandreeva-nv
Copy link
Contributor

@Aniket-20 , here's the guide: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/examples/jetson/concurrency_and_dynamic_batching/README.html

I'll close this issue since it seems like it was resolved. Feel free to reach out with any other questions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Further information is requested
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Aniket-20 @oandreeva-nv @lkomali