Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request] Support building Python-API for multiple ExecutionProviders #19268

Open
ORippler opened this issue Jan 25, 2024 · 0 comments
Open
Labels
ep:CUDA issues related to the CUDA execution provider ep:OpenVINO issues related to OpenVINO execution provider ep:TensorRT issues related to TensorRT execution provider feature request request for unsupported feature or enhancement

Comments

@ORippler
Copy link

Describe the feature request

Currently, apparently there is no "official" support for building onnxruntime with multiple ExecutionProviders, see also #18790.

The issue I linked to suggests building onnxruntime from scratch as a resolution, and I did exactly that.
However, there were several places in the (python) code where it's hard-coded/assumed that there is only one ExecutionProvider available. Two most prominent places that come to mind (and should be considered bugs if onnxruntime says it supports using mulitple ExecutionProviders in parallel):

  1. During building for example, package name and EP-flags such as is_openvino are mutually exclusive while they do not have to be, see e.g. https://github.com/microsoft/onnxruntime/blob/main/setup.py#L62-L88
  2. The error detection and fallback logic of InferenceSession are also assuming mutually exclusive ExecutionProviders, see for example here https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/onnxruntime_inference_collection.py#L437-L460. This lead to the inability of falling back from OpenVINOExeuctionProvider to CPUExecutionProvider when the available provider list is ["TensorrtExecutionProvider", "CUDAExecutionProvider", "OpenVINOExecutionProvider", "CPUExecutionProvider"], which is the case when compiling with trt, cuda & openvino.

Both of the above should be considered bugs that need to be solved if onnxruntime decides to actually support multiple ExecutionProviders officially

Describe scenario use case

Deployment inside docker where one wants to leverage multiple EPs/different hardware sets, see also #18790

@ORippler ORippler added the feature request request for unsupported feature or enhancement label Jan 25, 2024
@github-actions github-actions bot added ep:CUDA issues related to the CUDA execution provider ep:OpenVINO issues related to OpenVINO execution provider ep:TensorRT issues related to TensorRT execution provider labels Jan 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ep:CUDA issues related to the CUDA execution provider ep:OpenVINO issues related to OpenVINO execution provider ep:TensorRT issues related to TensorRT execution provider feature request request for unsupported feature or enhancement
Projects
None yet
Development

No branches or pull requests

1 participant