[Feature Request] Support building Python-API for multiple ExecutionProviders #19268
Labels
ep:CUDA
issues related to the CUDA execution provider
ep:OpenVINO
issues related to OpenVINO execution provider
ep:TensorRT
issues related to TensorRT execution provider
feature request
request for unsupported feature or enhancement
Describe the feature request
Currently, apparently there is no "official" support for building onnxruntime with multiple ExecutionProviders, see also #18790.
The issue I linked to suggests building onnxruntime from scratch as a resolution, and I did exactly that.
However, there were several places in the (python) code where it's hard-coded/assumed that there is only one
ExecutionProvider
available. Two most prominent places that come to mind (and should be considered bugs if onnxruntime says it supports using mulitple ExecutionProviders in parallel):is_openvino
are mutually exclusive while they do not have to be, see e.g. https://github.com/microsoft/onnxruntime/blob/main/setup.py#L62-L88InferenceSession
are also assuming mutually exclusiveExecutionProviders
, see for example here https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/onnxruntime_inference_collection.py#L437-L460. This lead to the inability of falling back fromOpenVINOExeuctionProvider
toCPUExecutionProvider
when the available provider list is["TensorrtExecutionProvider", "CUDAExecutionProvider", "OpenVINOExecutionProvider", "CPUExecutionProvider"]
, which is the case when compiling with trt, cuda & openvino.Both of the above should be considered bugs that need to be solved if onnxruntime decides to actually support multiple
ExecutionProviders
officiallyDescribe scenario use case
Deployment inside docker where one wants to leverage multiple EPs/different hardware sets, see also #18790
The text was updated successfully, but these errors were encountered: