Are there problems with the OpenVINO EP?

[AdarshAcharya5]

I built onnxruntime with openvino for C++ following instructions from here as well as python with:

pip install onnxruntime==1.17.1
pip install onnxruntime-openvino==1.17.1
pip install openvino==2023.3

However when I run inference on a model with :

options = ort.SessionOptions()
options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_DISABLE_ALL
session = ort.InferenceSession("model.onnx", 
                                sess_options=options, 
                                providers=['OpenVINOExecutionProvider'], 
                                provider_options=[{'device_type' : "CPU_FP32"}])

The following error is thrown :

*************** EP Error ***************
EP Error C:\Users\Administrator\Desktop\final_Windows_wheels\FINAL_08_March_2024\python311\onnxruntime\onnxruntime\core\session\provider_bridge_ort.cc:1209 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "C:\Users\Adarsh\anaconda3\envs\deeplearning\Lib\site-packages\onnxruntime\capi\onnxruntime_providers_openvino.dll"
 when using ['OpenVINOExecutionProvider']
Falling back to ['CPUExecutionProvider'] and retrying.
****************************************

Similarly when I run inference in C++ with:

mEnv = new Ort::Env(ORT_LOGGING_LEVEL_WARNING, "test");
mSessionOptions = new Ort::SessionOptions;
mSessionOptions->SetGraphOptimizationLevel(ORT_DISABLE_ALL);
std::unordered_map<std::string, std::string> options;
options["device_type"] = "CPU_FP32";
mSessionOptions->AppendExecutionProvider_OpenVINO_V2(options);

The program crashes with exit code 3 and the same error code 126. I debugged it and here's where it fails (File: env.cc , Line: 684):

The library isn't being loaded properly in both cases. I had no problems with my build except for some failed tests. I would like to know if this is a known issue in the latest release or if i've messed up my build somehow. Thanks!

Build script for reference:
.\build.bat --config RelWithDebInfo --use_openvino HETERO:GPU,CPU --build_shared_lib --build_wheel

To reproduce

Null

Urgency

Would be great if solved soon, but no strict deadline.

Platform

Windows

OS Version

11

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.17.1

ONNX Runtime API

C++

Architecture

X64

Execution Provider

OpenVINO

Execution Provider Library Version

OpenVINO 2023.3

[jywu-msft]

the error suggests it's not able to find the OpenVINO library dependencies.
can you ensure the path to the OpenVINO dll's is in your system PATH env variable?

[jywu-msft]

i think you can also follow this example https://github.com/microsoft/onnxruntime-inference-examples/blob/e8e455aae12ef2a4eaca2655120a95c505c66549/python/OpenVINO_EP/yolov8_object_detection/yolov8.py#L30
to add to PATH

[AdarshAcharya5]

@jywu-msft Thanks!. I did add the dll directories to Path and inference runs on Python now. However in C++, it still aborts with exit code 3. Is there something else i need to do?.

[AdarshAcharya5]

Nvm, i fixed it by giving the path to both Release and Debug folders. I had only given the path to the Release folder earlier. I will be closing this issue now. Thanks!