ONNXRT default CPU EP vs Openvino EP Performance

[Darshvino]

Hi ONNXRT team,

I was comparing the performance between ONNXRT default CPU EP vs Openvino EP performance; I found that Openvino EP is way faster than default CPU EP on Intel CPU.

Here are the timings with one thread(session_options.SetIntraOpNumThreads(1); ):

Resnet18:

Default CPU = 27 ms

OpenVino = 7 ms

One layer: Input shape: 1x256x56x56, Weight shape: 64x256x3x3

Default CPU = 7 ms

Openvino = 2 ms

May I know if these numbers seem to be right? which EP is faster on x86 for CNNs? And also wanted to get the comments on the above results.

Below is the info about the CPU I am using:

Look forward to your reply.

Thanks

[Darshvino]

Look forward to the reply

Thanks

[yufenglee]

Where the model comes from? And could you please check if openvino EP uses GPU for computing?

[Darshvino]

Hi @yufenglee,

Thank you for your response.

This is the resnet18 model I am using, and it is from onnx model zoo: https://drive.google.com/file/d/1uQt_UYHluOfTq_DzMdlyz4OYe7aihP7X/view?usp=sharing

Yeah, that may be the possible reason for the speedup, I will check if it uses GPU at the backend. Also, will the dedicated GPU(NVIDIA) be used or the integrated GPU?

[Darshvino]

@yufenglee,

Just to intimate you that I have built the ONNXRT for CPU FP32: ./build.sh --config RelWithDebInfo --use_openvino CPU_FP32 --build_shared_lib --parallel

[Darshvino]

Hi @yufenglee @snnn,

Look forward to your reply.

I believe Openvino is using multiple threads to run at the backend even though we set to run with one thread in ONNXRT.

[weimeng23]

openvino use multiple threads ( = physical cores)
I Set IntraOpNumThreads=1 but it didn't work.
You can use htop to watch utilization

[weimeng23]

see The PR: #18596
up to now, the latest release is ONNX Runtime v1.16.3
Looks like this PR version hasn't been released yet.