[BUG] [OpenVino EP] Only first result in session is correct. #19975
Comments
github-actions
bot
added
ep:CUDA
|
Just to note: the issue looks independent of OpenVino version - got experiments with different. Also built all from scratch many times on different systems - same results. |
|
Update: 1.14.1 also works, but the performance is about 10-15% lower. 1.15 and higher affected by issue. |
|
any update? |
|
Can we have access to the model. It seems there are 11 subgraphs being formed and 167 nodes are being placed on CPUEP. But it is hard to debug without the model. |
|
dumbmodel.onnx.zip Here i'm iterating over the same image. All result except first are broken. `f1race@build_server_nvidia:/opt/ort_dev$ ./test --image images/test/dumb100x100text.jpg [-] Output tensor element count: 390 [-] Output tensor element count: 390 |
|
Once again, this happened ONLY with OpenVINO EP with Onnxruntime >= 1.15 and any version of OpenVino. No issues with Onnxruntime 1.13.1 and 1.14.1 (lower not tested). CPUEP, XnnpackEP, CudaEP works well with this model and same inference code in any version of ORT including the latest one. |
|
I'm seeing a similar issue that occurs in Python with You'll need to install |
|
@sfatimar, Hi! Any updates? I've uploaded the model for bug investigation. |
|
Hi @debugmenot , I have tested the script suggested by @henxing using OpenVINO Toolkit v2024.1 (w_openvino_toolkit_windows_2024.1.0.dev20240405_x86_64) and OVEP v1.18.0 (this version update is now merged and available on the latest main of microsoft/onnruntime repo) on a Windows machine. I ran inference for 5 iterations and the PyTorch vs ORT OpenVINO EP results for every inference iterations were same and OVEP results were quite accurate upto 3 decimal precision against torch results. Please find the below run log for the same - ================================================================================ Onnxruntime: ================================================================================ Onnxruntime: ================================================================================ Onnxruntime: |
We are investigating the issues faced while running your model using OpenVINO EP execution provider. |
|
@ankitm3k hi! Did you confirm the bug? If so, any ETA for patch? |
|
Hi @debugmenot, I recommend you to use latest OpenVINO Toolkit v2024.1 along with the above patch to fix the same. I also have investigated the tensor outputs as a result of multiple inference iterations over the same input data and they were found to be consistent / accurate with the first inference results for my build. |
fix: updated data ops to support the complete graph on OVEP (microsoft#19975)
|
@ankitm3k Onnxruntime 1.18.1 + OpenVinoEP 2024.3 + your GRU OP Patch: i can prepare test project (source+model+image) for you. can you share your email please? |
|
But with patch behaviour is slightly different - results after first is a little bit differs with results without patch but looks approx the same (incorrect)... |
|
@ankitm3k I've finally found an issue, at least WHERE it is EXACTLY. If all works as expected :) Issue needs an investigation. its strange because unsqueeze is defined exactly same way as in 1.14.1 and 1.13.1 versions... |
Describe the issue
When running inference session ONLY with OpenVino EP and ORT > 1.13.1 any results except first are incorrect. There are no issues with ORT == 1.13.1 or CPU/CUDA/XNNPACK on any ORT version.
Getting this issue only on one model (Attention OCR) - model structure you can find at the bottom, other models works fine. seems there are some layers/functions in it that was broken after 1.13.1 build...
Description:
Ubuntu 22.04, Onnxruntime 1.17.1, OpenVino 2023.3, C++
Model: sort of Attention Decoder OCR, converted to onnx from pytorch.
Issue:
im inferencing the same image (also tried on sequence of different images durning the inference session). Only the FIRST result is correct. Second result and so on looks like partially "cropped" first result doesnt matter if next input data is new...
For example inferencing sequence of images with text "1234567890", "ABCDEFGHJK", "7777777777". Getting: "1234567890", "1200120012", "1200120012"...
Downgrade to ORT 1.13.1 solved the issue, but seems that something is broken after 1.13.1 build.
All other EP (CPU, CUDA, XNNPACK) works well with the same code.
Found one reference to similar issue in OpenVino github: openvinotoolkit/openvino#12966
Enabled verbose mode and found that node placements are differ between 1.17.1 (incorrect) and 1.13.1(correct) inference sessions, maybe it's matters, but doesn't explain why first result is always correct...:
correct inference session with node placements(1.13.1):
Incorrect inference result node placement (1.17.1)
as you can see the difference is only on last 8 lines (matmuls token ids differs). Hope it'll help...
F
To reproduce
Look description.
Urgency
Urgent
Platform
Linux
OS Version
Ubuntu 22.04
ONNX Runtime Installation
Built from Source
ONNX Runtime Version or Commit ID
1.17.1 release
ONNX Runtime API
C++
Architecture
X64
Execution Provider
OpenVINO
Execution Provider Library Version
2023.3
The text was updated successfully, but these errors were encountered: