onnxruntime_perf_test.exe failing starting v1.18.0 if the model path doesn't contain any path separator

[saurabhtangri]

Describe the issue

C:\dev\ort-src\onnxruntime-v1.18.0\build\Windows\RelWithDebInfo\RelWithDebInfo>onnxruntime_perf_test.exe -e cpu -I -m times -r 1000 simplified_ssd.onnx
C:\dev\ort-src\onnxruntime-v1.18.0\onnxruntime\test\onnx\TestCase.cc:705 OnnxTestCase::OnnxTestCase test case dir doesn't exist

To reproduce

Build ONNXRT for windows and run onnxruntime_perf_test

Urgency

Showstopper

Platform

Windows

OS Version

Win11

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

v1.18.0

ONNX Runtime API

Python

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

[snnn]

The latest code should work fine.

[saurabhtangri]

I had tried with 1.18.1 tag which behaved the same as 1.18.0 as below.

C:\dev\ort-src\onnxruntime-v1.18.0\build\Windows\RelWithDebInfo\RelWithDebInfo>onnxruntime_perf_test.exe -e cpu -I -m times -r 1000 simplified_ssd.onnx
C:\dev\ort-src\onnxruntime-v1.18.0\onnxruntime\test\onnx\TestCase.cc:705 OnnxTestCase::OnnxTestCase test case dir doesn't exist

C:\dev\ort-src\onnxruntime-v1.18.0\build\Windows\RelWithDebInfo\RelWithDebInfo>onnxruntime_perf_test.exe -t 10 simplified_ssd.onnx
C:\dev\ort-src\onnxruntime-v1.18.0\onnxruntime\test\onnx\TestCase.cc:705 OnnxTestCase::OnnxTestCase test case dir doesn't exist

C:\dev\ort-src\onnxruntime-v1.18.0\build\Windows\RelWithDebInfo\RelWithDebInfo>git status
HEAD detached at v1.18.0
Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git restore ..." to discard changes in working directory)
modified: ../../../../cmake/external/emsdk (new commits)
modified: ../../../../cmake/external/onnx (new commits)

no changes added to commit (use "git add" and/or "git commit -a")

==============================================================

C:\dev\ort-src\onnxruntime\build\Windows\RelWithDebInfo\RelWithDebInfo>onnxruntime_perf_test.exe -e cpu -I -m times -r 10 simplified_ssd.onnx
Session creation time cost: 0.0377832 s
First inference time cost: 63 ms
Total inference time cost: 0.146239 s
Total inference requests: 10
Average inference time cost: 14.6239 ms
Total inference run time: 0.147193 s
Number of inferences per second: 67.9381
Avg CPU usage: 99 %
Peak working set size: 245743616 bytes
Avg CPU usage:99
Peak working set size:245743616
Runs:10
Min Latency: 0.0129886 s
Max Latency: 0.0256459 s
P50 Latency: 0.013477 s
P90 Latency: 0.0256459 s
P95 Latency: 0.0256459 s
P99 Latency: 0.0256459 s
P999 Latency: 0.0256459 s

C:\dev\ort-src\onnxruntime\build\Windows\RelWithDebInfo\RelWithDebInfo>git status
HEAD detached at v1.17.1
Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git restore ..." to discard changes in working directory)
modified: ../../../../cmake/external/emsdk (new commits)
modified: ../../../../cmake/external/onnx (new commits)

no changes added to commit (use "git add" and/or "git commit -a")

[snnn]

Did you try the main branch? Does it work?

[mailkv23]

I have been seeing this issue for few weeks now. Its seen in main branch

[mailkv23]

I had tried with 1.18.1 tag which behaved the same as 1.18.0 as below.

C:\dev\ort-src\onnxruntime-v1.18.0\build\Windows\RelWithDebInfo\RelWithDebInfo>onnxruntime_perf_test.exe -e cpu -I -m times -r 1000 simplified_ssd.onnx C:\dev\ort-src\onnxruntime-v1.18.0\onnxruntime\test\onnx\TestCase.cc:705 OnnxTestCase::OnnxTestCase test case dir doesn't exist

C:\dev\ort-src\onnxruntime-v1.18.0\build\Windows\RelWithDebInfo\RelWithDebInfo>onnxruntime_perf_test.exe -t 10 simplified_ssd.onnx C:\dev\ort-src\onnxruntime-v1.18.0\onnxruntime\test\onnx\TestCase.cc:705 OnnxTestCase::OnnxTestCase test case dir doesn't exist

C:\dev\ort-src\onnxruntime-v1.18.0\build\Windows\RelWithDebInfo\RelWithDebInfo>git status HEAD detached at v1.18.0 Changes not staged for commit: (use "git add ..." to update what will be committed) (use "git restore ..." to discard changes in working directory) modified: ../../../../cmake/external/emsdk (new commits) modified: ../../../../cmake/external/onnx (new commits)

no changes added to commit (use "git add" and/or "git commit -a")

==============================================================

C:\dev\ort-src\onnxruntime\build\Windows\RelWithDebInfo\RelWithDebInfo>onnxruntime_perf_test.exe -e cpu -I -m times -r 10 simplified_ssd.onnx Session creation time cost: 0.0377832 s First inference time cost: 63 ms Total inference time cost: 0.146239 s Total inference requests: 10 Average inference time cost: 14.6239 ms Total inference run time: 0.147193 s Number of inferences per second: 67.9381 Avg CPU usage: 99 % Peak working set size: 245743616 bytes Avg CPU usage:99 Peak working set size:245743616 Runs:10 Min Latency: 0.0129886 s Max Latency: 0.0256459 s P50 Latency: 0.013477 s P90 Latency: 0.0256459 s P95 Latency: 0.0256459 s P99 Latency: 0.0256459 s P999 Latency: 0.0256459 s

C:\dev\ort-src\onnxruntime\build\Windows\RelWithDebInfo\RelWithDebInfo>git status HEAD detached at v1.17.1 Changes not staged for commit: (use "git add ..." to update what will be committed) (use "git restore ..." to discard changes in working directory) modified: ../../../../cmake/external/emsdk (new commits) modified: ../../../../cmake/external/onnx (new commits)

no changes added to commit (use "git add" and/or "git commit -a")

Try providing full path to the onnx model

[snnn]

I see. Now I can reproduce the issue.

[virajwad]

Seeing the same issue!

[snnn]

Thanks for reporting the issue. The fix will be included in the next release.

[snnn]

While waiting the release, you may use an absolute path instead.