[Performance] 2x Regression in 1st Inference time cost

[A-Satti]

Describe the issue

Comparing 1st inference time costs between ORT 1.16.3 and 1.14, a number of public models show significant regression in "Session Creation" and "Evaluate" time costs. This issue is reproducible on both MLAS and DirectML EPs, with both WinML and ORT APIs.

Average Evaluate times are not affected given a number of iterations.

The regression is observed also with 1.15 binaries revealing that the regression was introduced between 1.14.0 and 1.15.0

To reproduce

Using onnxruntime_perf_test or MicrosoftMLRunner with binaries from 1.16.3 and 1.14.0 note the "Session Creation" and "Evaluate" times (example - deeplabv3 shows a 2x regression).

cmd: MicrosoftMLRunner.exe -CPU -Perf -model deeplabv3_u8s8.onnx

Urgency

This issue affects scenarios where the 1st inference time is important and discourages software vendors from upgrading to ORT 1.16.3

Platform

Windows

OS Version

Windows 11

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.16.3

ONNX Runtime API

WinML

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

Model File

No response

Is this a quantized model?

Yes

[cbourjau]

Could you provide a flame graph or a profile that may shed some light on where the regression is located?

[A-Satti]

ort14_deeplabv3_-u8s8.json.json
ort16_deeplabv3-u8s8.json.json

Was not able to note sizeable regression in the operators.

We did find that disabling the optimization_level recovers Session Creation Time cost performance and with it enabled by default this value regresses for a number of models.