How to improve GPU utilization[Performance]

[hy846130226]

Describe the issue

Hey everyone,

I find that when i run onnx runtime, the GPU utilization is not high, I just use the 15%, is there any ways to use the left 75%?

Is creating more sessions a good plan?

To reproduce

Do not have.

Urgency

No response

Platform

Windows

OS Version

WIN10

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

Microsoft.ML.OnnxRuntime.DrectML

ONNX Runtime API

C++

Architecture

X64

Execution Provider

DirectML

Execution Provider Library Version

1.13.1

Model File

Do not have

Is this a quantized model?

Yes

[xadupre]

It is difficult to answer in any case. It also depends on your CPU and GPU. Is your GPU running at 100% and then does nothing before starting again or it is constantly running at 15% while the CPU does nothing? You could run a profiling to understand why the model is slow and which operator is the bottleneck: https://onnxruntime.ai/docs/performance/tune-performance/profiling-tools.html.

[pranavsharma]

cc @fdwr

[fdwr]

Is your model by chance heavily sequential with lots of small operators? Running it from Windows PIX can help spot calling patterns (GPU capture).

cc @smk2007

[hy846130226]

Thanks for your reply. @xadupre @pranavsharma @fdwr

We could discuss in topic #17686

They are the same questions.