[Performance] slower inference after upgrade from 1.19.2 to 1.20.1

[iperov]

Describe the issue

x15 slower inference after upgrade from 1.19.2 to 1.20.1

The problem arises in the interleaving of input tensor resolutions, for example in pyramid images for object detection.

CUDA : x15 slower
CPU : x2-3 slower

To reproduce

import numpy as np
import onnxruntime as rt
import time

class timeit:
    def __init__(self, msg : str = None):
        self._msg = msg if msg is not None else ''
    def __enter__(self):
        self.t = time.perf_counter()
    def __exit__(self, a,b,c):
        print(f'Time of {self._msg}: {time.perf_counter()-self.t}')

sess = rt.InferenceSession('YoloV7Face.onnx',
                           providers=[ ('CUDAExecutionProvider', {'device_id':0}) ])

input_name = sess.get_inputs()[0].name

# Various resolution image (for example pyramid images)
imgs = [np.zeros((1,3,256+i*32,256+i*32), np.uint8) for i in range(5)]

while True:
    with timeit(): 
        for img in imgs:
            sess.run(None, {input_name: img})

# 1.19.2 ~30ms 
# 1.20.1 ~500ms

Urgency

Not urgent. Will stay on 1.19.2

Platform

Windows

OS Version

10

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.20.1

ONNX Runtime API

Python

Architecture

X64

Execution Provider

CUDA, CPU

Execution Provider Library Version

CUDA 12.4

Model File

YoloV7Face.zip

Is this a quantized model?

No

[tianleiwu]

Initial finding is that there is long CPU time before launching Conv kernel (the kernel time is very close) for 1.20.1. It is likely caused by cudnn frontend introduced in #19470.
Another issue is cudnn algo search seems only run once for different input shape for 1.20.1. Need further investigations.

[iperov]

@tianleiwu but CPUep also slower. Perhaps problem not in cuda?

[henryruhs]

@iperov #22705 (comment)

[iperov]

@henryruhs what is this? As I said CPU ep is also slower. So the problem not in cuda.

[henryruhs]

then change the overall report, most of it is CUDA based. also kinda rude this reaction.

[iperov]

@henryruhs
"CPU ep : also x2-3 slower" are in two places: topic and in the comment above.

Flooding useless messages is rude.

[henryruhs]

Probably useless for those who are not smart enough, update the optset version of the models.

[iperov]

@henryruhs Updating the opset version is not required by the specification. Otherwise, for example onnxruntime would require it explicitly and would not run the model. This is a silly thought that might come to mind. By the way you keep clogging up the thread, and insulting.