[Feature Request] Detect TensorRT cache creation

[henryruhs]

Describe the feature request

We have the problem that creating the .engine and .profile files for TensorRT take too much time and there is no feedback for the user what is actual happening.

Therefore I suggest adding a general "get_state" to onnxruntime.InferenceSession in order to figure out why it is currently blocked. Like is_processing, is_caching etc.

I would be happy with any other solution to detect the "cache creation" for now

Describe scenario use case

Give the user better feedback while creating caches for TensorRT.

[henryruhs]

There seems to be no logic to invalidate the existing cache.

I experienced exceptions after updating the version of TensorRT.

[chilo-ms]

Thanks for the feedback.

TRT does require amount of time when building the engine if the model is complex.
You can enable TRT EP's provider option trt_detailed_build_log for now which gives you more verbose logging and it keeps printing out information when TRT is doing kernels profiling/selection for building the engine.

Caching/serialize the engine is quick and same as deserializing the engine cache. So, as long as there is a corresponding engine file on the disk, TRT EP will deserialize the engine without rebuilding it, only if the input shape changes it will rebuild the engine.

There seems to be no logic to invalidate the existing cache.

yea it seems no tool to validate engine, using engine cache built with different TRT throws exception directly.
TRT has the feature of building version compatible engine, but TRT EP hasn't integrated it.

[henryruhs]

Okay, it sounds like there is not too much you can do about both topics. I give trt_detailed_build_log a shot and close this issue in meanwhile. Thanks