Engine successfully converts to ONNX, but onnxruntime returns an error when attempting to run inference

[ninono12345]

Describe the issue

I can convert my ONNX model to pytorch successfully, but when attempting to run inference I get this error:
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Reshape node. Name:'/Reshape_7' Status Message: /onnxruntime_src/onnxruntime/core/providers/cpu/tensor/reshape_helper.h:28 onnxruntime::ReshapeHelper::ReshapeHelper(const onnxruntime::TensorShape&, onnxruntime::TensorShapeVector&, bool) i < input_shape.NumDimensions() was false. The dimension with value zero exceeds the dimension size of the input tensor

code used for conversion:

`input_names = ['sample_x', 'train_samples', 'target_labels', 'train_ltrb']`
`output_names = ['target_scores', 'bbreg_test_feat_enc', 'bbreg_weights']`

`torch.onnx.export(model,

                    model_inputs,
                    "tomp101_head_latest3.onnx",
                    verbose=False,
                    export_params=True,
                    do_constant_folding=True,
                    opset_version=16,
                    input_names=input_names,
                    output_names=output_names,
                    dynamic_axes={'sample_x':{0:'batch_size'},
                                'train_samples':{0:'batch_size'},
                                'target_labels':{0:'batch_size'},
                                'train_ltrb':{0:'batch_size'},
                                'target_scores':{0:'batch_size'},
                                'bbreg_test_feat_enc':{0:'batch_size'},
                                'bbreg_weights':{0:'batch_size'}})`

ALSO

If I convert the model without specifying the dynamic axes like so:

`torch.onnx.export(model,

                    model_inputs,
                    "tomp101_head_latest3.onnx",
                    verbose=False,
                    export_params=True,
                    do_constant_folding=True,
                    opset_version=16,
                    input_names=input_names,
                    output_names=output_names)`

I get this type of error:

params_dict = _C._jit_pass_onnx_constant_fold(
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

even though I made sure that the model and input tensors are only on CUDA or on 2nd attempt only on CPU!!!!

THANK YOU FOR YOUR HELP, if any additional info is needed, please let me know!!!!!

To reproduce

KEEP IN MIND THAT THE CODE WAS MODIFIED FROM THE ORIGINAL, SO SEPARATE FUNCTIONS USED FOR TRAINING AND INFERENCE.

now for INITIALIZING we use tompnet forward and for CONSTANT TRACKING we use the head of tompnet model, which is called as a separate independent from tompnet model with it's own forward function.

inside pytracking-master/pytracking/tracker/tomp/tomp101.py we initialize the model with model(value) and within the track function on each frame we call model.head(values)

download the tracking code from my google drive, and the pth model here

pytracking master must be placed inside directory: D:\pyth\pytracking-master
OR
you need to go into pytracking-master/pytracking/evaluation/local.py and change the directory paths accordingly (you will see it's simple)

to run the pytracking algorithm as is write python run_webcam.py tomp tomp101

tompnet model code: pytracking-master/pytracking/tracker/tomp/tomp101.py
the head of tompnet (THE ONE BEING CONVERTED TO ONNX IN THIS ISSUE!!!!!!): pytracking-master/ltr/models/transformer/heads.py

to convert the head (constant tracking) model to ONNX, inside the tomp101.py line 99 change to True do_train = True and onnx.export code at line 613 will be executed.

OR

to load the model in a separate file, enter this:

from pytracking.utils.loading import load_network
model = load_network("tomp101.pth.tar").to("cuda").eval()
model = model.head

---

UPDATE:

I've solved the issue by modifying the onnx model with graph surgeon /Reshape_7 from allowzero = 0 to allowzero = 1

Urgency

I am a student, and my professor gave me this project to convert this mode for inference to TensorRT through ONNX.
I have successfully converted and run inference on TensorRT, but the models accuracy is very bad, so because this model can't even run on onnxruntime, I'm guessing, maybe I can fix the accuracy problem, if I fix the other ones

Platform

Windows

OS Version

10

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

latest

ONNX Runtime API

Python

Architecture

X64

Execution Provider

Default CPU, CUDA

Execution Provider Library Version

CUDA 12.1

[xadupre]

Did you try with do_constant_folding=False? I don't know if it is a bug. It seems the converter tries to fold a constant but its inputs are on different devices. You should set this option to False if you plan to train the model with onnxruntime. You can alos export the model with inputs on cpu and use the onnx model on gpu.

[ninono12345]

UPDATE:

I've solved the issue by modifying the onnx model with graph surgeon /Reshape_7 from allowzero = 0 to allowzero = 1

and the error that says that there are cpu and cuda devices, it seems that I actually missed to transfer some tensors and submodules to cpu (I did conversion to onnx on cpu)