Batch inference time

[omidb]

Hi,

We have successfully created batch version of the model using onnx and trt. We are trying this on a A10 GPU, here is what we have observed: for a batch of 16 we get 96ms inference time and if we run that in non-batch mode we are getting 224ms for 16 images.

I wanted to check these numbers with you and see if they make sense. Also, do you have a batch implementation that we can compare against?

[jaybdub]

Hi @omidb ,

Thanks for reaching out.

I haven't tested a batch implementation yet, but would let you know if that changes.

Best,
John