modes/export/

[giscus[bot]]

modes/export/

Step-by-step guide on exporting your YOLOv8 models to various format like ONNX, TensorRT, CoreML and more for deployment. Explore now!.

https://docs.ultralytics.com/modes/export/

[alxAgu]

Where can we find working examples of a tf.js exported model?

[pderrenger]

@alxAgu currently, we don't have specific examples of a tf.js exported model directly within the YOLOv8 documentation. However, exporting a model to TensorFlow.js (tf.js) format typically involves converting the model to a format compatible with tf.js and then loading and running it within a web environment. For the export part you can use i.e.

yolo export model=yolov8n.pt format=tensorflowjs

For general guidance on how to use a tf.js model, you can refer to the TensorFlow.js documentation and tutorials, which provide comprehensive instructions on how to import and run models in the browser or Node.js.

If you're looking for examples specific to YOLO models, you might find community-contributed projects or tutorials by searching online for "YOLO TensorFlow.js example" or similar queries. These resources can often provide practical demonstrations of how to integrate YOLO models with tf.js.

[alxAgu]

@pderrenger, are there any plans to provide a working example of a YOLOv8 model exported to work in the browser with TensorFlow.js? There's a big appetite for this and it's honestly the only thing preventing us from recommending Ultralytics HUB to our clients.

[pderrenger]

Hi @alxAgu, thanks for reaching out! 🚀 We currently don't have a specific example for YOLOv8 models exported to work with TensorFlow.js. However, we appreciate the feedback and understand the demand for such examples. Your suggestion is definitely on our radar, and we'll consider it for future updates to better support web-based applications. Meanwhile, you can export the model to TensorFlow.js format using the command:

yolo export model=yolov8n.pt format=tfjs

For integrating the exported model into a browser environment, you might find the TensorFlow.js documentation helpful. Thanks for recommending Ultralytics, and we hope to meet your needs soon!

[Bharathteja07]

How to use exported engine file for inference of images in a directory?

[pderrenger]

Hey there! It looks like you're encountering an error while trying to deserialize a TensorRT engine. This error often occurs if there's a mismatch between the TensorRT version used to create the .engine file and the version you're using to load it. Here are a few steps you can try to resolve this issue:

1. Version Check: Ensure that the TensorRT version used for creating the .engine file is the same as the one you're using to load it. Version mismatches can lead to serialization errors.

2. Re-export the Model: If possible, re-export your YOLOv8 model to the TensorRT .engine format using the same environment where you're attempting to load it. This ensures compatibility.

3. TensorRT Documentation: Consult the TensorRT documentation for guidance on engine serialization and deserialization. There might be specific flags or steps required for your version.

4. Jetson Nano Deployment: For deploying on the NVIDIA Jetson Nano with JetPack 4.6.4, ensure that all components (CUDA, cuDNN, TensorRT) are correctly installed and match the versions expected by your .engine file. The NVIDIA Jetson Linux Developer Guide can be a helpful resource.

5. Sample Code: When loading and running inference with the .engine file, ensure your code correctly handles input preprocessing and output parsing specific to YOLOv8. The TensorRT Developer Guide provides examples on how to perform these steps.

If you continue facing issues, consider sharing more details about your environment and steps followed on the Ultralytics YOLOv8 GitHub Issues page. The community and developers are quite responsive and may offer more targeted advice. Good luck! 🚀

[Bharathteja07]

How can we integrate .engine model file with SAHI. Does SAHI support .engine files? my images are of size 5328*4206

[pderrenger]

Hey there! 😊 Currently, SAHI doesn't natively support .engine files directly. However, you can still use your TensorRT .engine model with SAHI by creating a custom prediction function. Here's a simplified approach:

1. Load Your Engine: Use TensorRT APIs to load your .engine file as you normally would.
2. Custom Prediction Function: Implement a function that takes an image, performs preprocessing, runs it through the TensorRT engine, and then post-processes the outputs to the format SAHI expects.
3. Integrate with SAHI: Use SAHI's get_prediction method to integrate your custom prediction function.

For large images like yours (5328x4206), consider tiling them into smaller patches that match the input size expected by your model, process each tile, and then aggregate the results.

Here's a very basic snippet to get you started:

import tensorrt as trt
import cv2
import numpy as np

TRT_LOGGER = trt.Logger(trt.Logger.WARNING)
runtime = trt.Runtime(TRT_LOGGER)

def load_engine(engine_path):
    with open(engine_path, 'rb') as f:
        engine_data = f.read()
    engine = runtime.deserialize_cuda_engine(engine_data)
    return engine

def custom_predict(image, engine):
    # Preprocess your image to match the model's expected input
    # Run inference with the engine
    # Post-process the engine's output to match SAHI's expected format
    pass

# Example usage
engine = load_engine('path/to/your/yolov8n.engine')
image = cv2.imread('path/to/your/image.jpg')
predictions = custom_predict(image, engine)

Remember, you'll need to adapt the custom_predict function to handle your specific preprocessing, inference, and post-processing steps.

For deploying on NVIDIA Jetson Nano, ensure your JetPack version is compatible with the TensorRT version used to generate the .engine file. If you encounter any issues, consider re-exporting your model with the TensorRT version available on your Jetson Nano.

Hope this helps you get started! If you have more questions or need further assistance, feel free to ask. Happy coding! 🚀

[Bharathteja07]

I am trying to run this on dGPU, I tried using ONNX with SAHI but it is taking a lot of time for processing each image.

1. As it is mentioned in the docs that ONNX (3x CPU) and Tensorrt (5X GPU), does that mean onnx doesnt use GPU? (BTW I have choosen cuda:0".
2. Could you go more in details of custom_predict function mentioned above? This I will be using with tenssort engine files

[pderrenger]

Hey there! 🌟 It looks like you're diving deep into optimizing your model's performance on a dGPU. Let's address your questions:

1. ONNX and GPU Usage: While ONNX itself is a format and doesn't dictate where the inference runs, the framework you use to run ONNX models (like ONNX Runtime) can indeed leverage GPUs. Specifying cuda:0 should direct ONNX Runtime to use your GPU. If you're experiencing slow processing, it might be worth exploring the batch size, model optimization, or the specific ONNX Runtime settings you're using.

2. Custom Predict Function Details: The custom_predict function is essentially where you'd handle the loading of your TensorRT engine, preprocessing of your input images (resizing, normalization), executing the inference, and then post-processing the outputs (decoding the detections). Since you're interested in using TensorRT .engine files, here's a simplified structure of what this function might look like:

import tensorrt as trt
import pycuda.driver as cuda
import pycuda.autoinit  # This is needed for initializing CUDA driver

def custom_predict(image, engine):
    # Assuming 'image' is your preprocessed input image ready for the model
    # And 'engine' is your loaded TensorRT engine

    # Allocate buffers for input and output
    # [Your code to allocate buffers]

    # Create execution context
    context = engine.create_execution_context()

    # Copy input data to the buffer
    # [Your code to copy data to input buffer]

    # Run inference
    context.execute_v2(bindings=[int(d_input), int(d_output)])

    # Copy results from buffer
    # [Your code to copy data from output buffer]

    # Post-process the results to extract bounding boxes, etc.
    # [Your post-processing code]

    return processed_results

This is a very high-level overview. You'll need to fill in the details based on your specific model input/output formats and the preprocessing/postprocessing required.

For deploying on the NVIDIA Jetson Nano with JetPack 4.6.4, ensure you've installed the compatible versions of TensorRT and other dependencies. The Jetson platform is fantastic for edge AI applications, and optimizing your model with TensorRT can significantly boost inference speed.

Regarding the error you encountered, it suggests a version mismatch or an issue with the engine file itself. Double-check the TensorRT versions used for creating and loading the engine, and consider regenerating the engine file if possible.

Hope this helps clear things up! If you have more questions or need further assistance, feel free to reach out. Happy coding! 🚀

[biodatasciencearg]

I trained a custom model taking yolov8n.pt (backbone) and I want to do a model registry in MLFLOW of the model in the .engine format. It's possible directly without the export step? Someone deal with something similar? Tks for your help!

[pderrenger]

Hey there! 👋 It seems like there's a discrepancy in the class detection between your custom YOLOv8 model and the TensorRT .engine file. This usually happens due to a mismatch in the class labels or an issue during the conversion process. Here are a couple of things you might want to check:

1. Class Labels File: Ensure that the class labels file used during training is correctly referenced during the export process. The number of classes should match your custom dataset.

2. Export Command: Your export command looks good, but double-check if the model path and other parameters are correctly specified. Also, ensure that the model argument points to your custom trained .pt file.

3. TensorRT Conversion: After converting to .engine, verify that the conversion process didn't introduce any changes to the class predictions. Sometimes, the conversion process might default to a pre-set number of classes (like 91 for COCO).

Here's a simplified version of your export command for clarity:

from ultralytics import YOLO

# Load your custom trained model
model = YOLO('/content/drive/MyDrive/Colab Notebooks/people_count1/head_model.pt')

# Export to TensorRT .engine format
model.export(format='engine', imgsz=640, half=True)

If you're still facing issues, it might be helpful to explicitly specify the number of classes during the export process if the API allows, or check the documentation for any additional steps required for custom models.

Hope this helps! Let us know if you have any more questions. 😊

[sanvir19]

"I am training my custom model using the pre-trained model YOLOv8-L, which has 91 classes. When I perform inference with my custom model, it works correctly. However, after converting the custom model to TensorRT, the converted model no longer contains my custom classes; instead, it retains the 91 classes from the pre-trained model."

I am using the following code snippet to convert my model to TensorRT:
from ultralytics import YOLO
model = YOLO('custom_model.pt')
model.export(format='engine',
imgsz=640,
optimize=False,
half=True,
int8=False,
dynamic=True,
opset=None)
I have provided the correct path to the custom model.

[pderrenger]

Hey there! 👋 It looks like the issue you're encountering with the TensorRT conversion retaining the original 91 classes from the pre-trained YOLOv8-L model, instead of your custom classes, might be related to how the model's class information is handled during the export process.

When exporting your custom model to TensorRT, it's crucial to ensure that the class information from your custom dataset is correctly preserved. The discrepancy in class numbers suggests that the export process might not be correctly applying your custom model's class structure.

Here's a quick suggestion:

• Double-check that your custom dataset's class labels are correctly defined and accessible during the export process. This might involve specifying the path to your dataset's class labels file if required by the export function.

Unfortunately, the Ultralytics export function doesn't explicitly allow for specifying class labels during the export. However, the model should inherently carry over its learned classes from training. It's possible that during the conversion to TensorRT, the default COCO class labels are being assumed.

If you continue to face issues, consider reaching out on the Ultralytics GitHub issues page or forums with details of your training setup and export process. The community or the developers might offer insights specific to your scenario.

For now, ensure your custom model is correctly recognizing classes post-training and pre-export to isolate the issue to the export process. 🛠️

Hope this helps a bit! Let me know if there's anything else I can do for you.

[sanvir19]

Yes, you are correct. When I am trying to perform inference with my model, it loads its default model trained on the COCO dataset. When I found out about this issue, I provided the path to load my custom model, but then I encountered the error "jetson.inference -- detectNet failed to load network," which indicates that the model may not be loading correctly.

Could you please guide me on how to properly load the model through the entire process?

i use this code for inference:
import jetson.inference
import jetson.utils
import cv2
import time

net = jetson.inference.detectNet(argv=['--model=/home/nano/jetson-inference/data/networks/yolov8/head_mode_newl.engine',
'--labels=/home/nano/jetson-inference/data/networks/yolov8/lable.txt',
'--input-blob=input_0',
'--output-cvg=scores',
'--output-bbox=boxes'], threshold=0.5)

num_classes = net.GetNumClasses()
class_names = [net.GetClassDesc(i) for i in range(num_classes)]

[pderrenger]

@sanvir19 hey there! 👋 It looks like you're encountering an issue where your custom model, when converted to TensorRT and used with jetson.inference, defaults back to detecting the original 91 classes from the pre-trained model rather than your custom classes for head detection.

This issue might stem from how the TensorRT engine is being loaded, especially if the class labels aren't being correctly applied or recognized. Here's a suggestion to ensure your custom classes are correctly utilized:

1. Verify Class Labels: Ensure that your labels.txt file correctly lists your custom classes, and it's in the correct format expected by jetson.inference.

2. Correct Model Path: Double-check the path to your .engine file to ensure it's correctly pointing to your custom model.

3. TensorRT Conversion: When converting your model to TensorRT, ensure that the conversion process is aware of your custom classes. This might involve specifying class labels during the conversion process if supported.

Your code snippet for conversion looks good, but you might want to ensure that the --labels argument in your detectNet initialization points to the correct file and that the file is formatted correctly. Each class should be on a new line, corresponding to the class IDs your model predicts.

If you're still facing issues, consider revisiting the conversion step to ensure it's correctly tailored for your custom classes. Unfortunately, the direct integration with jetson.inference might have limitations or specific requirements for class labels and model formats, so checking the documentation for any additional steps or requirements might also help.

Hope this helps you move forward! If you have more questions or need further assistance, feel free to ask. 😊

[mrawding]

Hi, I appreciate the really awesome work within Ultralytics. I have a simple question.

What is the difference between best_int8.tflite and best_integer_quant.tflite is the quant variation simplifed more than int8 for edge devices? Curious on which one is "better" or at least the differences between the two

[pderrenger]

Hey there! Thanks for the kind words and for reaching out with your question. 😊

The best_int8.tflite and best_integer_quant.tflite files are both TensorFlow Lite models that have been quantized to use 8-bit integer weights and activations, which can reduce the model size and speed up inference, particularly on edge devices.

The main difference typically lies in the quantization technique used. best_int8.tflite usually refers to a model quantized using full integer quantization with int8 precision, while best_integer_quant.tflite might be using a different or more specific form of integer quantization.

As for which one is "better," it really depends on your specific use case and the device you're deploying to. Full integer quantization (best_int8.tflite) is generally more compatible with a wider range of devices, including those with dedicated hardware accelerators for int8 operations. However, if you're targeting a device that supports the specific quantization used in best_integer_quant.tflite, that might be the better choice for you.

In terms of simplicity, both are optimized for edge devices, but you might find one performs better than the other in your particular scenario. It's often a good idea to test both on your target device to see which gives you the best balance of speed and accuracy.

Hope that helps clear things up! If you need more detailed guidance, feel free to check out our export documentation or ask here. Happy coding! 🚀

[himanshu1027]

Hello @pderrenger

Can you plz help me out with how can i use Paddlepaddle Format to extract the text from the images?

Your response is very imp to me i am waiting for your reply.

[pderrenger]

Hello @himanshu1027,

To extract text from images using a model exported in PaddlePaddle format, you'll first need to ensure your model is trained for the task of text detection or OCR (Optical Character Recognition). Once you have a suitable model, you can export it to the PaddlePaddle format using the following code snippet:

from ultralytics import YOLO

# Load your trained model
model = YOLO('path/to/your/trained_model.pt')

# Export the model to PaddlePaddle format
model.export(format='paddle')

After exporting, you can use the PaddlePaddle inference engine to load the model and perform text detection on your images. Here's a basic example of how you might set up the inference:

import paddle
from paddle.inference import Config, create_predictor

# Setup the model
config = Config("path/to/your_model/model.pdmodel", "path/to/your_model/model.pdiparams")
config.enable_use_gpu(100, 0)
predictor = create_predictor(config)

# Prepare your image (assuming 'img' is your preprocessed image tensor)
input_names = predictor.get_input_names()
input_tensor = predictor.get_input_handle(input_names[0])
input_tensor.reshape(img.shape)
input_tensor.copy_from_cpu(img.numpy())

# Run inference
predictor.run()
output_names = predictor.get_output_names()
output_tensor = predictor.get_output_handle(output_names[0])
output_data = output_tensor.copy_to_cpu()

# Process the output_data to extract text

Make sure to adjust the preprocessing and postprocessing steps according to your specific model requirements and the format of your input data.

For more detailed guidance on using PaddlePaddle for inference, you might want to check out the PaddlePaddle documentation.

Hope this helps! Let me know if you have any more questions. 😊

[idaymand]

my code

from ultralytics import YOLO
import tensorflow as tf
import onnx2tf as onnx2tf

model = YOLO('yolov8n_web_model/yolov8n.pt') # load an official model

model = YOLO('/path_to_model/best.pt')
model.export(format="tfjs")

i got an error

ERROR: The trace log is below.
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.12/site-packages/onnx2tf/utils/common_functions.py", line 310, in print_wrapper_func
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/onnx2tf/utils/common_functions.py", line 383, in inverted_operation_enable_disable_wrapper_func
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/onnx2tf/ops/Conv.py", line 246, in make_node
input_tensor = get_padding_as_op(
^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/onnx2tf/utils/common_functions.py", line 2031, in get_padding_as_op
return tf.pad(x, padding)
^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/home/ubuntu/.local/lib/python3.12/site-packages/keras/src/backend/common/keras_tensor.py", line 92, in tf_tensor
raise ValueError(
ValueError: A KerasTensor cannot be used as input to a TensorFlow function. A KerasTensor is a symbolic placeholder for a shape and dtype, used when constructing Keras Functional models or Keras Functions. You can only use it as input to a Keras layer or a Keras operation (from the namespaces keras.layers and keras.operations). You are likely doing something like:

x = Input(...)
...
tf_fn(x)  # Invalid.

What you should do instead is wrap tf_fn in a layer:

class MyLayer(Layer):
    def call(self, x):
        return tf_fn(x)

x = MyLayer()(x)

ERROR: input_onnx_file_path: /home/ubuntu/Python/runs/detect/train155/weights/best.onnx
ERROR: onnx_op_name: Conv_0
ERROR: Read this and deal with it. https://github.com/PINTO0309/onnx2tf#parameter-replacement
ERROR: Alternatively, if the input OP has a dynamic dimension, use the -b or -ois option to rewrite it to a static shape and try again.
ERROR: If the input OP of ONNX before conversion is NHWC or an irregular channel arrangement other than NCHW, use the -kt or -kat option.
ERROR: Also, for models that include NonMaxSuppression in the post-processing, try the -onwdt option.

TensorFlow SavedModel: export failure ❌ 7.4s: SavedModel file does not exist at: /home/ubuntu/Python/runs/detect/train155/weights/best_saved_model/{saved_model.pbtxt|saved_model.pb}

what is wrong and what i need to do for fix?
my env
python 3.8
yolov8
tensorflow 2.16

thanks a lot

[pderrenger]

@idaymand hey there! 👋 It looks like you're encountering an error while trying to export a YOLOv8 model to TensorFlow.js (tfjs) format. The error message suggests an issue with handling KerasTensors directly in TensorFlow functions, which typically happens when attempting operations not directly supported by Keras layers or operations.

From your code snippet, it seems the error occurs during the conversion process, possibly within the onnx2tf conversion step, which is not directly related to the Ultralytics export functionality. However, to address your main concern about exporting to tfjs, here's a simplified approach using Ultralytics YOLO directly for tfjs export:

from ultralytics import YOLO

# Assuming '/path_to_model/best.pt' is your trained model
model = YOLO('/path_to_model/best.pt')  
model.export(format="tfjs") 

Make sure your environment is set up correctly with the required versions of TensorFlow and Ultralytics YOLO. Since you're using Python 3.8 and TensorFlow 2.16, the setup seems appropriate.

If you're still facing issues, consider the following steps:

1. Ensure that your model is correctly loaded before attempting the export.
2. Verify that all dependencies, especially those related to Ultralytics YOLO and TensorFlow, are correctly installed and compatible.
3. If the problem persists, you might want to explore exporting to ONNX format first and then converting to TensorFlow.js using a more direct approach or tool specifically designed for ONNX to TensorFlow.js conversion.

For more detailed guidance on exporting models and troubleshooting, the Ultralytics documentation on model export is a great resource. It covers a wide range of export formats and provides tips for optimizing your exported models.

Hope this helps! If you have further questions or run into more issues, feel free to reach out. 😊

[idaymand]

Hi @pderrenger! I see that by default ultralytics do export to onnx as step in export process yolov8 model to tfjs. I get an error on the step of export from onnx to tfjs. Log is below

my code

from ultralytics import YOLO

model = YOLO('/my_path/best.pt')
model.export(format="tfjs") # export the model to tfjs format

log

PyTorch: starting from '/home/ubuntu/Python/runs/detect/train155/weights/best.pt' with input shape (1, 3, 640, 640) BCHW and output shape(s) (1, 103, 8400) (6.7 MB)

TensorFlow SavedModel: starting export with tensorflow 2.12.0...

ONNX: starting export with onnx 1.14.0 opset 12...
ONNX: simplifying with onnxsim 0.4.35...
ONNX: export success ✅ 1.8s, saved as '/home/ubuntu/Python/runs/detect/train155/weights/best.onnx' (13.0 MB)
TensorFlow SavedModel: running 'onnx2tf -i "/home/ubuntu/Python/runs/detect/train155/weights/best.onnx" -o "/home/ubuntu/Python/runs/detect/train155/weights/best_saved_model" -nuo --non_verbose'

ERROR: The trace log is below.
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.12/site-packages/onnx2tf/utils/common_functions.py", line 310, in print_wrapper_func
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/onnx2tf/utils/common_functions.py", line 383, in inverted_operation_enable_disable_wrapper_func
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/onnx2tf/ops/Conv.py", line 246, in make_node
input_tensor = get_padding_as_op(
^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/onnx2tf/utils/common_functions.py", line 2031, in get_padding_as_op
return tf.pad(x, padding)
^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.local/lib/python3.12/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/home/ubuntu/.local/lib/python3.12/site-packages/keras/src/backend/common/keras_tensor.py", line 92, in tf_tensor
raise ValueError(
ValueError: A KerasTensor cannot be used as input to a TensorFlow function. A KerasTensor is a symbolic placeholder for a shape and dtype, used when constructing Keras Functional models or Keras Functions. You can only use it as input to a Keras layer or a Keras operation (from the namespaces keras.layers and keras.operations). You are likely doing something like:

x = Input(...)
...
tf_fn(x)  # Invalid.

What you should do instead is wrap tf_fn in a layer:

class MyLayer(Layer):
    def call(self, x):
        return tf_fn(x)

x = MyLayer()(x)

ERROR: input_onnx_file_path: /home/ubuntu/Python/runs/detect/train155/weights/best.onnx
ERROR: onnx_op_name: Conv_0
ERROR: Read this and deal with it. https://github.com/PINTO0309/onnx2tf#parameter-replacement
ERROR: Alternatively, if the input OP has a dynamic dimension, use the -b or -ois option to rewrite it to a static shape and try again.
ERROR: If the input OP of ONNX before conversion is NHWC or an irregular channel arrangement other than NCHW, use the -kt or -kat option.
ERROR: Also, for models that include NonMaxSuppression in the post-processing, try the -onwdt option.

TensorFlow SavedModel: export failure ❌ 5.7s: SavedModel file does not exist at: /home/ubuntu/Python/runs/detect/train155/weights/best_saved_model/{saved_model.pbtxt|saved_model.pb}

---

OSError Traceback (most recent call last)
Cell In[29], line 5
1 from ultralytics import YOLO
4 model = YOLO('/home/ubuntu/Python/runs/detect/train155/weights/best.pt')
----> 5 model.export(format="tfjs") # export the model to tfjs format

File ~/Python/python-environments/Torch_1.8/lib/python3.8/site-packages/ultralytics/engine/model.py:313, in Model.export(self, **kwargs)
311 custom = {'imgsz': self.model.args['imgsz'], 'batch': 1, 'data': None, 'verbose': False} # method defaults
312 args = {**self.overrides, **custom, **kwargs, 'mode': 'export'} # highest priority args on the right
--> 313 return Exporter(overrides=args, _callbacks=self.callbacks)(model=self.model)

File ~/Python/python-environments/Torch_1.8/lib/python3.8/site-packages/torch/autograd/grad_mode.py:28, in _DecoratorContextManager.call..decorate_context(*args, **kwargs)
25 @functools.wraps(func)
26 def decorate_context(*args, **kwargs):
27 with self.class():
---> 28 return func(*args, **kwargs)

File ~/Python/python-environments/Torch_1.8/lib/python3.8/site-packages/ultralytics/engine/exporter.py:261, in Exporter.call(self, model)
259 if any((saved_model, pb, tflite, edgetpu, tfjs)): # TensorFlow formats
260 self.args.int8 |= edgetpu
--> 261 f[5], keras_model = self.export_saved_model()
262 if pb or tfjs: # pb prerequisite to tfjs
263 f[6], _ = self.export_pb(keras_model=keras_model)

File ~/Python/python-environments/Torch_1.8/lib/python3.8/site-packages/ultralytics/engine/exporter.py:122, in try_export..outer_func(*args, **kwargs)
120 except Exception as e:
121 LOGGER.info(f'{prefix} export failure ❌ {dt.t:.1f}s: {e}')
--> 122 raise e

File ~/Python/python-environments/Torch_1.8/lib/python3.8/site-packages/ultralytics/engine/exporter.py:117, in try_export..outer_func(*args, **kwargs)
115 try:
116 with Profile() as dt:
--> 117 f, model = inner_func(*args, **kwargs)
118 LOGGER.info(f"{prefix} export success ✅ {dt.t:.1f}s, saved as '{f}' ({file_size(f):.1f} MB)")
119 return f, model

File ~/Python/python-environments/Torch_1.8/lib/python3.8/site-packages/ultralytics/engine/exporter.py:712, in Exporter.export_saved_model(self, prefix)
709 for file in f.rglob('*.tflite'):
710 f.unlink() if 'quant_with_int16_act.tflite' in str(f) else self._add_tflite_metadata(file)
--> 712 return str(f), tf.saved_model.load(f, tags=None, options=None)

File ~/Python/python-environments/Torch_1.8/lib/python3.8/site-packages/tensorflow/python/saved_model/load.py:836, in load(export_dir, tags, options)
834 if isinstance(export_dir, os.PathLike):
835 export_dir = os.fspath(export_dir)
--> 836 result = load_partial(export_dir, None, tags, options)["root"]
837 return result

File ~/Python/python-environments/Torch_1.8/lib/python3.8/site-packages/tensorflow/python/saved_model/load.py:941, in load_partial(export_dir, filters, tags, options)
936 if tags is not None and not isinstance(tags, set):
937 # Supports e.g. tags=SERVING and tags=[SERVING]. Sets aren't considered
938 # sequences for nest.flatten, so we put those through as-is.
939 tags = nest.flatten(tags)
940 saved_model_proto, debug_info = (
--> 941 loader_impl.parse_saved_model_with_debug_info(export_dir))
943 if (len(saved_model_proto.meta_graphs) == 1 and
944 saved_model_proto.meta_graphs[0].HasField("object_graph_def")):
945 metrics.IncrementReadApi(_LOAD_V2_LABEL)

File ~/Python/python-environments/Torch_1.8/lib/python3.8/site-packages/tensorflow/python/saved_model/loader_impl.py:58, in parse_saved_model_with_debug_info(export_dir)
45 def parse_saved_model_with_debug_info(export_dir):
46 """Reads the savedmodel as well as the graph debug info.
47
48 Args:
(...)
56 parsed. Missing graph debug info file is fine.
57 """
---> 58 saved_model = parse_saved_model(export_dir)
60 debug_info_path = file_io.join(
61 path_helpers.get_debug_dir(export_dir),
62 constants.DEBUG_INFO_FILENAME_PB)
63 debug_info = graph_debug_info_pb2.GraphDebugInfo()

File ~/Python/python-environments/Torch_1.8/lib/python3.8/site-packages/tensorflow/python/saved_model/loader_impl.py:116, in parse_saved_model(export_dir)
114 raise IOError(f"Cannot parse file {path_to_pbtxt}: {str(e)}.")
115 else:
--> 116 raise IOError(
117 f"SavedModel file does not exist at: {export_dir}{os.path.sep}"
118 f"{{{constants.SAVED_MODEL_FILENAME_PBTXT}|"
119 f"{constants.SAVED_MODEL_FILENAME_PB}}}")

OSError: SavedModel file does not exist at: /home/ubuntu/Python/runs/detect/train155/weights/best_saved_model/{saved_model.pbtxt|saved_model.pb}

Help me please to solve this problem.
Thanks a lot

[AttilaSzucs21]

Hello I had similar problem, I downgraded my tensorflow from 2.16.1 to 2.15.1 and now it works. (In my case I tried tflite conversion so maybe it wont work in your case.)

[pderrenger]

Hey! 👋 It's great to hear that downgrading TensorFlow resolved the issue for you. TensorFlow compatibility can indeed be tricky, especially with newer versions sometimes introducing changes that affect certain functionalities.

Since our export process involves intermediate steps like converting to ONNX before reaching TensorFlow formats, there might still be discrepancies or requirements specific to those versions. If downgrading worked for your situation, it definitely might help others facing similar issues. For those still encountering errors, ensuring all conversion tools and dependencies are compatible with their TensorFlow version is crucial, as outlined in our documentation and error logs. Thanks a lot for sharing your solution! 😊

[pderrenger]

Hey there! 🌟

Thanks for sharing your experience. Downgrading TensorFlow has helped resolve compatibility issues for a lot of users, especially with model conversions. While it worked for your tflite conversion, it might indeed help in resolving OPs compatibility issues that might arise with TensorFlow.js exports as well.

If you face any more challenges, consider checking if your ONNX model uses ops supported by the version of TensorFlow you're using. Sometimes, adjusting the opset version during the ONNX export can help!

Happy coding with YOLOv8! 🚀 If you need more assistance, feel free to ask.

[Ibraheem-Ahmad]

Hello!
Would anyone be able to tell me why this code does not work:
model.export(format=='onnx', int8 = True, dyamic = True, imgsz = (width,height))

the error I get is "TypeError: Model.export() takes 1 positional argument but 2 were given"

[Aisha-Hassan1]

erro ...
wasm-core-impl.ts:219
Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'byteLength')
at t.run (wasm-core-impl.ts:219:31)
at t.run (proxy-wrapper.ts:215:17)
at t.OnnxruntimeWebAssemblySessionHandler.run (session-handler.ts:84:15)
at i.run (inference-session-impl.js:94:44)
at runModel (main.html:96:55) ........

when inputTensors => array
const inputTensors = [imageTensor1, imageTensor2];

[Aisha-Hassan1]

[batch_size, channels, height, width]. If your model expects not a single image of shape ???

[glenn-jocher]

Hey there! 👋 When converting a YOLOv8 model for batch processing, you generally don't need to specify the batch size during export, especially with dynamic batching enabled. 🔄 Your export command:

model.export(format='onnx', int8=True, dynamic=True)

is perfectly fine for creating an ONNX model that can handle arrays of images. Just ensure your data is prepped correctly when making actual predictions! 📊

For GPU acceleration with ONNX, leveraging frameworks like ONNX Runtime with CUDA support would be the way to go! 🔥 Make sure your environment is set up for GPU use, and you're specifying the execution provider accordingly.

Feel free to check out more on dynamic batching and ONNX export on the Ultralytics docs page! Hope this helps, and happy coding! 🚀

[Aisha-Hassan1]

code

const inputs = {
images: [inputTensors[0], inputTensors[1]]
};

const { output0, output1 } = await yolov8.run(inputs);

erro

Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'byteLength')
at t.run (wasm-core-impl.ts:219:31)
at t.run (proxy-wrapper.ts:215:17)
at t.OnnxruntimeWebAssemblySessionHandler.run (session-handler.ts:84:15)
at i.run (inference-session-impl.js:94:44)
at runModel (main.html:107:53)

[glenn-jocher]

Hey there! 👋 It looks like there might be a slight mix-up in how you're constructing the input to your model run. When dealing with arrays of tensors in ONNX.js, you should pass a single tensor that concatenates your image tensors along the batch dimension if the model is set up for batch processing.

For your issue with inputTensors causing an error about byteLength, it's likely because the model expects one tensor that combines all input images. Instead of:

const inputs = {
    images: [inputTensors[0], inputTensors[1]]
};

You should concatenate your inputTensors into one tensor if your model expects a batch input. Unfortunately, directly passing an array like this might not work as expected with ONNX.js since it seems to expect the input as a single tensor representing the batch. The error you're encountering (TypeError: Cannot read properties of undefined (reading 'byteLength')) usually occurs when the input format to the session.run method is not as expected.

This could look different based on how your model is trained and how ort.Tensor handles batches. Ensure your model is trained with dynamic batch size if you want to process multiple images in a single forward pass. If you're not sure how to concatenate tensors in JavaScript (as direct tensor concatenation might not be as straightforward as in Python), you might have to loop over your images and process them one at a time, or look into adjusting your model to accept variable batch sizes more gracefully.

Given your current setup, if your model does not explicitly support batched inputs, you might need to process each image individually:

for(const tensor of inputTensors) {
    const inputs = { images: tensor };
    const outputs = await yolov8.run(inputs);
    // Process the outputs for each image here
}

Remember, different models expect the input in specific formats, including batch size. Ensure your preprocessing aligns with these expectations! 🚀

[JamesMcCullochDickens]

Are there any examples of getting the output of a pose estimator model in C++ using a torchscript file. I'm getting an output of shape (1, 56, 8400) for an input of size (1, 3, 640, 640) with two people in the sample picture. How should I interpret/post-process this output?

[pderrenger]

Hey there! 👋 It looks like you're diving into pose estimation with a YOLOv8 model exported as a TorchScript file. The output shape (1, 56, 8400) you're seeing is typical for pose estimation models, where 56 likely represents the number of keypoints (x, y coordinates) multiplied by the number of attributes per keypoint, and 8400 is the flattened spatial grid (for a 640x640 input, this might be 80x80 grid cells, each predicting keypoints for potential objects).

To interpret this output, you'd typically:

1. Reshape or slice the tensor to isolate keypoints for each detected person.
2. Scale the keypoints back to the original image dimensions if they're normalized.
3. Apply any necessary post-processing, like thresholding based on confidence scores if included in your output.

Here's a rough idea of how you might start to interpret the output in C++ (pseudo-code):

// Assuming output is a 1x56x8400 tensor
int num_keypoints = 28; // Example: 28 keypoints (x, y) per person
int grid_cells = 8400; // Spatial grid cells
for (int i = 0; i < grid_cells; i++) {
    // Example: Extracting the first person's keypoints
    // This part depends on how your model structures its output
    float x = output[0][i % num_keypoints * 2][i];
    float y = output[0][(i % num_keypoints * 2) + 1][i];
    // Scale x and y back to image dimensions if necessary
    // Apply any post-processing like thresholding
}

This is quite simplified and will need adjustments based on your specific model's output format. If your model includes confidence scores or other attributes per keypoint, you'll need to account for those in your parsing logic.

For more detailed guidance on exporting models and handling their outputs, our Model Export with Ultralytics YOLO page might offer some insights, even though it's more focused on the export process itself.

Hope this helps! Let us know if you have more questions. 😊

[JustinGeorgi]

I trained a yolov5 detection model a little while ago and have successfully converted that model to tensorflowjs. That tfjs model works as expected in code only slightly modified from the example available at https://github.com/zldrobit/tfjs-yolov5-example. My version of the relevant section:

const [modelWidth, modelHeight] = model.inputs[0].shape.slice(1, 3)

const input = tf.tidy(() => {
  return tf.image.resizeBilinear(tf.browser.fromPixels(imageData), [modelWidth, modelHeight]).div(255.0).expandDims(0)
})

var results = model.executeAsync(input).then(res => {
  const [boxes, scores, classes, valid_detections] = res
  const boxes_data = boxes.dataSync()
  const scores_data = scores.dataSync()
  const classes_data = classes.dataSync()
  const valid_detections_data = valid_detections.dataSync()[0]

I have now trained a yolov8 detection model on very similar data. The comments in exporter.py indicate that in that example script v8 models should be a drop in replacement for v5 models:

https://github.com/ultralytics/ultralytics/blob/main/ultralytics/engine/exporter.py#L45-L49

However, that does not seem to be the case. The v5 model output is the 4 length array of tensors (which is why the destructuring assignment works), but the v8 model output is a single tensor of shape [1, X, 8400] thus the example code results in an error complaining that the model result is non-iterable when attempting to destructure.

From what I understand, the [1, X, 8400] is the expected output shape of the v8 model. Is further processing of the v8 model required, or did I do something wrong during the pt -> tfjs export?

[pderrenger]

Hey there! 😊 It looks like you've encountered a difference in output formats between YOLOv5 and YOLOv8 models when exporting to TensorFlow.js. Indeed, YOLOv8 models output a single tensor of shape [1, X, 8400], which is different from the 4-length array of tensors you'd get from YOLOv5 models.

For YOLOv8, you'll need to process the single output tensor to extract the bounding boxes, scores, and class IDs. This involves reshaping and slicing the tensor based on your model's specific output format. Here's a simplified example of how you might start to approach this:

model.executeAsync(input).then(res => {
  // Assuming res is the single tensor output [1, X, 8400]
  const data = res.dataSync();
  
  // Example processing (you'll need to adjust based on your model specifics)
  const numDetections = data[0];
  for (let i = 0; i < numDetections; i++) {
    const offset = i * 7; // Example offset, adjust based on your output format
    const score = data[offset + 2]; // Example index for score
    // Extract other values (x, y, width, height, class) similarly
  }
  // Further processing...
});

This is a very general example, and you'll need to adjust the indices based on how your YOLOv8 model formats its output tensor. The key takeaway is that the output tensor will contain all the information you need, but it will require some additional processing to extract and use effectively.

For more detailed guidance on handling YOLOv8's output format and integrating it with TensorFlow.js, I recommend checking out our Model Export documentation, which provides insights into exporting models and might offer further clues on handling different output formats.

If you have any more questions or need further assistance, feel free to ask. Happy coding! 🚀

[JustinGeorgi]

Thank you, for the reply. I thought reshaping and slicing should be something like that, but was concerned because that doesn't make any sense with the output that I'm getting.

For example:

const numDetections = data[0];

implies that, after reshaping the results, data[0] should be an integer value. As should any values that refer to class indicies of the detected objects. There are no integer values in the result array.

I know the original pt model works, it can correctly identify objects when I do a quick yolo predict ... command line test. So now, I geuss, that leaves me with two options: the pt -> tfjs has gone wrong or my input value for executeAsync is incorrect. I didn't set any additional preprocessing parameters on the origin model build, so the only things my js code does is reshape the image to the size defined in the model (which is correct) and normalize the data.

Do the yolov8 models have any additional pre-processing that I am unaware of or is it most likely that my exported model is faulty?

[pderrenger]

Hey there! 👋 It sounds like you're diving deep into the intricacies of model outputs and preprocessing, which is fantastic! From what you've described, it seems there might be a bit of confusion regarding the output format of YOLOv8 models and how to handle them in TensorFlow.js.

Indeed, YOLOv8 models output a single tensor with a shape like [1, X, 8400], which is quite different from YOLOv5's multi-tensor output. This single tensor packs all the detection information, including bounding boxes, objectness scores, and class probabilities, in a compact form. The key to using this output effectively is to correctly parse and reshape this tensor to extract the information you need.

Regarding preprocessing, YOLOv8 models generally follow the same preprocessing steps as YOLOv5 models: resizing the input image to the model's input dimensions and normalizing pixel values to the range [0, 1]. If you've got that covered, your preprocessing should be on point! 🎯

If the output tensor doesn't seem to contain integer values where you expect them (like class indices), it might be due to how the data is packed in the tensor. The actual values you're looking for are there but might require a different approach to extract correctly.

Given your situation, it doesn't immediately sound like a fault in the pt -> tfjs conversion process, especially if the model works as expected in a yolo predict test. It's more likely about adapting the post-processing step to match YOLOv8's output format.

For a more tailored solution, I'd recommend revisiting the Model Export documentation and focusing on the specifics of handling YOLOv8's output tensor in TensorFlow.js. Unfortunately, without a direct example of parsing YOLOv8's output in TensorFlow.js, I can't provide a precise code snippet right now. However, understanding the structure of the output tensor and applying the correct tensor operations to unpack it will be key.

Keep experimenting, and don't hesitate to reach out if you hit another snag or have more questions. Happy coding! 😄

[Ibraheem-Ahmad]

I was wondering if anyone could help me with this code: I exported my custom trained yolov8n.pt model to .onnx but now my code is not working(model.export(format='onnx', int8=True, dynamic=True)). I am having trouble using the outputs after running inference.

My Code:
"import cv2
import onnxruntime as ort
import numpy as np

def load_image(image_path):
# Load the image from the specified path
original_image = cv2.imread(image_path)
if original_image is None:
raise ValueError(f"Could not load image at path: {image_path}")

# Resize the image to the required dimensions (640x640)
resized_image = cv2.resize(original_image, (640, 640))

# Rearrange the image dimensions from HWC to CHW format and add a batch dimension
image_data = np.transpose(resized_image, (2, 0, 1))
image_data = np.expand_dims(image_data, axis=0).astype(np.float32)

return original_image, image_data

def draw_bounding_boxes(image, detections, confidence_threshold=0.5):
for detection in detections:
x1, y1, x2, y2, score = detection
if score >= confidence_threshold:
start_point = (int(x1), int(y1))
end_point = (int(x2), int(y2))
color = (255, 0, 0) # Bounding box color
thickness = 2 # Bounding box thickness
image = cv2.rectangle(image, start_point, end_point, color, thickness)
return image

def main(model_path, image_path):
# Load ONNX model
ort_session = ort.InferenceSession(model_path)
input_name = ort_session.get_inputs()[0].name
input_shape = ort_session.get_inputs()[0].shape

# Load and preprocess the image
original_image, image_data = load_image(image_path)

# Run inference
detections = ort_session.run(None, {input_name: image_data})[0]

for output in detections[0]:
    print(f"Output {output}: Shape = {output.shape}, Type = {output.dtype}")
    # Print values for smaller tensors
    if output.size < 500:  # Arbitrary size threshold to avoid flooding the console
        print(f"Values:\n{output}")
# Assume detections are in the format [batch, num_detections, 6] where each detection is [x1, y1, x2, y2, score, class]
detections = detections # Assuming batch size of 1 for simplicity


# Draw bounding boxes on the original image
final_image = draw_bounding_boxes(original_image, detections)

# Save or display the image
#cv2.imwrite('output.jpg', final_image)
cv2.imshow('Detected Objects', final_image)
cv2.waitKey(0)
cv2.destroyAllWindows()

if name == "main":
model_path = 'weights\best.onnx' # Replace with your model path
image_path = "Capture.PNG" # Replace with your image path
main(model_path, image_path)
"

Error:
line 28, in draw_bounding_boxes
if score >= confidence_threshold:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

[glenn-jocher]

Hey there! It looks like you're encountering an issue with handling the output from your ONNX model after running inference. The error you're seeing is likely due to attempting to directly compare a numpy array with a scalar value, which numpy doesn't allow without specifying whether you want to use any() or all() for the comparison.

However, the root cause seems to be how you're processing the detections output from the ONNX model. Typically, the output from an ONNX model will be a numpy array, but you need to ensure you're accessing the detection results correctly. Here's a simplified approach to process the model output and draw bounding boxes:

# Assuming detections are in the format [batch, num_detections, 6] where each detection is [x1, y1, x2, y2, score, class]
# And assuming batch size of 1 for simplicity
detections = detections[0][0]  # Adjust based on your model's output format

# Draw bounding boxes on the original image
final_image = draw_bounding_boxes(original_image, detections, confidence_threshold=0.5)

Make sure the detections variable is correctly formatted before passing it to draw_bounding_boxes. The error you're encountering suggests that detections might not be unpacked correctly, leading to the comparison issue.

If you're still having trouble, could you share more details about the shape and type of the detections output right after inference? That might help pinpoint the issue further. 😊

[amarnathreddy0201]

@Ibraheem-Ahmad you got answer. I am also facing same issue.

[amarnathreddy0201]

import onnxruntime as ort
import cv2
classes = [
"person", "bicycle", "car", "motorcycle", "airplane", "bus", "train", "truck",
"boat", "traffic light", "fire hydrant", "stop sign", "parking meter", "bench",
"bird", "cat", "dog", "horse", "sheep", "cow", "elephant", "bear", "zebra",
"giraffe", "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee",
"skis", "snowboard", "sports ball", "kite", "baseball bat", "baseball glove",
"skateboard", "surfboard", "tennis racket", "bottle", "wine glass", "cup", "fork",
"knife", "spoon", "bowl", "banana", "apple", "sandwich", "orange", "broccoli",
"carrot", "hot dog", "pizza", "donut", "cake", "chair", "couch", "potted plant",
"bed", "dining table", "toilet", "tv", "laptop", "mouse", "remote", "keyboard",
"cell phone", "microwave", "oven", "toaster", "sink", "refrigerator", "book",
"clock", "vase", "scissors", "teddy bear", "hair drier", "toothbrush"
] # Truncated for brevity; fill in all COCO labels
import numpy as np

image_path = r"C:\Users\amarn\OneDrive\Pictures\father mother.jpg"

image_path = r"C:\Users\amarn\OneDrive\Pictures\Screenshots\Screenshot 2024-07-09 143306.png"
image = cv2.imread(image_path,1)
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
image = cv2.resize(image, (640, 640))

Normalize image to [0, 1]

image_normalized = image.astype(np.float32) / 255.0

Transpose the image to (channels, height, width) and add batch dimension

input_tensor = np.transpose(image_normalized, (2, 0, 1)) # (C, H, W)
input_tensor = np.expand_dims(input_tensor, axis=0) # (1, C, H, W)

ort_sess = ort.InferenceSession('yolov8n.onnx')
input_name = ort_sess.get_inputs()[0].name
output_names = [output.name for output in ort_sess.get_outputs()]
outputs = ort_sess.run(None, {input_name: input_tensor})

Extract data from output

output = outputs[0] # shape (1, 84, 8400)
output = np.squeeze(output) # shape (84, 8400)

num_classes = len(classes)
num_boxes = output.shape[1]

Assuming the format (x_center, y_center, width, height, scores...)

boxes = output[0:4, :]
scores = output[4:5, :].squeeze()
class_scores = output[5:, :]

Apply a confidence threshold

confidence_threshold = 0.4
mask = scores > confidence_threshold

boxes = boxes[:, mask]
scores = scores[mask]
class_scores = class_scores[:, mask]

Calculate class labels and scores

class_ids = np.argmax(class_scores, axis=0)
class_confidences = np.max(class_scores, axis=0)

Draw bounding boxes

for i in range(boxes.shape[1]):
x_center, y_center, width, height = boxes[:, i]
class_id = class_ids[i]
confidence = class_confidences[i]
print("#################")
# Convert from center coordinates to box coordinates
x1 = int((x_center - width / 2) )
y1 = int((y_center - height / 2) )
x2 = int((x_center + width / 2) )
y2 = int((y_center + height / 2))
print(x1, y1, x2, y2)
# Draw the box and label on the image
label = f"{classes[class_id]}: {confidence:.2f}"
print(label)
cv2.rectangle(image, (x1, y1), (x2, y2), (0, 255, 0), 2)
cv2.putText(image, label, (x1, y1 - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)

Display the image with bounding boxes

image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR) # Convert back to BGR for OpenCV
cv2.imshow("Detections", image)
cv2.waitKey(0)
cv2.destroyAllWindows()

I followed this methodology for detecting the person. but it showing airplane in the bounding boxes.

[glenn-jocher]

@amarnathreddy0201 it looks like the issue might be related to how the outputs from the ONNX model are being processed. The error suggests that the scores array is not being handled correctly. Ensure that the outputs are correctly unpacked and processed before drawing bounding boxes. You might want to check the shape and content of the detections array right after the inference to confirm it matches your expectations. If the problem persists, please verify if the issue is reproducible with the latest version of the package. For more detailed guidance, refer to the Ultralytics export documentation.

[larryKra]

"batch_size" is not in arguments as previous versions?

[glenn-jocher]

Hey there! 👋

In YOLOv8, the batch_size parameter is typically set within your dataset YAML file rather than as a direct argument in training commands. This allows for more flexibility and dataset-specific batch size configurations. Here's a quick example of how it's defined in a dataset YAML:

# Example dataset.yaml
train: /path/to/train/images
val: /path/to/val/images

# Define your batch size here
batch_size: 16
...

Then, when you run your training command, YOLOv8 will automatically pick up the batch_size from your dataset configuration. No need to pass it as a separate argument! 😊

Hope this helps! If you have any more questions, feel free to ask.

[yasirgultak]

I converted the model I trained with costum data to tflite format. Before converting, I set the int8 argument to true. But when I examined the tflite format from the netron website, I saw that the input information is still float32. Is this normal or is there a bug?

Also thank you very much for answering every question without getting bored.

[glenn-jocher]

Hey there! 😊 It's great to hear you're exploring the TFLite format for your custom model. Regarding your observation about the input information still being float32 even after setting int8 to true, this is actually expected behavior. The int8 quantization primarily affects the weights and activations within the model to reduce size and potentially speed up inference on compatible hardware. However, the input and output layers often remain as float32 for compatibility reasons, allowing you to feed in standard floating-point data and receive floating-point outputs without additional conversion steps.

If you're looking to further optimize or have specific requirements for input/output data types, you might consider post-processing steps or exploring additional TensorFlow tools for model optimization.

And no worries at all, we're here to help! If you have any more questions or need further assistance, feel free to ask. Happy coding! 🚀

[usama-sherazi]

!yolo export model=/content/drive/MyDrive/best-1-1.pt format=tflite

export failure ❌ 33.0s: generic_type: cannot initialize type "StatusCode": an object with that name is already defined
Traceback (most recent call last):
File "/usr/local/bin/yolo", line 8, in
can anyone help me to export to tflite....

[usama-sherazi]

Firstly, a big thank you to @glenn-jocher for the support.

During my search for a solution, I discovered that TensorFlow versions greater than 2.13.0 have issues with YOLO.

As Google Colab had version 2.15.0 installed, I resolved the issue by adding the following snippet before the conversion:

%pip install tensorflow==2.13.0
This ensured compatibility and resolved the problem. Thank you again for the assistance!

[glenn-jocher]

Hey there! 🎉 Big thanks for sharing your experience and solution - it's super valuable! 🙌 It's really interesting to hear about the version-specific intricacies with TensorFlow and YOLO. Downgrading TensorFlow to 2.13.0 to ensure compatibility is a clever workaround. Your discovery is a great addition to the community knowledge pool, and it might just be the key for others facing similar issues. 👍

If anyone else encounters similar hiccups, trying out different TensorFlow versions based on your model requirements could indeed be a fruitful debugging step.

Super grateful for your input, and cheers for contributing to smoother YOLO model deployments! 🚀 If there's more to ponder or if new challenges pop up, the community is here to help. Keep up the fantastic work!

[usama-sherazi]

yolo export model=/content/yolov8nsss (3).pt format=tflite

/bin/bash: -c: line 1: syntax error near unexpected token (' /bin/bash: -c: line 1: yolo export model=/content/yolov8nsss (3).pt format=tflite'

Hi i am converting Yolov8n file to tflite ia am facing this issue suggest me solution
thank you

[eladoni1]

Isnt that because your file name? I do not use the CLI, but try to add " (quotes) in the parameters, or, change the file name to something without paranthesys.

So, either :
yolo export model="/content/yolov8nsss (3).pt" format="tflite"

Or, if that doesnt work, rename :
yolov8nsss (3).pt

To :

yolov8nsss.pt

(Just without paranthesys).

[glenn-jocher]

@eladoni1 hey! Great catch on figuring out the solution with the file name! 🌟 Yes, file paths with special characters, including spaces and parentheses, can sometimes throw a wrench into command line processing. Wrapping the path in quotes or simplifying the file name are both solid approaches.

For others who might come across similar issues, here's a quick recap based on the awesome tip shared:

To handle file names with spaces or special characters when using the command line, you can either:

• Wrap the path in quotes, like so:

yolo export model="/content/yolov8nsss (3).pt" format="tflite"

• Or, rename the file to remove spaces and special characters, making it:

yolov8nsss.pt

Both methods help ensure the command line correctly understands the file path, aiding in a smooth export process. 🚀 Thanks again for highlighting this solution, and if there's anything more you're curious about or need help with, feel free to ask!

[rajazain2001]

Hi I havr tried all TFLITE export formats to convert the best.pt to .tflite but non is working. I have also checked my runtime and all the latest imports pip install -U ultralytics, and I have also tried the code you gave to someone in the comments but the issue is not resolvig

Step 1: Export to TensorFlow SavedModel

!yolo export model='/content/drive/MyDrive/best-1-1.pt' format=saved_model

Step 2: Convert the exported SavedModel to TensorFlow Lite

import tensorflow as tf
saved_model_dir = 'path/to/exported/saved_model' # Update this path
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
tflite_model = converter.convert()

Save the TFLite model

with open('/content/drive/MyDrive/yolov8_model.tflite', 'wb') as f:
f.write(tflite_model) .

but the same error comes back.
can you please provide me with some specific solution I wil be very thankful to you.
you can reach me at [email protected]

[pderrenger]

@rajazain2001 hi there,

It sounds like you're encountering issues while trying to convert your best.pt model to TensorFlow Lite format. Let's try a couple of things to troubleshoot:

1. Ensure Compatibility: Make sure that your model architecture and operations are compatible with TFLite. Some custom layers or operations might not be supported directly.

2. Update TensorFlow: Sometimes, updating TensorFlow to the latest version can resolve conversion issues:

   pip install --upgrade tensorflow

3. Use TensorFlow 2.x: Ensure you are using TensorFlow 2.x, as TensorFlow 1.x might not support some functionalities.

4. Check the Error Message: The specific error message you're receiving can give more clues on what might be going wrong.

5. Simplify the Model: If possible, try simplifying the model or using a standard model to test if the issue is with the model complexity.

If these steps don't resolve the issue, could you please share the specific error message you're encountering? This will help in diagnosing the problem more effectively.

For more detailed guidance on exporting models, you might find our export documentation helpful.

[itachi1232gg]

can we export sam/mobile sam model to tensorRT or onnx?

[pderrenger]

Absolutely! You can export SAM/MobileSAM models to both TensorRT and ONNX formats using the Ultralytics YOLOv8 export functionality. Here's how you can do it:

For ONNX:

yolo export model=path/to/your/sam_model.pt format=onnx

For TensorRT:

yolo export model=path/to/your/sam_model.pt format=engine

Just replace path/to/your/sam_model.pt with the actual path to your trained SAM or MobileSAM model. Let me know if you need further assistance! 😊

[itachi1232gg]

Thanks for answering, I tried yolo export model=path/to/your/sam_model.pt format=engine , it raises error : AttributeError: 'Sam' object has no attribute 'args'. SAM and YOLO extend from different base class.

[pderrenger]

Hey there! 👋 Thanks for reaching out with your issue. It seems like you've encountered an attribute error concerning the SAM model during export to TensorRT. This error suggests the SAM class is missing an expected attribute, likely due to differences in the base class structures between SAM and YOLO models.

A potential workaround could involve ensuring that your SAM model object has all necessary attributes before export. However, without diving deeper into the codebase, it's challenging to provide an exact solution.

Could you try exporting the model to ONNX first and then converting that ONNX file to TensorRT? This two-step process might bypass the immediate issue:

# Export to ONNX
yolo export model=path/to/your/sam_model.pt format=onnx

# Convert ONNX to TensorRT (using an external tool or script)

If this doesn't resolve the issue, could you provide more details about your SAM model and the environment you're working in? That could give us better insight into the problem. 🛠️

Also, we're continuously improving our models and export capabilities. So, your feedback is invaluable for making YOLOv8 even better!

[itachi1232gg]

I think exporter only supports yolo models. In order to convert SAM model to ONNX or TensorRT, I should write anther convert script.

[pderrenger]

Hey! 👋 Thanks for reaching out. Yes, the current exporter is primarily designed for YOLO models. However, for SAM/MobileSAM models, converting to ONNX or TensorRT does indeed require a bit of custom handling.

To get started with ONNX export, you might consider something along these lines after your model is loaded:

import torch.onnx

# Assuming `model` is your SAM/MobileSAM model
torch.onnx.export(model, dummy_input, 'model_name.onnx', verbose=True, input_names=['input'], output_names=['output'])

This code snippet assumes a dummy input that matches the expected dimensions of the model.

For TensorRT, after exporting to ONNX, tools like onnx-tensorrt can help with the conversion:

onnx2trt model_name.onnx -o model_name.trt

Although these steps are a good starting point, depending on the SAM model's specifics, adjustments might be necessary. If you encounter challenges along the way or need further assistance, feel free to ask! We're here to help. 😊

[seungdae251]

Hi. Is there a way to export the model file into a trt (.engine) file that can use DLA(Deep Learning Accelerator) core on an edge device like Jetson?

[glenn-jocher]

@seungdae251 to export a YOLO model to a TensorRT (.engine) file for use with a DLA on Jetson, use the format='engine' option during export. Ensure your Jetson environment supports TensorRT and DLA configurations. For detailed steps, please refer to the Ultralytics export documentation.

[AlgosDuCouteau]

Hi, can you show me an example code to use with the engine format? Is

model = YOLO("yolov8l.engine")
results = model.predict("datasets/coco8/images/train/000000000009.jpg", save=True, half=True, device=0)

good? since with the .pt the model can detect the correct objects but not for the .engine. anyway this is how i export to .engine

out = model.export(format="engine", half=True, device=0)

[glenn-jocher]

@AlgosDuCouteau your code for using the .engine format looks mostly correct. Ensure that the exported model is compatible with your hardware and that all necessary dependencies for TensorRT are installed. If detection issues persist, verify the export process and consider testing with different export settings. For more details, refer to the Export Documentation.

[abdullah000123]

Thnks

…

On Mon, Oct 14, 2024, 8:07 AM AlgosDuCouteau ***@***.***> wrote: Hi, can you show me an example code to use with the engine format? Is model = YOLO("yolov8l.engine") results = model.predict("datasets/coco8/images/train/000000000009.jpg", save=True, half=True, device=0) good? since with the .pt the model can detect the correct objects but not for the .engine. anyway this is how i export to .engine out = model.export(format="engine", half=True, device=0) — Reply to this email directly, view it on GitHub <#7933 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BECV2N4XNXNSBOHSPYJJ4VTZ3MYQZAVCNFSM6AAAAABCTE3SQOVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAOJTGE4TAMA> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[glenn-jocher]

@abdullah000123 to use the .engine format, ensure your export command is correct. Your code snippet looks fine, but if detection isn't working as expected, verify that TensorRT is properly installed and compatible with your hardware. For more details, visit the export documentation: https://docs.ultralytics.com/modes/export/.

[zoldaten]

hello! thanks for yolo.

is it possible to change image input type(to uint8) and outputs(score,location,number of detections) while conversion to tflite ?
i did conversion with:
yolo export model=yolov8n.pt imgsz=640 format=tflite int8
it`s ok.
but it doesnt match a bit with android app:
(EfficientDet-Lite4)
(https://drive.google.com/file/d/1xfUOLgD5hGoRVvBsG2mHjIK_2yboOI6i/view?usp=sharing)
but yolo gives float32 as input and output:
(yolov8n)
(https://drive.google.com/file/d/1Jo2zN4OxFo0EiFNXWVd1d8mfNtGhD32z/view?usp=sharing)

[glenn-jocher]

@zoldaten hello! To change the input type to uint8 and adjust outputs during TFLite conversion, you'll need to modify the model's input and output layers manually after export. Unfortunately, direct support for uint8 inputs isn't available in the export command. For more details, please refer to the TFLite documentation.

[LJ-Hao]

model = YOLO("/home/jiahao/Tutorial-of-AI-Kit-with-Raspberry-Pi-From-Zero-to-Hero/runs/detect/train12/weights/best.pt")

model.export(format="saved_model")

get an error :

Ultralytics 8.3.13 🚀 Python-3.10.12 torch-2.4.1+cu121 CPU (AMD Ryzen 5 5600G with Radeon Graphics)
YOLO11n summary (fused): 238 layers, 2,582,737 parameters, 0 gradients, 6.3 GFLOPs

PyTorch: starting from '/home/jiahao/Tutorial-of-AI-Kit-with-Raspberry-Pi-From-Zero-to-Hero/runs/detect/train12/weights/best.pt' with input shape (1, 3, 640, 640) BCHW and output shape(s) (1, 7, 8400) (5.2 MB)

TensorFlow SavedModel: starting export with tensorflow 2.12.0...
TensorFlow SavedModel: export failure ❌ 0.0s: Another metric with the same name already exists.

[glenn-jocher]

It seems you're encountering an error related to TensorFlow's metrics during export. This might be due to a naming conflict in your environment. Try resetting your TensorFlow session or environment and attempt the export again. For more detailed guidance, please refer to the export documentation: https://docs.ultralytics.com/modes/export/.

[faizaanscnz]

I am trying to load a YOLO TensorFlow Lite model using the GPU, but the current approach is not working. I have attempted loading with
model = YOLO('yolo11n_float32.tflite').to(device)
and with the argument
model = YOLO('yolo11n_float32.tflite', device='gpu'),
but neither seems to activate GPU usage. Could you advise on the correct way to load a YOLO TFLite model with GPU support?

[glenn-jocher]

@faizaanscnz to use a YOLO TensorFlow Lite model with GPU support, ensure your device supports TFLite GPU delegate. You may need to configure the TFLite interpreter with GPU options separately, as the YOLO class does not directly handle GPU delegation. For more details, refer to the TensorFlow Lite documentation on GPU support.

[Mrgorlay]

Hello,@pderrenger，@glenn-jocher

"Hi, does YOLOv8 support converting PT (PyTorch) models into apps with the .app extension, or into binary files without an extension?"

[glenn-jocher]

@Mrgorlay hello, YOLOv8 supports exporting models to various formats like ONNX, TensorRT, and CoreML, but not directly to .app or binary files. For more details, please refer to the export documentation.

[yangyang117]

how to use the exported onnx file?
for the segment task, I got an onnx file, and when I put the data in, got a list, list[0] is (1, 40, 8400), list[1] is (1, 32, 160, 160), how to use this data to get the right mask?

[glenn-jocher]

@yangyang117 to use the ONNX output for segmentation, you'll need to process the model's outputs. The first tensor likely contains bounding box predictions, while the second may relate to feature maps for segmentation. You can use a post-processing step to convert these into masks. For detailed guidance, refer to the ONNX Runtime documentation or relevant segmentation tutorials.

[Olofasenius]

Is there any roadmap for fixing the dependency-problems for the export case (onnx2tf seems not to be working with the export-method in ultralytics)

Problem at last line: No module named Imp

########################################
the code run:
########################################
from ultralytics import YOLO

model = YOLO('yolo11n.pt')

model.export(format="tflite",
optimize=True)

########################################
Pip-installation:
########################################
(at 2024-10-23)
pip install tensorflow==2.17.0
pip install tf_keras sng4onnx onnx_graphsurgeon onnx onnx2tf==1.20.0 onnxslim tflite_support onnxruntime

########################################
The output
########################################

Ultralytics 8.3.20 🚀 Python-3.12.7 torch-2.5.0+cpu CPU (12th Gen Intel Core(TM) i7-12700H)
YOLO11n summary (fused): 238 layers, 2,616,248 parameters, 0 gradients, 6.5 GFLOPs

PyTorch: starting from 'yolo11n.pt' with input shape (1, 3, 640, 640) BCHW and output shape(s) (1, 84, 8400) (5.4 MB)

TensorFlow SavedModel: starting export with tensorflow 2.17.0...
WARNING:tensorflow:From C:\Users\olofa\AppData\Local\Programs\Python\Python312\Lib\site-packages\tf_keras\src\losses.py:2976: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

ONNX: starting export with onnx 1.17.0 opset 19...
ONNX: slimming with onnxslim 0.1.35...
ONNX: export success ✅ 0.8s, saved as 'yolo11n.onnx' (10.3 MB)
TensorFlow SavedModel: starting TFLite export with onnx2tf 1.20.0...
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1729672763.807807 25316 devices.cc:76] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0 (Note: TensorFlow was not compiled with CUDA or ROCm support)
W0000 00:00:1729672764.322866 25316 tf_tfl_flatbuffer_helpers.cc:392] Ignored output_format.
W0000 00:00:1729672764.323310 25316 tf_tfl_flatbuffer_helpers.cc:395] Ignored drop_control_dependency.
I0000 00:00:1729672764.871802 25316 devices.cc:76] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0 (Note: TensorFlow was not compiled with CUDA or ROCm support)
W0000 00:00:1729672765.369984 25316 tf_tfl_flatbuffer_helpers.cc:392] Ignored output_format.
W0000 00:00:1729672765.370501 25316 tf_tfl_flatbuffer_helpers.cc:395] Ignored drop_control_dependency.
TensorFlow SavedModel: export failure ❌ 13.0s: No module named 'imp'
Traceback (most recent call last):
File "C:\Users\olofa\AppData\Local\Programs\Python\Python312\Lib\site-packages\ultralytics\engine\exporter.py", line 1019, in _add_tflite_metadata
from tensorflow_lite_support.metadata import metadata_schema_py_generated as schema # noqa
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'tensorflow_lite_support'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\olofa\Downloads\yolov11_tflite\exporting.py", line 6, in
model.export(format="tflite",
File "C:\Users\olofa\AppData\Local\Programs\Python\Python312\Lib\site-packages\ultralytics\engine\model.py", line 734, in export
return Exporter(overrides=args, _callbacks=self.callbacks)(model=self.model)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\olofa\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\utils_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\olofa\AppData\Local\Programs\Python\Python312\Lib\site-packages\ultralytics\engine\exporter.py", line 321, in call
f[5], keras_model = self.export_saved_model()
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\olofa\AppData\Local\Programs\Python\Python312\Lib\site-packages\ultralytics\engine\exporter.py", line 140, in outer_func
raise e
File "C:\Users\olofa\AppData\Local\Programs\Python\Python312\Lib\site-packages\ultralytics\engine\exporter.py", line 135, in outer_func
f, model = inner_func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\olofa\AppData\Local\Programs\Python\Python312\Lib\site-packages\ultralytics\engine\exporter.py", line 903, in export_saved_model
f.unlink() if "quant_with_int16_act.tflite" in str(f) else self._add_tflite_metadata(file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\olofa\AppData\Local\Programs\Python\Python312\Lib\site-packages\ultralytics\engine\exporter.py", line 1022, in add_tflite_metadata
from tflite_support import metadata # noqa
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\olofa\AppData\Local\Programs\Python\Python312\Lib\site-packages\tflite_support\metadata.py", line 28, in
from tflite_support import flatbuffers
File "C:\Users\olofa\AppData\Local\Programs\Python\Python312\Lib\site-packages\tflite_support\flatbuffers_init.py", line 15, in
from .builder import Builder
File "C:\Users\olofa\AppData\Local\Programs\Python\Python312\Lib\site-packages\tflite_support\flatbuffers\builder.py", line 15, in
from . import number_types as N
File "C:\Users\olofa\AppData\Local\Programs\Python\Python312\Lib\site-packages\tflite_support\flatbuffers\number_types.py", line 18, in
from . import packer
File "C:\Users\olofa\AppData\Local\Programs\Python\Python312\Lib\site-packages\tflite_support\flatbuffers\packer.py", line 22, in
from . import compat
File "C:\Users\olofa\AppData\Local\Programs\Python\Python312\Lib\site-packages\tflite_support\flatbuffers\compat.py", line 19, in
import imp
ModuleNotFoundError: No module named 'imp'

[glenn-jocher]

@Olofasenius it seems like the issue is related to missing dependencies. Please ensure that tensorflow_lite_support is installed. For further assistance, you might want to check the compatibility of your Python and package versions. For more details, visit our export documentation.

[Olofasenius]

Thanks @glenn-jocher, I have been trying to install the required packages, but it seems like there are cyclical dependencies.

I can't find the export documentation regarding the reqired python, pip and other package versions.

If it would be possible, is there any pretrained tensorflow lite models of the standard model weights I could use instead of trying to export it myself?

[glenn-jocher]

Unfortunately, we don't provide pre-trained TensorFlow Lite models. For exporting, please ensure all dependencies are correctly installed and compatible with your Python version. For more details, refer to our export documentation.

[cobaltautomationdev]

model.export(format="onnx") only support oonx<=1.16.0, newest 1.17.0 not support

[glenn-jocher]

@cobaltautomationdev thank you for pointing this out. Currently, ONNX export supports up to version 1.16.0. For updates on compatibility with newer versions, please check our documentation or GitHub repository.

[BaoNguyenz]

Why is it that when i format an 'onnx' file to 'int8' and 'dynamic' it work, but when i format the 'onnx' file to 'float16' and 'dynamic' it does't work.
export_path = model.export(format='onnx',
half = True,
dynamic=True
)

[glenn-jocher]

@BaoNguyenz the issue you're experiencing might be due to hardware or software limitations with FP16 precision. Not all platforms support FP16 with dynamic input sizes. Consider checking your hardware compatibility or using INT8 quantization if supported. For more details, visit the export documentation at https://docs.ultralytics.com/modes/export/.

[FishSmallL]

hi, could i order the version of onnx to 13 , 11 or 10. Is it possible todowngrade the onnx version？

[glenn-jocher]

@FishSmallL yes, you can specify the ONNX opset version during export by using the opset argument. For example, use model.export(format="onnx", opset=13) to set the opset version to 13.

[FishSmallL]

Thanks you so much

[glenn-jocher]

Thank you for your appreciation! If you have any further questions or need assistance, feel free to ask.

[Edward-YS]

Hello！
when i run code like this
#########################
from ultralytics import YOLO

Load a model

model = YOLO("./runs/detect/train_v8_bird10/weights/best.pt") # load a custom trained model

Export the model

model.export(format="torchscript", imgsz=320)
#########################
its output is like this
################################
D:\Programming\anaconda\envs\pytorch-1.12.1\python.exe D:\auto_Tracking\yolov8-deepsort-tracking-main_train\runs\pt2othermodel.py
Ultralytics YOLOv8.2.53 🚀 Python-3.9.0 torch-1.12.1+cu116 CPU (AMD Ryzen 9 7945HX with Radeon Graphics)
Model summary (fused): 168 layers, 3005843 parameters, 0 gradients

PyTorch: starting from 'runs\detect\train_v8_bird10\weights\best.pt' with input shape (1, 3, 320, 320) BCHW and output shape(s) (1, 5, 2100) (5.9 MB)

TorchScript: starting export with torch 1.12.1+cu116...
TorchScript: export success ✅ 1.1s, saved as 'runs\detect\train_v8_bird10\weights\best.torchscript' (11.8 MB)
################################
why "output shape(s) (1, 5, 2100) (5.9 MB)" reminds me output'size is 5.9 MB but "best.torchscript' (11.8 MB)"reminds me output'size is 11.8 MB as double?

[glenn-jocher]

@Edward-YS the discrepancy in file sizes is due to the additional metadata and optimizations included in the TorchScript model, which result in a larger file size compared to the raw output tensor size. For more details on model export, you can refer to the Ultralytics documentation.

[Edward-YS]

So, it's right? Originally, i thought it' wrong.

[glenn-jocher]

Yes, the output shape refers to the tensor dimensions, while the TorchScript file size includes additional metadata and optimizations, resulting in a larger file size.

[Ranjith3436]

i am using Tesla T4 GPU but i can't achive 3ms inference time. it takes 7 - 8 ms. for openvino model.

[glenn-jocher]

Inference times can vary due to several factors, including model complexity and hardware configuration. Consider optimizing your model further or adjusting export settings for better performance.

[Ranjith3436]

i am using Tesla T4 GPU but i can't achive 3ms inference time. it takes 7 - 8 ms. for openvino model.

[glenn-jocher]

@Ranjith3436 achieving 3ms inference time on a Tesla T4 with OpenVINO might require further optimization. Consider adjusting model export settings, such as enabling FP16 or INT8 quantization, and ensure your environment is configured to fully utilize the GPU's capabilities.

[Aniket1210]

Hello Team,
I'm using
Ubuntu 22.04.5 LTS
cuda_12.2.r12.2
Cudnn 8.9.7.29
tourch 2.2.2+cu121
OpenCV version: 4.8.1
This all configuration for Traffic Sign Detection and Classification i have train the model and given to the Inferencing , it give proper result with smooth output in Inferencing.
but this same detection and classification 'best_det.pt' & 'best_cls.pt' file to the my ' nvidia jetson agx orin 64gb developer kit ',
It Having Jetpack 6.1 version
I have run the same code in that board but output getting very slow like Inferencing not showing smooth video.
This is the speed when I'm using same code on nvidia jetson agx (Speed: 5.0ms preprocess, 67.5ms inference, 5.0ms postprocess per image at shape (1, 3, 384, 640)
And this speed when Im using in my my GPU system following are configuration
Hardware Model :- ASUSTEK COMPUTER INC. ROG Strix G834JY_G834JY
Memory :- 32.0 GiB
Processor :- 13th Gen Intel® Core™* j9-13980HX × 32
Graphics :- NVIDIA Corporation / Mesa Intel® Graphics (RPL-5) & (NVIDIA GeForce RTX 4090 )
Disk Capacity :- 2.0 TB
OS Name :- Ubuntu 22.04.4 LTS
OS Type :- 64-bit
GNOME Version :- 42.9
Windowing System :- X11
following are the speed when Im using same etection and classification 'best_det.pt' & 'best_cls.pt' file ROG Strix GPU system
Speed: 1.6ms preprocess, 17.8ms inference, 0.5ms postprocess per image at shape (1, 3, 384, 640)

So could you let me know why this happen, what is the reason my code not running smooth with speed up on 'nvidia jetson agx orin 64gb developer kit'

[Aniket1210]

Hello @glenn-jocher Can you help me for this query ?

[glenn-jocher]

@Aniket1210 the performance difference on your NVIDIA Jetson AGX Orin compared to your desktop system is likely due to hardware differences. The Jetson has a different architecture optimized for power efficiency rather than raw speed. To improve inference speed, consider exporting your model to TensorRT format, enabling optimizations like INT8 quantization, and reducing input image size if possible. These actions can significantly boost performance on NVIDIA hardware. For more details, you might explore the Export documentation for guidance on optimizing your model for deployment.

[Aniket1210]

@glenn-jocher Thanks for this guide.
I export this .pt file using TensorRT formats with same version TensorRT (10.3.0) In both system jetson board also my GPU system, Even python version also same in both GPU system also and Jetson board also.

But still I'm getting error like TensorRT model exported with a different version than 10.3.0

python3 main.py --show
WARNING ⚠️ Unable to automatically guess model task, assuming 'task=detect'. Explicitly define task for your model, i.e. 'task=detect', 'segment', 'classify','pose' or 'obb'.
Loading best_det.engine for TensorRT inference...
[11/19/2024-15:53:59] [TRT] [I] Loaded engine size: 46 MiB
[11/19/2024-15:53:59] [TRT] [E] IRuntime::deserializeCudaEngine: Error Code 1: Serialization (Serialization assertion plan->header.pad == expectedPlatformTag failed.Platform specific tag mismatch detected. TensorRT plan files are only supported on the target runtime platform they were created on.)
ERROR: TensorRT model exported with a different version than 10.3.0

Traceback (most recent call last):
File "/home/finv/tsr_inference/main.py", line 50, in
output = det.classify(frame,det.model_det)
File "/home/finv/tsr_inference/object_detection.py", line 93, in classify
df = self.__forward(frame,model1)
File "/home/finv/tsr_inference/object_detection.py", line 46, in __forward
results = model.predict(source=frame,verbose=True,imgsz=640,conf = 0.60, half=True)
File "/home/finv/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/model.py", line 547, in predict
self.predictor.setup_model(model=self.model, verbose=is_cli)
File "/home/finv/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/predictor.py", line 304, in setup_model
self.model = AutoBackend(
File "/home/finv/tsr/tsrenv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/finv/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/nn/autobackend.py", line 291, in init
raise e
File "/home/finv/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/nn/autobackend.py", line 288, in init
context = model.create_execution_context()
AttributeError: 'NoneType' object has no attribute 'create_execution_context'

could you tell me which reason I'm getting this error ?

[glenn-jocher]

The error you're encountering suggests a mismatch in the TensorRT serialization between the systems. TensorRT engine files are not portable across different versions or hardware configurations. You'll need to export the model directly on the Jetson AGX Orin with the same TensorRT version installed to ensure compatibility. For more details on exporting to TensorRT, refer to the TensorRT integration guide.