"index out of bound exception" while reading the width and height from the input tensor

[ShubhamSureWash]

Hi, I am trying to integrate the custom quantized Mobilenet SSD v2 model to unity. I am getting an "index out of bound exception" while reading the width and height from the input tensor using the following command.
int height = CocoSsdMobilenet._inputTensor.Dims[1];

I checked my model and it appears to be fine. Below is the screenshot of it.

if I am commenting these lines and initializing height and width as 300, on clicking on the play button unity engine is crashing with giving any exception. Please suggest.

Our quantized TFLite model works well in the sample project provided in Android Studio by TensorFlow but its giving exception here.

[emgucv]

Can you provide the model and test code to reproduce the issue?

As per following ticket:
tensorflow/tensorflow#22532
tflite on Windows do not implement quantization and is not able to load quantized model. I believe Quantized model is only supported for macOS, Linux, Android and iOS. If you are using TF Lite with Unity on Windows editor you may not be able to load a quantized model. Try to develop on with Unity on MacOS instead.

[ShubhamSureWash]

Hi team, Below are the attached sample program used for running tflite model. When I am trying with float model its working fine. I tried the quantised model in mac but then also the same error message. detectq.tflite is my quantised model. Thanks, Shubham Pandey Vision Engineer Surewash Ltd. FLoat_Model_Program_Working.zip <https://drive.google.com/a/surewash.com/file/d/1cntYYjxb4JD1P2ItW48gfc_UrpZK09aY/view?usp=drive_web> Quantised_Model_Not_Working.zip <https://drive.google.com/a/surewash.com/file/d/15qD9li7Ed6Us6PFt_lmdJ6kq47WdRZrI/view?usp=drive_web>

…

On Thu, 6 Feb 2020 at 21:22, Emgu CV ***@***.***> wrote: Can you provide the model and test code to reproduce the issue? As per following ticket: tensorflow/tensorflow#22532 <tensorflow/tensorflow#22532> tflite on Windows do not implement quantization and is not able to load quantized model. I believe Quantized model is only supported for macOS, Linux, Android and iOS. If you are using TF Lite with Unity on Windows editor you may not be able to load a quantized model. Try to develop on with Unity on MacOS instead. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#30?email_source=notifications&email_token=ANUHGW3KU3HJXVE7TFEIBATRBR5THA5CNFSM4KQOXPHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELA2R4Y#issuecomment-583117043>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANUHGW4BLGSRGOLDYYBC72DRBR5THANCNFSM4KQOXPHA> .

[emgucv]

@ctheodorak Can you be more specific?

This is similar to asking: I used the library and other components to build a product, but my product doesn't work. Why is it happening? ... we would not be able to provide a meaning answer.

If you find a bug in a function, please point out the specific name of the function, provide the given input, what the function is returning right now and the expected output. We will be able to take a look.

If you have a working tflite code in python, along with the model and demo images, we can provide consulting service to convert python code to C# for a fee.

[ctheodorak]

The model i gave you https://github.com/ctheodorak/DemoTflite/tree/master/samples/New%20folder
it works fine (it is a float model) with this code on pc

I get nice results, but if i build an apk, on android it detects nothing
Is it a problem that is a float model and not a quantized one??

[emgucv]

@ShubhamSureWash
Replying to your original post. I have downloaded the following model used in Google's Android demo:
http://storage.googleapis.com/download.tensorflow.org/models/tflite/coco_ssd_mobilenet_v1_1.0_quant_2018_06_29.zip
Read the dimension of the input and output tensors using the following code:

        private static String IntArrayToString(int[] values)
        {
            return String.Format("[{0}]", String.Join(",", values));
        }

        private static String GetModelInfo(String fileName)
        {
            try
            {
                StringBuilder modelResult = new StringBuilder();
                modelResult.Append(String.Format("File Name:{0}{1}", fileName, Environment.NewLine));
                //string contents = System.Text.Encoding.UTF8.GetString(fileData.DataArray);
                using (FlatBufferModel fbm = new FlatBufferModel(fileName))
                {
                    if (!fbm.CheckModelIdentifier())
                        throw new Exception("Model identifier check failed");

                    using (Interpreter interpreter = new Interpreter(fbm))
                    {
                        Status allocateTensorStatus = interpreter.AllocateTensors();
                        if (allocateTensorStatus == Status.Error)
                            throw new Exception("Failed to allocate tensor");
                        int[] input = interpreter.InputIndices;
                        for (int i = 0; i < input.Length; i++)
                        {
                            Tensor inputTensor = interpreter.GetTensor(input[i]);

                            modelResult.Append(String.Format("Input {0} ({1}): type-{2}; dim-{3}{4}", i, inputTensor.Name,
                                inputTensor.Type, IntArrayToString( inputTensor.Dims ), Environment.NewLine));
                        }

                        int[] output = interpreter.OutputIndices;
                        for (int i = 0; i < output.Length; i++)
                        {
                            Tensor outputTensor = interpreter.GetTensor(output[i]);

                            modelResult.Append(String.Format("Output {0} ({1}): type-{2}; dim-{3}{4}", i, outputTensor.Name,
                                outputTensor.Type, IntArrayToString( outputTensor.Dims ), Environment.NewLine));
                        }

                    }
                }

                return modelResult.ToString();
            }
            catch (Exception ex)
            {
                return String.Format("Exception processing file {0}: {1} ", fileName, ex.ToString());

            }
        }

and got the output:

File Name:detect.tflite
Input 0 (normalized_input_image_tensor): type-UInt8; dim-[1,300,300,3]
Output 0 (TFLite_Detection_PostProcess): type-Float32; dim-[1,10,4]
Output 1 (TFLite_Detection_PostProcess:1): type-Float32; dim-[1,10]
Output 2 (TFLite_Detection_PostProcess:2): type-Float32; dim-[1,10]
Output 3 (TFLite_Detection_PostProcess:3): type-Float32; dim-[1]

I don't have the "index out of bound exception" reported in your original post.

[ShubhamSureWash]

It works fine with model provided with google sample but when I use it with my custom model it gives exception stated above. My model works fine in Sample project provided by TensorFlow in Android and iOS but it fails when tried to integrate with EMGU. In the above comment I have attached the sample code which I am using and also the my model. Emgu is working fine with my float model but its failing with my quantised model.

Dose EMGU Unity plugin 1.13 supports TensorFlow 1.14 ? I have trained my model in TensorFlow 1.14.

[ctheodorak]

@ShubhamSureWash
I had the same problem

I took only an array with one element when integrate mobilenet_v2_quantized_coco instead of [1,height,widht,3]
You wont have the same problem with mobilenet_v1_quantized, i trained one and i get the results i want

[ctheodorak]

@emgucv

How i can make my scene run faster?
When i open devices's came the frames are very low cause of model's result in update function
How can i increase the fps?

[emgucv]

@ShubhamSureWash Can you use the code I pasted above on your custom model? Paste the output here so we can see. I suspect your model's input tensor's only have a single dimension. Accessing the second or 3rd dimension will results in exception.

[emgucv]

@ctheodorak Profile your code, check how much time it is spent on preparing data for tflite and how much time is spend on inference. If inference cost the most time. Look up on tensorflow or tflite on how to speed up your model.

[ctheodorak]

I train the mobilenet_v2 quantized and i used your code

I took this as an output
"File Name:C:/Users/user/AppData/LocalLow/DefaultCompany/Detector\detect.tflite\r\nInput 0 (BoxPredictor_0/ClassPredictor/Conv2D_bias): type-Int32; dim-[15]\r\nOutput 0 (TFLite_Detection_PostProcess): type-Float32; dim-[1,10,4]\r\nOutput 1 (TFLite_Detection_PostProcess:1): type-Float32; dim-[1,10]\r\nOutput 2 (TFLite_Detection_PostProcess:2): type-Float32; dim-[1,10]\r\nOutput 3 (TFLite_Detection_PostProcess:3): type-Float32; dim-[1]\r\n"

and not
File Name:detect.tflite
Input 0 (normalized_input_image_tensor): type-UInt8; dim-[1,600,600,3]
Output 0 (TFLite_Detection_PostProcess): type-Float32; dim-[1,10,4]
Output 1 (TFLite_Detection_PostProcess:1): type-Float32; dim-[1,10]
Output 2 (TFLite_Detection_PostProcess:2): type-Float32; dim-[1,10]
Output 3 (TFLite_Detection_PostProcess:3): type-Float32; dim-[1]
which was supposed
BUT look at this python script
`import os
import argparse
import cv2
import numpy as np
import sys
import glob
import importlib.util

parser = argparse.ArgumentParser()
parser.add_argument('--modeldir', help='Folder the .tflite file is located in',
required=True)
parser.add_argument('--graph', help='Name of the .tflite file, if different than detect.tflite',
default='detect.tflite')
parser.add_argument('--labels', help='Name of the labelmap file, if different than labelmap.txt',
default='labelmap.txt')
parser.add_argument('--threshold', help='Minimum confidence threshold for displaying detected objects',
default=0.5)
parser.add_argument('--image', help='Name of the single image to perform detection on. To run detection on multiple images, use --imagedir',
default=None)
parser.add_argument('--imagedir', help='Name of the folder containing images to perform detection on. Folder must contain only images.',
default=None)
parser.add_argument('--edgetpu', help='Use Coral Edge TPU Accelerator to speed up detection',
action='store_true')

args = parser.parse_args()

MODEL_NAME = args.modeldir
GRAPH_NAME = args.graph
LABELMAP_NAME = args.labels
min_conf_threshold = float(args.threshold)
use_TPU = args.edgetpu

IM_NAME = args.image
IM_DIR = args.imagedir

if (IM_NAME and IM_DIR):
print('Error! Please only use the --image argument or the --imagedir argument, not both. Issue "python TFLite_detection_image.py -h" for help.')
sys.exit()

if (not IM_NAME and not IM_DIR):
IM_NAME = 'test1.jpg'

pkg = importlib.util.find_spec('tensorflow')
if pkg is None:
from tflite_runtime.interpreter import Interpreter
if use_TPU:
from tflite_runtime.interpreter import load_delegate
else:
from tensorflow.lite.python.interpreter import Interpreter
if use_TPU:
from tensorflow.lite.python.interpreter import load_delegate

if use_TPU:
//If user has specified the name of the .tflite file, use that name, otherwise use default 'edgetpu.tflite'
if (GRAPH_NAME == 'detect.tflite'):
GRAPH_NAME = 'edgetpu.tflite'

CWD_PATH = os.getcwd()

if IM_DIR:
PATH_TO_IMAGES = os.path.join(CWD_PATH,IM_DIR)
images = glob.glob(PATH_TO_IMAGES + '/*')

elif IM_NAME:
PATH_TO_IMAGES = os.path.join(CWD_PATH,IM_NAME)
images = glob.glob(PATH_TO_IMAGES)

PATH_TO_CKPT = os.path.join(CWD_PATH,MODEL_NAME,GRAPH_NAME)

PATH_TO_LABELS = os.path.join(CWD_PATH,MODEL_NAME,LABELMAP_NAME)

with open(PATH_TO_LABELS, 'r') as f:
labels = [line.strip() for line in f.readlines()]

if labels[0] == '???':
del(labels[0])

if use_TPU:
interpreter = Interpreter(model_path=PATH_TO_CKPT,
experimental_delegates=[load_delegate('libedgetpu.so.1.0')])
print(PATH_TO_CKPT)
else:
interpreter = Interpreter(model_path=PATH_TO_CKPT)

interpreter.allocate_tensors()

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
height = input_details[0]['shape'][1]
width = input_details[0]['shape'][2]

floating_model = (input_details[0]['dtype'] == np.float32)

input_mean = 127.5
input_std = 127.5

for image_path in images:

//Load image and resize to expected shape [1xHxWx3]
image = cv2.imread(image_path)
image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
imH, imW, _ = image.shape 
image_resized = cv2.resize(image_rgb, (width, height))
input_data = np.expand_dims(image_resized, axis=0)

if floating_model:
    input_data = (np.float32(input_data) - input_mean) / input_std
print(input_details)

interpreter.set_tensor(input_details[0]['index'],input_data)
interpreter.invoke()

boxes = interpreter.get_tensor(output_details[0]['index'])[0] # Bounding box coordinates of detected objects
classes = interpreter.get_tensor(output_details[1]['index'])[0] 
scores = interpreter.get_tensor(output_details[2]['index'])[0] 
//num = interpreter.get_tensor(output_details[3]['index'])[0]  

for i in range(len(scores)):
    if ((scores[i] > min_conf_threshold) and (scores[i] <= 1.0)):

them to be within image using max() and min()
ymin = int(max(1,(boxes[i][0] * imH)))
xmin = int(max(1,(boxes[i][1] * imW)))
ymax = int(min(imH,(boxes[i][2] * imH)))
xmax = int(min(imW,(boxes[i][3] * imW)))

        cv2.rectangle(image, (xmin,ymin), (xmax,ymax), (10, 255, 0), 2)

        object_name = labels[int(classes[i])] 
        label = '%s: %d%%' % (object_name, int(scores[i]*100)) 
        labelSize, baseLine = cv2.getTextSize(label, cv2.FONT_HERSHEY_SIMPLEX, 0.7, 2) 
        label_ymin = max(ymin, labelSize[1] + 10) 
        cv2.rectangle(image, (xmin, label_ymin-labelSize[1]-10), (xmin+labelSize[0], label_ymin+baseLine-10), (255, 255, 255), cv2.FILLED) 
        cv2.putText(image, label, (xmin, label_ymin-7), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 0, 0), 2)

cv2.imshow('Object detector', image)

if cv2.waitKey(0) == ord('q'):
    break

cv2.destroyAllWindows()`
It gives this output

My model :https://github.com/ctheodorak/DemoTflite/tree/master/MobilenetV2_qunatized
And also it works fine with the python script why i get the same exception in your plugin.