Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Mobile] onnxruntime-react-native app crashes during session.run #17623

Closed
seuklee1025 opened this issue Sep 20, 2023 · 12 comments
Closed

[Mobile] onnxruntime-react-native app crashes during session.run #17623

seuklee1025 opened this issue Sep 20, 2023 · 12 comments
Labels
api:Javascript issues related to the Javascript API platform:mobile issues related to ONNX Runtime mobile; typically submitted using template

Comments

@seuklee1025
Copy link

seuklee1025 commented Sep 20, 2023

Describe the issue

I am using onnxruntime-react-native. There is an issue where the app crashes during session.run.

This happens consistently with any ORT model and across various versions.

We created a blank array to match the shape of the input and challenged many of the models released in the port file.

The problem is that the app turns off as soon as you infer without errors

I want to know the solution.

To reproduce

all ort inference code

Urgency

No response

Platform

Android

OS Version

1.16

ONNX Runtime Installation

Built from Source

Compiler Version (if 'Built from Source')

No response

Package Name (if 'Released Package')

None

ONNX Runtime Version or Commit ID

1.16

ONNX Runtime API

JavaScript

Architecture

ARM64

Execution Provider

Default CPU

Execution Provider Library Version

No response

@seuklee1025 seuklee1025 added the platform:mobile issues related to ONNX Runtime mobile; typically submitted using template label Sep 20, 2023
@github-actions github-actions bot added the api:Javascript issues related to the Javascript API label Sep 20, 2023
@seuklee1025
Copy link
Author

import { Asset } from "expo-asset";
import React, { useEffect } from "react";
import * as ort from 'onnxruntime-react-native'
import { Alert, View, Text } from "react-native";

const App = () => {

  useEffect(() => {
    initLoadModel();
  }, [])

  const initLoadModel = async () => {
    try {
      const assets = await Asset.loadAsync(require('../assets/models/mnist.ort'));
      const modelUri = assets[0].localUri;
      if (!modelUri) {
        Alert.alert('failed to get model URI', `${assets[0]}`);
      } else {
        const myModel = await ort.InferenceSession.create(modelUri);
        Alert.alert(
          'model loaded successfully',
          `input names: ${myModel.inputNames}, output names: ${myModel.outputNames}`);


        const inputData = new Float32Array(28 * 28);
        const feeds: Record<string, ort.Tensor> = {};
        feeds[myModel.inputNames[0]] = new ort.Tensor(inputData, [1, 28, 28]);


        console.log("feeds :: ", feeds)


        const fetches = await myModel.run(feeds,);

        console.log("run 후에")
        const output = fetches[myModel.outputNames[0]];
        console.log(output)
        if (!output) {
          Alert.alert('failed to get output', `${myModel.outputNames[0]}`);
        } else {
          Alert.alert(
            'model inference successfully',
            `output shape: ${output.dims}, output data: ${output.data}`);
        }
      }
    } catch (e) {
      Alert.alert('failed to load model', `${e}`);
      throw e;
    }

  }
  return (
    <View>
      <Text>테스트 진행중입니다!</Text>
    </View>
  )
}
export default App;

@seuklee1025
Copy link
Author

seuklee1025 commented Sep 20, 2023

The code died in run and was tested with a poco phone.

const fetches = await myModel.run(feeds,);

@YUNQIUGUO
Copy link
Contributor

YUNQIUGUO commented Sep 20, 2023

Just to confirm, you are using onnxruntime-react-native 1.16.0 (released couple hours ago)?

Also, does the crash only happen on real device or does it also happen in simulator as well?

@seuklee1025
Copy link
Author

seuklee1025 commented Sep 20, 2023

@YUNQIUGUO

The simulator did not conduct the test, but proceeded with the pocophone.

1.14, 1.15, and 1.16, which came out about five hours ago, all proceeded.

@skottmckay
Copy link
Contributor

What shape input does the model require? Most often I've seen mnist take a 4D input with shape {1, 1, 28, 28} and run will fail if there's a shape mismatch. I would have expected an error message to be available somewhere though.

@seuklee1025
Copy link
Author

@skottmckay

Although we did not check what the exact input of the mnist was, we found that other models (1, 3, 224, 224) that knew the exact input were also interrupted by session.run.

We also found that if the input is different, the model outputs an input shape related error.

@seuklee1025
Copy link
Author

seuklee1025 commented Sep 20, 2023

If the input is different, the app does not die and only causes the error that the input is different.
In other words, the app dies only when inference is performed in session.run

@seuklee1025
Copy link
Author

seuklee1025 commented Sep 20, 2023

I'm uploading the library in case it's a library problem

{
  "name": "testtest",
  "version": "0.0.1",
  "private": true,
  "scripts": {
    "android": "react-native run-android",
    "ios": "react-native run-ios",
    "lint": "eslint .",
    "start": "react-native start",
    "test": "jest"
  },
  "dependencies": {
    "@tensorflow/tfjs": "3.7.0",
    "@tensorflow/tfjs-backend-webgl": "3.7.0",
    "@tensorflow/tfjs-converter": "3.7.0",
    "@tensorflow/tfjs-core": "3.7.0",
    "@tensorflow/tfjs-react-native": "0.8.0",
    "expo": "^49.0.0",
    "expo-asset": "^8.10.1",
    "install-expo-modules": "^0.6.3",
    "onnxruntime-react-native": "1.16.0", 
    "react": "18.2.0",
    "react-native": "0.72.4"
  },
  "devDependencies": {
    "@babel/core": "^7.20.0",
    "@babel/preset-env": "^7.20.0",
    "@babel/runtime": "^7.20.0",
    "@react-native/eslint-config": "^0.72.2",
    "@react-native/metro-config": "^0.72.11",
    "@tsconfig/react-native": "^3.0.0",
    "@types/react": "^18.0.24",
    "@types/react-test-renderer": "^18.0.0",
    "babel-jest": "^29.2.1",
    "eslint": "^8.19.0",
    "jest": "^29.2.1",
    "metro-react-native-babel-preset": "0.76.8",
    "prettier": "^2.4.1",
    "react-test-renderer": "18.2.0",
    "typescript": "4.8.4"
  },
  "engines": {
    "node": ">=16"
  },
  "main": "index.js",
  "license": "MIT"
}

@bartproo
Copy link

bartproo commented Sep 21, 2023

I have the same issue as mentioned here #17541. Input shape is not a problem. I think there is a bug with the native code as the crash is from C.

@saeu5407
Copy link

I hope the problem is solved quickly!

@skottmckay
Copy link
Contributor

Can you try using the XNNPACK execution provider to see if that avoids the issue? Requires onnxruntime 1.16.

Details on how to enable that in this comment: #17541 (comment)

@skottmckay
Copy link
Contributor

Closing. Issue is covered in more detail in #17541 and #17647

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api:Javascript issues related to the Javascript API platform:mobile issues related to ONNX Runtime mobile; typically submitted using template
Projects
None yet
Development

No branches or pull requests

5 participants