🐛 App crash when open camera on Android when using useFrameProcessor

[diegomeire]

What's happening?

When building for Android, whenever I use the following with frameProcessor={frameProcessor} the app crashes right at the start. If I remove frameProcessor it works normally.

Here's my dependencies on package.json:

  "dependencies": {
    "expo": "~51.0.28",
    "expo-status-bar": "~1.12.1",
    "metro": "^0.80.12",
    "react": "18.2.0",
    "react-native": "0.74.5",
    "react-native-asset": "^2.1.1",
    "react-native-fast-tflite": "^1.4.0",
    "react-native-fs": "^2.20.0",
    "react-native-vision-camera": "^4.5.3",
    "react-native-worklets-core": "^1.3.3",
    "vision-camera-resize-plugin": "^3.1.0"
  },
  "devDependencies": {
    "@babel/core": "^7.20.0",
    "@types/react": "~18.2.45",
    "typescript": "^5.1.3"
  },

Reproduceable Code

/* eslint-disable @typescript-eslint/no-var-requires */
import * as React from 'react'

import { StyleSheet, View, Text, ActivityIndicator } from 'react-native'
import {
  Tensor,
  TensorflowModel,
  useTensorflowModel,
} from 'react-native-fast-tflite'
import {
  Camera,
  useCameraDevice,
  useCameraPermission,
  useFrameProcessor,
} from 'react-native-vision-camera'
import { useResizePlugin } from 'vision-camera-resize-plugin'

function tensorToString(tensor: Tensor): string {
  return `\n  - ${tensor.dataType} ${tensor.name}[${tensor.shape}]`
}
function modelToString(model: TensorflowModel): string {
  return (
    `TFLite Model (${model.delegate}):\n` +
    `- Inputs: ${model.inputs.map(tensorToString).join('')}\n` +
    `- Outputs: ${model.outputs.map(tensorToString).join('')}`
  )
}

export default function App(): React.ReactNode {
  const { hasPermission, requestPermission } = useCameraPermission()
  const device = useCameraDevice('back')

  // from https://www.kaggle.com/models/tensorflow/efficientdet/frameworks/tfLite
  const model = useTensorflowModel(require('./assets/lush_naked.tflite'), 'nnapi')
  const actualModel = model.state === 'loaded' ? model.model : undefined

  React.useEffect(() => {
    if (actualModel == null) return
    console.log(`Model loaded! Shape:\n${modelToString(actualModel)}]`)
  }, [actualModel])

  const { resize } = useResizePlugin()

  const frameProcessor = useFrameProcessor((frame) => {
    'worklet'
  }, [])
  React.useEffect(() => {
    requestPermission()
  }, [requestPermission])

  console.log(`Model: ${model.state} (${model.model != null})`)

  return (
    <View style={styles.container}>
      {hasPermission && device != null ? (
        <Camera
          device={device}
          style={StyleSheet.absoluteFill}
          isActive={true}
          frameProcessor={frameProcessor}
          pixelFormat="yuv"
        />
      ) : (
        <Text>No Camera available.</Text>
      )}

      {model.state === 'loading' && (
        <ActivityIndicator size="small" color="white" />
      )}

      {model.state === 'error' && (
        <Text>Failed to load model! {model.error.message}</Text>
      )}
    </View>
  )
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
    alignItems: 'center',
    justifyContent: 'center',
  },
})

Relevant log output

2024-10-06 18:18:26.272 30315-30469 ImageAnalysisAnalyzer   com...meire.lushlensobjectdetection  E  Failed to acquire image.
                                                                                                    java.lang.IllegalStateException: maxImages (6) has already been acquired, call #close before acquiring more.
                                                                                                    	at android.media.ImageReader.acquireNextImage(ImageReader.java:560)
                                                                                                    	at androidx.camera.core.AndroidImageReaderProxy.acquireNextImage(AndroidImageReaderProxy.java:86)
                                                                                                    	at androidx.camera.core.SafeCloseImageReaderProxy.acquireNextImage(SafeCloseImageReaderProxy.java:86)
                                                                                                    	at androidx.camera.core.ImageAnalysisBlockingAnalyzer.acquireImage(ImageAnalysisBlockingAnalyzer.java:39)
                                                                                                    	at androidx.camera.core.ImageAnalysisAbstractAnalyzer.onImageAvailable(ImageAnalysisAbstractAnalyzer.java:126)
                                                                                                    	at androidx.camera.core.SafeCloseImageReaderProxy.lambda$setOnImageAvailableListener$1$androidx-camera-core-SafeCloseImageReaderProxy(SafeCloseImageReaderProxy.java:211)
                                                                                                    	at androidx.camera.core.SafeCloseImageReaderProxy$$ExternalSyntheticLambda1.onImageAvailable(Unknown Source:4)
                                                                                                    	at androidx.camera.core.AndroidImageReaderProxy.lambda$setOnImageAvailableListener$0$androidx-camera-core-AndroidImageReaderProxy(AndroidImageReaderProxy.java:167)
                                                                                                    	at androidx.camera.core.AndroidImageReaderProxy$$ExternalSyntheticLambda0.run(Unknown Source:4)
                                                                                                    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                                                                                                    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
                                                                                                    	at java.lang.Thread.run(Thread.java:1012)
2024-10-06 18:18:26.284 30315-30469 ImageReader_JNI         com...meire.lushlensobjectdetection  W  Unable to acquire a buffer item, very likely client tried to acquire more than maxImages buffers
2024-10-06 18:18:26.286 30315-30469 ImageAnalysisAnalyzer   com...meire.lushlensobjectdetection  E  Failed to acquire image.
                                                                                                    java.lang.IllegalStateException: maxImages (6) has already been acquired, call #close before acquiring more.
                                                                                                    	at android.media.ImageReader.acquireNextImage(ImageReader.java:560)
                                                                                                    	at androidx.camera.core.AndroidImageReaderProxy.acquireNextImage(AndroidImageReaderProxy.java:86)
                                                                                                    	at androidx.camera.core.SafeCloseImageReaderProxy.acquireNextImage(SafeCloseImageReaderProxy.java:86)
                                                                                                    	at androidx.camera.core.ImageAnalysisBlockingAnalyzer.acquireImage(ImageAnalysisBlockingAnalyzer.java:39)
                                                                                                    	at androidx.camera.core.ImageAnalysisAbstractAnalyzer.onImageAvailable(ImageAnalysisAbstractAnalyzer.java:126)
                                                                                                    	at androidx.camera.core.SafeCloseImageReaderProxy.lambda$setOnImageAvailableListener$1$androidx-camera-core-SafeCloseImageReaderProxy(SafeCloseImageReaderProxy.java:211)
                                                                                                    	at androidx.camera.core.SafeCloseImageReaderProxy$$ExternalSyntheticLambda1.onImageAvailable(Unknown Source:4)
                                                                                                    	at androidx.camera.core.AndroidImageReaderProxy.lambda$setOnImageAvailableListener$0$androidx-camera-core-AndroidImageReaderProxy(AndroidImageReaderProxy.java:167)
                                                                                                    	at androidx.camera.core.AndroidImageReaderProxy$$ExternalSyntheticLambda0.run(Unknown Source:4)
                                                                                                    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                                                                                                    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
                                                                                                    	at java.lang.Thread.run(Thread.java:1012)
2024-10-06 18:18:26.296 30315-30361 HermesVM                com...meire.lushlensobjectdetection  I  Compiling JS failed: 1:1:invalid empty parentheses '( )',  Buffer size 3 starts with: 280a29 and has protection mode(s): rw-p
2024-10-06 18:18:26.309 30315-30469 ImageReader_JNI         com...meire.lushlensobjectdetection  W  Unable to acquire a buffer item, very likely client tried to acquire more than maxImages buffers
2024-10-06 18:18:26.309 30315-30469 ImageAnalysisAnalyzer   com...meire.lushlensobjectdetection  E  Failed to acquire image.
                                                                                                    java.lang.IllegalStateException: maxImages (6) has already been acquired, call #close before acquiring more.
                                                                                                    	at android.media.ImageReader.acquireNextImage(ImageReader.java:560)
                                                                                                    	at androidx.camera.core.AndroidImageReaderProxy.acquireNextImage(AndroidImageReaderProxy.java:86)
                                                                                                    	at androidx.camera.core.SafeCloseImageReaderProxy.acquireNextImage(SafeCloseImageReaderProxy.java:86)
                                                                                                    	at androidx.camera.core.ImageAnalysisBlockingAnalyzer.acquireImage(ImageAnalysisBlockingAnalyzer.java:39)
                                                                                                    	at androidx.camera.core.ImageAnalysisAbstractAnalyzer.onImageAvailable(ImageAnalysisAbstractAnalyzer.java:126)
                                                                                                    	at androidx.camera.core.SafeCloseImageReaderProxy.lambda$setOnImageAvailableListener$1$androidx-camera-core-SafeCloseImageReaderProxy(SafeCloseImageReaderProxy.java:211)
                                                                                                    	at androidx.camera.core.SafeCloseImageReaderProxy$$ExternalSyntheticLambda1.onImageAvailable(Unknown Source:4)
                                                                                                    	at androidx.camera.core.AndroidImageReaderProxy.lambda$setOnImageAvailableListener$0$androidx-camera-core-AndroidImageReaderProxy(AndroidImageReaderProxy.java:167)
                                                                                                    	at androidx.camera.core.AndroidImageReaderProxy$$ExternalSyntheticLambda0.run(Unknown Source:4)
                                                                                                    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                                                                                                    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
                                                                                                    	at java.lang.Thread.run(Thread.java:1012)
2024-10-06 18:18:26.336 30315-30469 ImageReader_JNI         com...meire.lushlensobjectdetection  W  Unable to acquire a buffer item, very likely client tried to acquire more than maxImages buffers
2024-10-06 18:18:26.339 30315-30469 ImageAnalysisAnalyzer   com...meire.lushlensobjectdetection  E  Failed to acquire image.
                                                                                                    java.lang.IllegalStateException: maxImages (6) has already been acquired, call #close before acquiring more.
                                                                                                    	at android.media.ImageReader.acquireNextImage(ImageReader.java:560)
                                                                                                    	at androidx.camera.core.AndroidImageReaderProxy.acquireNextImage(AndroidImageReaderProxy.java:86)
                                                                                                    	at androidx.camera.core.SafeCloseImageReaderProxy.acquireNextImage(SafeCloseImageReaderProxy.java:86)
                                                                                                    	at androidx.camera.core.ImageAnalysisBlockingAnalyzer.acquireImage(ImageAnalysisBlockingAnalyzer.java:39)
                                                                                                    	at androidx.camera.core.ImageAnalysisAbstractAnalyzer.onImageAvailable(ImageAnalysisAbstractAnalyzer.java:126)
                                                                                                    	at androidx.camera.core.SafeCloseImageReaderProxy.lambda$setOnImageAvailableListener$1$androidx-camera-core-SafeCloseImageReaderProxy(SafeCloseImageReaderProxy.java:211)
                                                                                                    	at androidx.camera.core.SafeCloseImageReaderProxy$$ExternalSyntheticLambda1.onImageAvailable(Unknown Source:4)
                                                                                                    	at androidx.camera.core.AndroidImageReaderProxy.lambda$setOnImageAvailableListener$0$androidx-camera-core-AndroidImageReaderProxy(AndroidImageReaderProxy.java:167)
                                                                                                    	at androidx.camera.core.AndroidImageReaderProxy$$ExternalSyntheticLambda0.run(Unknown Source:4)
                                                                                                    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                                                                                                    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
                                                                                                    	at java.lang.Thread.run(Thread.java:1012)
2024-10-06 18:18:26.370 30315-30469 ImageReader_JNI         com...meire.lushlensobjectdetection  W  Unable to acquire a buffer item, very likely client tried to acquire more than maxImages buffers
2024-10-06 18:18:26.371 30315-30469 ImageAnalysisAnalyzer   com...meire.lushlensobjectdetection  E  Failed to acquire image.
                                                                                                    java.lang.IllegalStateException: maxImages (6) has already been acquired, call #close before acquiring more.
                                                                                                    	at android.media.ImageReader.acquireNextImage(ImageReader.java:560)
                                                                                                    	at androidx.camera.core.AndroidImageReaderProxy.acquireNextImage(AndroidImageReaderProxy.java:86)
                                                                                                    	at androidx.camera.core.SafeCloseImageReaderProxy.acquireNextImage(SafeCloseImageReaderProxy.java:86)
                                                                                                    	at androidx.camera.core.ImageAnalysisBlockingAnalyzer.acquireImage(ImageAnalysisBlockingAnalyzer.java:39)
                                                                                                    	at androidx.camera.core.ImageAnalysisAbstractAnalyzer.onImageAvailable(ImageAnalysisAbstractAnalyzer.java:126)
                                                                                                    	at androidx.camera.core.SafeCloseImageReaderProxy.lambda$setOnImageAvailableListener$1$androidx-camera-core-SafeCloseImageReaderProxy(SafeCloseImageReaderProxy.java:211)
                                                                                                    	at androidx.camera.core.SafeCloseImageReaderProxy$$ExternalSyntheticLambda1.onImageAvailable(Unknown Source:4)
                                                                                                    	at androidx.camera.core.AndroidImageReaderProxy.lambda$setOnImageAvailableListener$0$androidx-camera-core-AndroidImageReaderProxy(AndroidImageReaderProxy.java:167)
                                                                                                    	at androidx.camera.core.AndroidImageReaderProxy$$ExternalSyntheticLambda0.run(Unknown Source:4)
                                                                                                    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                                                                                                    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
                                                                                                    	at java.lang.Thread.run(Thread.java:1012)
2024-10-06 18:18:26.401 30315-30469 ImageReader_JNI         com...meire.lushlensobjectdetection  W  Unable to acquire a buffer item, very likely client tried to acquire more than maxImages buffers
2024-10-06 18:18:26.403 30315-30469 ImageAnalysisAnalyzer   com...meire.lushlensobjectdetection  E  Failed to acquire image.
                                                                                                    java.lang.IllegalStateException: maxImages (6) has already been acquired, call #close before acquiring more.
                                                                                                    	at android.media.ImageReader.acquireNextImage(ImageReader.java:560)
                                                                                                    	at androidx.camera.core.AndroidImageReaderProxy.acquireNextImage(AndroidImageReaderProxy.java:86)
                                                                                                    	at androidx.camera.core.SafeCloseImageReaderProxy.acquireNextImage(SafeCloseImageReaderProxy.java:86)
                                                                                                    	at androidx.camera.core.ImageAnalysisBlockingAnalyzer.acquireImage(ImageAnalysisBlockingAnalyzer.java:39)
                                                                                                    	at androidx.camera.core.ImageAnalysisAbstractAnalyzer.onImageAvailable(ImageAnalysisAbstractAnalyzer.java:126)
                                                                                                    	at androidx.camera.core.SafeCloseImageReaderProxy.lambda$setOnImageAvailableListener$1$androidx-camera-core-SafeCloseImageReaderProxy(SafeCloseImageReaderProxy.java:211)
                                                                                                    	at androidx.camera.core.SafeCloseImageReaderProxy$$ExternalSyntheticLambda1.onImageAvailable(Unknown Source:4)
                                                                                                    	at androidx.camera.core.AndroidImageReaderProxy.lambda$setOnImageAvailableListener$0$androidx-camera-core-AndroidImageReaderProxy(AndroidImageReaderProxy.java:167)
                                                                                                    	at androidx.camera.core.AndroidImageReaderProxy$$ExternalSyntheticLambda0.run(Unknown Source:4)
                                                                                                    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                                                                                                    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
                                                                                                    	at java.lang.Thread.run(Thread.java:1012)
2024-10-06 18:18:26.434 30315-30361 AndroidRuntime          com...meire.lushlensobjectdetection  E  FATAL EXCEPTION: mrousavy/VisionCamera.video
                                                                                                    Process: com.diego.meire.lushlensobjectdetection, PID: 30315
                                                                                                    com.facebook.jni.CppException: Compiling JS failed: 1:1:invalid empty parentheses '( )' Buffer size 3 starts with: 280a29 and has protection mode(s): rw-p
                                                                                                    	at com.mrousavy.camera.frameprocessors.FrameProcessor.call(Native Method)
                                                                                                    	at com.mrousavy.camera.react.CameraView.onFrame(CameraView.kt:310)
                                                                                                    	at com.mrousavy.camera.core.FrameProcessorPipeline.analyze(FrameProcessorPipeline.kt:15)
                                                                                                    	at androidx.camera.core.ImageAnalysis.lambda$setAnalyzer$3(ImageAnalysis.java:573)
                                                                                                    	at androidx.camera.core.ImageAnalysis$$ExternalSyntheticLambda5.analyze(Unknown Source:2)
                                                                                                    	at androidx.camera.core.ImageAnalysisAbstractAnalyzer.lambda$analyzeImage$0$androidx-camera-core-ImageAnalysisAbstractAnalyzer(ImageAnalysisAbstractAnalyzer.java:284)
                                                                                                    	at androidx.camera.core.ImageAnalysisAbstractAnalyzer$$ExternalSyntheticLambda1.run(Unknown Source:14)
                                                                                                    	at android.os.Handler.handleCallback(Handler.java:938)
                                                                                                    	at android.os.Handler.dispatchMessage(Handler.java:99)
                                                                                                    	at android.os.Looper.loopOnce(Looper.java:201)
                                                                                                    	at android.os.Looper.loop(Looper.java:288)
                                                                                                    	at android.os.HandlerThread.run(HandlerThread.java:67)

Camera Device

"formats": [],
  "sensorOrientation": "landscape-left",
  "hardwareLevel": "full",
  "maxZoom": 7,
  "minZoom": 1,
  "maxExposure": 24,
  "supportsLowLightBoost": false,
  "neutralZoom": 1,
  "physicalDevices": [
    "wide-angle-camera"
  ],
  "supportsFocus": true,
  "supportsRawCapture": false,
  "isMultiCam": false,
  "minFocusDistance": 10,
  "minExposure": -24,
  "name": "0 (BACK) androidx.camera.camera2",
  "hasFlash": true,
  "hasTorch": true,
  "position": "back",
  "id": "0"
}

Device

Google Pixel 3

VisionCamera Version

4.5.3

Can you reproduce this issue in the VisionCamera Example app?

Yes, I can reproduce the same issue in the Example app here

Additional information

• I am using Expo
• I have enabled Frame Processors (react-native-worklets-core)
• I have read the Troubleshooting Guide
• I agree to follow this project's Code of Conduct
• I searched for similar issues in this repository and found none.

[maintenance-hans]

Guten Tag, Hans here 🍻! It looks like you have provided a detailed description of your issue, along with relevant logs and code. However, I noticed you mentioned that you can reproduce the issue in the example app. That's great, but in order to assist mrousavy better, please verify if the issue persists with the latest version of the library.

Additionally, it would help if you could provide the exact steps to reproduce the crash, including any specific configurations or environment details. If you haven’t done so already, you might try isolating the frameProcessor to ensure it is coded correctly and not causing the crash.

Keep up ze good work, and I'm sure mrousavy will take a look once he has a chance!

Note: If you think I made a mistake, please ping @mrousavy to take a look.

[ebrimasamba]

Am having the same problem. The moment i comment the frameprocessor line on the camera, the app doesn't crash.

These are the current versions am using
"react-native-vision-camera": "^4.5.3",
"react-native-worklets-core": "^1.3.3"

   const frameProcessor = useFrameProcessor((frame) => {
    "worklet";
    console.log(`Frame: ${frame.width}x${frame.height} (${frame.pixelFormat})`);
  }, []);

 <Camera
    frameProcessor={frameProcessor}
    style={styles.camera}
    device={device}
    isActive={isActive}
    pixelFormat="yuv"
  />

[mrousavy]

Does it help if you choose a smaller format? Lower videoResolution

[ebrimasamba]

It worked after i downgraded to 4.5.2 and restarted my pc, it works now even after i upgraded to the latest version, i think it must’ve been a caching problem on my end

…

On Thu, Oct 10, 2024 at 14:10 Marc Rousavy ***@***.***> wrote: Does it help if you choose a smaller format? Lower videoResolution — Reply to this email directly, view it on GitHub <#3229 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM7NRTGNPGXGZZDMP6Z6B7DZ22DEXAVCNFSM6AAAAABPOQO76SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBVGIYDIMJRGI> . You are receiving this because you commented.Message ID: ***@***.***>