Running RealESRGAN on Android

[md-rifatkhan]

Describe the issue

Someone help me by providing Java code to use this model on Android? Sorry for creating these type of stupid issue.

Python Code:

import cv2
import numpy as np
import onnxruntime as rt
import torch
import time

# Load the ONNX model with CUDA execution provider
sess = rt.InferenceSession('RealESRGAN0-x4plus.onnx')
print("loaded model.")

# Load the input image
in_image = cv2.imread('inputs.jpg', cv2.IMREAD_UNCHANGED)
print("loaded input image.")

# Convert BGR to RGB and transpose dimensions
in_mat = cv2.cvtColor(in_image, cv2.COLOR_BGR2RGB)
in_mat = np.transpose(in_mat, (2, 1, 0))[np.newaxis]
in_mat = in_mat.astype(np.float32)
in_mat = in_mat / 255

# Start time for measuring inference time
start_time = time.time()

# Get input and output names
input_name = sess.get_inputs()[0].name
output_name = sess.get_outputs()[0].name

print("Input name: ", input_name)
print("Output name: ", output_name)


# Convert input to torch tensor and move it to GPU
in_mat = torch.tensor(in_mat)

# Run inference
out_mat = sess.run([output_name], {input_name: in_mat.cpu().numpy()})[0]


# Measure and print elapsed time
elapsed_time = time.time() - start_time
print('Inference time: ',elapsed_time)
# Save the output image
out_mat = (out_mat.squeeze().transpose((2, 1, 0)) * 255).clip(0, 255).astype(np.uint8)
cv2.imwrite('output.jpg', out_mat)
print("Output image saved.")

To reproduce

I'm using RealESRGAN0-x4plus.onnx

Urgency

No response

Platform

Android

OS Version

13

ONNX Runtime Installation

Released Package

Compiler Version (if 'Built from Source')

No response

Package Name (if 'Released Package')

onnxruntime-android

ONNX Runtime Version or Commit ID

1.16.2

ONNX Runtime API

Java/Kotlin

Architecture

ARM64

Execution Provider

Other / Unknown

Execution Provider Library Version

No response

[skottmckay]

You may be able to use this script as an example of how to add pre/post processing to that specific model (assuming there's no major difference between the _x4 and _x4plus variants) :

https://github.com/microsoft/onnxruntime-inference-examples/blob/55a43e0a3a3011faaa19ff7363338519f8ffa0d1/mobile/examples/super_resolution/MAUI/real_esrgan_ort_e2e.py

With that you can provide jpg or png bytes as the input and it will return a png (could also return jpg by changing this line)

This example app uses a different model, but the usage is the same - provide bytes from a jpg or png as input, with it returning the bytes of the upscaled image.

https://github.com/microsoft/onnxruntime-inference-examples/blob/059bcedca432d6eca44fdc18f73a6d0f3fd2535b/mobile/examples/super_resolution/android/app/src/main/java/ai/onnxruntime/example/superresolution/SuperResPerformer.kt#L20-L49

[md-rifatkhan]

Exception caught when perform super resolution
java.lang.ClassCastException: java.nio.HeapByteBuffer cannot be cast to java.nio.ShortBuffer
at ai.onnxruntime.OrtUtil.prepareBuffer(OrtUtil.java:524)
at ai.onnxruntime.OnnxTensor.createTensor(OnnxTensor.java:690)
at ai.onnxruntime.OnnxTensor.createTensor(OnnxTensor.java:546)
at ai.onnxruntime.OnnxTensor.createTensor(OnnxTensor.java:525)

Thank you. I've completed all steps and converted the realesrgan-x4 from the repo model and converted FLOAT 32 to FLOAT 16. But facing this error.

internal data class Result(
    var outputBitmap: Bitmap? = null
) {}

 internal class SuperResPerformer {
    fun upscale(inputStream: InputStream, ortEnv: OrtEnvironment, ortSession: OrtSession): Result {
        var result = Result()

        // Step 1: convert image into byte array (raw image bytes)
        val rawImageBytes = inputStream.readBytes()
        Log.d("GG", "Raw Image Size: ${rawImageBytes.size} bytes")

        // Step 2: get the shape of the byte array and make ort tensor
        val shape = longArrayOf(rawImageBytes.size.toLong())

        val inputTensor = OnnxTensor.createTensor(
            ortEnv,
            ByteBuffer.wrap(rawImageBytes),
            shape,
            OnnxJavaType.FLOAT16
        )
        inputTensor.use {
            Log.d("GG", "Input Tensor Shape: ${Arrays.toString(shape)}")

            // Step 3: call ort inferenceSession run
            val output = ortSession.run(Collections.singletonMap("input", inputTensor))

            // Step 4: output analysis
            output.use {
                val rawOutput = (output?.get(0)?.value) as ByteArray
                val outputImageBitmap =
                    byteArrayToBitmap(rawOutput)

                // Step 5: set output result
                result.outputBitmap = outputImageBitmap
            }
        }
        return result
    }

    private fun byteArrayToBitmap(data: ByteArray): Bitmap {
        return BitmapFactory.decodeByteArray(data, 0, data.size)
    }
}

[Craigacp]

The OnnxTensor.createTensor method doesn't do any casting for you, so you'll need to upcast the uint8 to float16 yourself, or add a cast operation to the front of your ONNX model.

[md-rifatkhan]

If you don't mind, can you help me by providing the code?
here is the Model Information:

Name: torch_jit
Opset Version: 15
Producer: pytorch (Version: 2.0.1)

Input Information:
Name: input
Shape: [1, 3, 240, 240]
Data Type: float16

Output Information:
Name: output
Shape: [1, 3, 960, 960]
Data Type: float16

Model Inputs:
Name: input, Type: tensor_type {
  elem_type: 10
  shape {
    dim {
      dim_value: 1
    }
    dim {
      dim_value: 3
    }
    dim {
      dim_value: 240
    }
    dim {
      dim_value: 240
    }
  }
}


Model Outputs:
Name: output, Type: tensor_type {
  elem_type: 10
  shape {
    dim {
      dim_value: 1
    }
    dim {
      dim_value: 3
    }
    dim {
      dim_value: 960
    }
    dim {
      dim_value: 960
    }
  }
}

[Craigacp]

There's no direct cast, so you'll need to first convert your uint8 byte buffer into a FloatBuffer by copying each element, upcasting the uint8 to a bigger integer (making sure you don't flip the sign) before calling put on your new FloatBuffer, then call Fp16Conversions.convertFloatBufferToFp16Buffer.

[skottmckay]

Note that currently there's limited fp16 support on mobile, so most of the model will convert data to fp32 to run, which can result in worse performance than fp32. Quantizing to 8-bit is going to be the best option if you run into performance issues.