How to convert quantized ONNX model from Tensor-Oriented format to Operator-Oriented format?

[hoangtv2000]

Describe the issue

I have the quantized model represented as the graph below and I want to convert all of QDQ operators in this model to QOperator operators, what should I do?

To reproduce

have not repoduced yet.

Urgency

No response

Platform

Linux

OS Version

Ubuntu 22.04

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.15.1

ONNX Runtime API

Python

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

Tasks

Preview Give feedback

No tasks being tracked yet.

[github-actions]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

[UsingtcNower]

I have the same question. Hi @hoangtv2000 do you find the solution?

[yufenglee]

@UsingtcNower and @hoangtv2000, you can run the onnxruntime offline-mode to optimize the model to Operator oriented: https://onnxruntime.ai/docs/performance/model-optimizations/graph-optimizations.html#offline-mode. BTW, why do you want to do this?

[hoangtv2000]

@UsingtcNower and @hoangtv2000, you can run the onnxruntime offline-mode to optimize the model to Operator oriented: https://onnxruntime.ai/docs/performance/model-optimizations/graph-optimizations.html#offline-mode. BTW, why do you want to do this?

I want to use your method to run on our self-designed chip which runs efficiently by integer arithmetic-only format.

[hoangtv2000]

I have the same question. Hi @hoangtv2000 do you find the solution?

You should use built-in static post-training quantization function of onnxruntime library quantize_static, remember to set quant_format to QuantFormat.QOperator if you want your ouput model has only QOperator, set per_channel to False if you want per-tensor quantization and activation_type & weight_type to QuantType.QUInt8 if you want your weight and activation tensors in uint8 format.