Skip to content

Commit

Permalink
Added a tool to quantize Gather to GatherBlockQuantized (#21697)
Browse files Browse the repository at this point in the history
### Description
Added code in MatMul4BitsQuantizer to quantize Gather to
GatherBlockQuantized.

Only Gather with constant data is quantized.

Since quantized data is in int4, the quantized model will force upgrade
to onnx opset 21.

The implementation purely relies on numpy. If optimization is needed,
C++ kernels can be added later.

Only support default RTN algorithm since GatherBlockQuantized require
zero points to have the same type as quantized data.

### Motivation and Context
Support quantizing gather to int4 in Web scenario.
  • Loading branch information
fajin-corp authored Aug 19, 2024
1 parent 7ae0b4c commit 64674c5
Show file tree
Hide file tree
Showing 3 changed files with 438 additions and 53 deletions.
Loading

0 comments on commit 64674c5

Please sign in to comment.