Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GPU] Integrate dynamic quantization for onednn #26940

Open
wants to merge 27 commits into
base: master
Choose a base branch
from
Open

[GPU] Integrate dynamic quantization for onednn #26940

wants to merge 27 commits into from
+419 −153

Conversation

isanghao
Copy link
Contributor

@isanghao isanghao commented Oct 8, 2024 •
edited
Loading

Details:

  • Integrated grouped dynamic quantization from onednn
  • Integrated asymmetric per-token dynamic quantization from onednn
  • Those are not enabled by default, yet

Tickets:

  • 148732, 157869, 157589

@github-actions github-actions bot added the category: GPU OpenVINO GPU plugin label Oct 8, 2024
@isanghao
WIP: dynamic_quantization at arbitrary group size
8d54fbe 
It works with group_size, but only for dyn_quan_ref

workaround for onednn accuracy issue

dyn_quan_opt is implemented

bug fix

test fix

do not use dyn_quan for u8 wei and wzp

fix after rebase

dyn_quan group size test
@isanghao
it works with dyn_quan_ref
e51fce7
@isanghao
it works with dyn_quan_opt
936f31c
@isanghao
code cleanup. it works with gs32, per_token
f0d4277
@isanghao
WIP: test build fix
186268a
@isanghao
unittext fix
9b410df
@isanghao
functional test fix
025cae9
@isanghao
activation zero-point works, but wrong result
6121205
@isanghao
support for u8 asym quantization
bbc4d85
@isanghao
bugfix: use onednn
e142383
@isanghao
azp does not support 8bit weight
4da19b1
@isanghao
remove unnecessary logs
045d86f
@isanghao
dyn_quan_opt kernel works
c0ba69b 
TODO:
* Check sym e2e accuracy
* check asym e2e accuracy
* test case for dyn_quan_opt for asym
@isanghao
test update
94430e1
@isanghao
support static-shape sd with dyn_quan
93c52fe
@isanghao
minor fix
5b46b29
@isanghao
code cleanup: save/load
8c2ab6b
@isanghao
code cleanup: dyn_quan_azp option from debug_config
f5edb08
@isanghao
code cleanup: remove is_dynamic
8d3d2b3
@isanghao
code cleanup: treat 2d tensor gracefully
6c850a6
@isanghao
code cleanup: treat 2d tensor gracefully
2262d96
@isanghao
code cleanup: enable dyn_quan by default for iGPU
d96c9ed
@isanghao
code cleanup: fix for unit tests
5ca8940
@isanghao
code cleanup
16e0d51
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 2, 2024

This PR will be closed in a week because of 2 weeks of no activity.

@github-actions github-actions bot added the Stale label Dec 2, 2024
@isanghao isanghao removed the Stale label Dec 2, 2024
@isanghao isanghao marked this pull request as ready for review December 2, 2024 09:21
@isanghao isanghao requested review from a team as code owners December 2, 2024 09:21
@isanghao isanghao changed the title [GPU] grouped dynamic quantization for onednn [GPU] Integrate dynamic quantization for onednn Dec 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@byungilm byungilm byungilm approved these changes

Assignees
No one assigned
Labels
category: GPU OpenVINO GPU plugin
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@isanghao @byungilm