Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for MatMulNBits #3496

Merged
merged 8 commits into from
Oct 10, 2024
Merged

Add support for MatMulNBits #3496

merged 8 commits into from
Oct 10, 2024

Conversation

music-dino
Copy link
Collaborator

@music-dino music-dino requested a review from causten as a code owner October 2, 2024 12:51
@music-dino music-dino added the Onnx Operators Adding or modifying an Onnx Operator in the MIGraphX codebase label Oct 2, 2024
@TedThemistokleous TedThemistokleous added roadmap Tasks to finish for a release high priority A PR with high priority for review and merging. labels Oct 2, 2024
@codecov Codecov
Copy link

codecov bot commented Oct 2, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 98.52941% with 1 line in your changes missing coverage. Please review.

Project coverage is 92.04%. Comparing base (e2e9219) to head (ffc5304).
Report is 91 commits behind head on develop.

Files with missing lines Patch % Lines
src/onnx/parse_matmulnbits.cpp 98.52% 1 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff             @@
##           develop    #3496      +/-   ##
===========================================
+ Coverage    92.02%   92.04%   +0.02%     
===========================================
  Files          509      510       +1     
  Lines        21005    21073      +68     
===========================================
+ Hits         19330    19397      +67     
- Misses        1675     1676       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚨 Try these New Features:

@migraphx-bot
Copy link
Collaborator

Test Batch Rate new
ffc530		 Rate old
e2e921		 Diff Compare
torchvision-resnet50 64 3,259.98 3,255.97 0.12% ✅
torchvision-resnet50_fp16 64 6,967.09 6,971.30 -0.06% ✅
torchvision-densenet121 32 2,432.55 2,431.20 0.06% ✅
torchvision-densenet121_fp16 32 4,098.34 4,087.45 0.27% ✅
torchvision-inceptionv3 32 1,637.25 1,635.71 0.09% ✅
torchvision-inceptionv3_fp16 32 2,753.28 2,755.82 -0.09% ✅
cadene-inceptionv4 16 780.20 780.33 -0.02% ✅
cadene-resnext64x4 16 808.00 806.94 0.13% ✅
slim-mobilenet 64 7,468.23 7,464.88 0.04% ✅
slim-nasnetalarge 64 208.42 208.51 -0.04% ✅
slim-resnet50v2 64 3,439.94 3,435.02 0.14% ✅
bert-mrpc-onnx 8 1,146.28 1,146.59 -0.03% ✅
bert-mrpc-tf 1 320.26 318.44 0.57% ✅
pytorch-examples-wlang-gru 1 385.11 421.02 -8.53% 🔴
pytorch-examples-wlang-lstm 1 381.64 374.33 1.95% ✅
torchvision-resnet50_1 1 767.07 803.63 -4.55% 🔴
cadene-dpn92_1 1 396.42 395.85 0.15% ✅
cadene-resnext101_1 1 371.70 379.14 -1.96% ✅
onnx-taau-downsample 1 366.21 366.52 -0.09% ✅
dlrm-criteoterabyte 1 35.03 35.01 0.04% ✅
dlrm-criteoterabyte_fp16 1 58.17 58.19 -0.03% ✅
agentmodel 1 8,135.64 8,063.61 0.89% ✅
unet_fp16 2 59.03 58.95 0.14% ✅
resnet50v1_fp16 1 906.46 950.25 -4.61% 🔴
resnet50v1_int8 1 979.66 971.97 0.79% ✅
bert_base_cased_fp16 64 1,171.44 1,170.73 0.06% ✅
bert_large_uncased_fp16 32 363.68 363.46 0.06% ✅
bert_large_fp16 1 211.40 213.45 -0.96% ✅
distilgpt2_fp16 16 2,204.69 2,199.49 0.24% ✅
yolov5s 1 547.73 539.25 1.57% ✅
tinyllama 1 43.46 43.47 -0.03% ✅
vicuna-fastchat 1 170.22 174.86 -2.65% ✅
whisper-tiny-encoder 1 418.13 417.66 0.11% ✅
whisper-tiny-decoder 1 427.72 427.92 -0.05% ✅

This build is not recommended to merge 🔴

@migraphx-bot
Copy link
Collaborator


     ✅ bert-mrpc-onnx: PASSED: MIGraphX meets tolerance

     ✅ bert-mrpc-tf: PASSED: MIGraphX meets tolerance

     ✅ pytorch-examples-wlang-gru: PASSED: MIGraphX meets tolerance

     ✅ pytorch-examples-wlang-lstm: PASSED: MIGraphX meets tolerance

     ✅ torchvision-resnet50_1: PASSED: MIGraphX meets tolerance

     ✅ cadene-dpn92_1: PASSED: MIGraphX meets tolerance

     ✅ cadene-resnext101_1: PASSED: MIGraphX meets tolerance

     ✅ dlrm-criteoterabyte: PASSED: MIGraphX meets tolerance

     ✅ agentmodel: PASSED: MIGraphX meets tolerance

     ✅ unet: PASSED: MIGraphX meets tolerance

     ✅ resnet50v1: PASSED: MIGraphX meets tolerance

     ✅ bert_base_cased_fp16: PASSED: MIGraphX meets tolerance

🔴bert_large_uncased_fp16: FAILED: MIGraphX is not within tolerance - check verbose output


     ✅ bert_large: PASSED: MIGraphX meets tolerance

     ✅ yolov5s: PASSED: MIGraphX meets tolerance

     ✅ tinyllama: PASSED: MIGraphX meets tolerance

     ✅ vicuna-fastchat: PASSED: MIGraphX meets tolerance

     ✅ whisper-tiny-encoder: PASSED: MIGraphX meets tolerance

     ✅ whisper-tiny-decoder: PASSED: MIGraphX meets tolerance

     ✅ distilgpt2_fp16: PASSED: MIGraphX meets tolerance

@TedThemistokleous
Copy link
Collaborator

Odd CI is all green but here its showing it didn't run. Rekicked this as I don't think this update is showing up on github.

TedThemistokleous
TedThemistokleous reviewed Oct 3, 2024
View reviewed changes
Copy link
Collaborator

@TedThemistokleous TedThemistokleous left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks alright. Will see what CI says in the morning. Just another question I have about how you handle blob_sizes.

@music-dino
Copy link
Collaborator Author

Overall looks alright. Will see what CI says in the morning. Just another question I have about how you handle blob_sizes.

There's a clang_debug failure, but I can't access the logs.

@TedThemistokleous
Copy link
Collaborator

Overall looks alright. Will see what CI says in the morning. Just another question I have about how you handle blob_sizes.

There's a clang_debug failure, but I can't access the logs.

Looks clean from my end. Dont worry about it.

TedThemistokleous
TedThemistokleous approved these changes Oct 4, 2024
View reviewed changes
Copy link
Collaborator

@TedThemistokleous TedThemistokleous left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. looks fine and nothing else jumps out at me.

@causten causten merged commit a1e3396 into develop Oct 10, 2024
49 checks passed
@causten causten deleted the matmulnbits branch October 10, 2024 02:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@TedThemistokleous TedThemistokleous TedThemistokleous approved these changes

@causten causten Awaiting requested review from causten causten is a code owner

@pfultz2 pfultz2 Awaiting requested review from pfultz2

@turneram turneram Awaiting requested review from turneram

Assignees

@music-dino music-dino

Labels
high priority A PR with high priority for review and merging. Onnx Operators Adding or modifying an Onnx Operator in the MIGraphX codebase roadmap Tasks to finish for a release
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add support for MatMulNBits
4 participants
@music-dino @migraphx-bot @TedThemistokleous @causten