Generic "truncate_float" class for bf16 and fp16 quantization #3591

richagadgil · 2024-11-05T22:31:36Z

No description provided.

…ric_float

migraphx-bot · 2024-11-09T03:05:59Z

Test	Batch	Rate new ffec08	Rate old 4b96e1	Diff	Compare
torchvision-resnet50	64	3,260.10	3,260.40	-0.01%	✅
torchvision-resnet50_fp16	64	6,978.76	6,981.88	-0.04%	✅
torchvision-densenet121	32	2,435.50	2,436.50	-0.04%	✅
torchvision-densenet121_fp16	32	4,070.11	4,081.96	-0.29%	✅
torchvision-inceptionv3	32	1,638.92	1,638.04	0.05%	✅
torchvision-inceptionv3_fp16	32	2,764.34	2,760.86	0.13%	✅
cadene-inceptionv4	16	775.84	776.56	-0.09%	✅
cadene-resnext64x4	16	811.94	811.67	0.03%	✅
slim-mobilenet	64	7,535.93	7,540.50	-0.06%	✅
slim-nasnetalarge	64	211.50	211.49	0.00%	✅
slim-resnet50v2	64	3,504.60	3,506.73	-0.06%	✅
bert-mrpc-onnx	8	1,150.82	1,147.08	0.33%	✅
bert-mrpc-tf	1	463.44	465.87	-0.52%	✅
pytorch-examples-wlang-gru	1	416.07	423.73	-1.81%	✅
pytorch-examples-wlang-lstm	1	381.39	389.07	-1.97%	✅
torchvision-resnet50_1	1	770.18	788.22	-2.29%	✅
cadene-dpn92_1	1	397.62	402.19	-1.14%	✅
cadene-resnext101_1	1	382.61	382.83	-0.06%	✅
onnx-taau-downsample	1	343.21	343.07	0.04%	✅
dlrm-criteoterabyte	1	33.32	33.34	-0.05%	✅
dlrm-criteoterabyte_fp16	1	52.73	52.75	-0.04%	✅
agentmodel	1	8,446.16	8,325.15	1.45%	✅
unet_fp16	2	58.86	58.80	0.10%	✅
resnet50v1_fp16	1	944.40	953.06	-0.91%	✅
resnet50v1_int8	1	1,007.47	1,005.99	0.15%	✅
bert_base_cased_fp16	64	1,171.84	1,170.44	0.12%	✅
bert_large_uncased_fp16	32	363.32	363.37	-0.01%	✅
bert_large_fp16	1	198.91	198.99	-0.04%	✅
distilgpt2_fp16	16	2,205.06	2,201.23	0.17%	✅
yolov5s	1	540.06	536.00	0.76%	✅
tinyllama	1	43.43	43.45	-0.04%	✅
vicuna-fastchat	1	173.01	174.10	-0.63%	✅
whisper-tiny-encoder	1	419.04	418.74	0.07%	✅
whisper-tiny-decoder	1	428.75	425.97	0.65%	✅

This build is OK for merge ✅

migraphx-bot · 2024-11-09T03:06:01Z

❌bert-mrpc-onnx: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 249, in main
migraphx_arg = migraphx.argument(test_input)
RuntimeError: NumPy type info missing for N8migraphx14version_2_11_013generic_floatILj7ELj8ELj0EEE

❌bert-mrpc-tf: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 249, in main
migraphx_arg = migraphx.argument(test_input)
RuntimeError: NumPy type info missing for N8migraphx14version_2_11_013generic_floatILj7ELj8ELj0EEE

❌pytorch-examples-wlang-gru: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 249, in main
migraphx_arg = migraphx.argument(test_input)
RuntimeError: NumPy type info missing for N8migraphx14version_2_11_013generic_floatILj7ELj8ELj0EEE

❌pytorch-examples-wlang-lstm: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 249, in main
migraphx_arg = migraphx.argument(test_input)
RuntimeError: NumPy type info missing for N8migraphx14version_2_11_013generic_floatILj7ELj8ELj0EEE

❌torchvision-resnet50_1: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 249, in main
migraphx_arg = migraphx.argument(test_input)
RuntimeError: NumPy type info missing for N8migraphx14version_2_11_013generic_floatILj7ELj8ELj0EEE

❌cadene-dpn92_1: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 249, in main
migraphx_arg = migraphx.argument(test_input)
RuntimeError: NumPy type info missing for N8migraphx14version_2_11_013generic_floatILj7ELj8ELj0EEE

❌cadene-resnext101_1: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 249, in main
migraphx_arg = migraphx.argument(test_input)
RuntimeError: NumPy type info missing for N8migraphx14version_2_11_013generic_floatILj7ELj8ELj0EEE

❌dlrm-criteoterabyte: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 249, in main
migraphx_arg = migraphx.argument(test_input)
RuntimeError: NumPy type info missing for N8migraphx14version_2_11_013generic_floatILj7ELj8ELj0EEE

❌agentmodel: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 249, in main
migraphx_arg = migraphx.argument(test_input)
RuntimeError: NumPy type info missing for N8migraphx14version_2_11_013generic_floatILj7ELj8ELj0EEE

❌unet: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 249, in main
migraphx_arg = migraphx.argument(test_input)
RuntimeError: NumPy type info missing for N8migraphx14version_2_11_013generic_floatILj7ELj8ELj0EEE

❌resnet50v1: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 249, in main
migraphx_arg = migraphx.argument(test_input)
RuntimeError: NumPy type info missing for N8migraphx14version_2_11_013generic_floatILj7ELj8ELj0EEE

❌bert_base_cased_fp16: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 249, in main
migraphx_arg = migraphx.argument(test_input)
RuntimeError: NumPy type info missing for N8migraphx14version_2_11_013generic_floatILj7ELj8ELj0EEE

❌bert_large_uncased_fp16: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 249, in main
migraphx_arg = migraphx.argument(test_input)
RuntimeError: NumPy type info missing for N8migraphx14version_2_11_013generic_floatILj7ELj8ELj0EEE

❌bert_large: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 249, in main
migraphx_arg = migraphx.argument(test_input)
RuntimeError: NumPy type info missing for N8migraphx14version_2_11_013generic_floatILj7ELj8ELj0EEE

❌yolov5s: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 249, in main
migraphx_arg = migraphx.argument(test_input)
RuntimeError: NumPy type info missing for N8migraphx14version_2_11_013generic_floatILj7ELj8ELj0EEE

❌tinyllama: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 249, in main
migraphx_arg = migraphx.argument(test_input)
RuntimeError: NumPy type info missing for N8migraphx14version_2_11_013generic_floatILj7ELj8ELj0EEE

❌vicuna-fastchat: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 249, in main
migraphx_arg = migraphx.argument(test_input)
RuntimeError: NumPy type info missing for N8migraphx14version_2_11_013generic_floatILj7ELj8ELj0EEE

❌whisper-tiny-encoder: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 249, in main
migraphx_arg = migraphx.argument(test_input)
RuntimeError: NumPy type info missing for N8migraphx14version_2_11_013generic_floatILj7ELj8ELj0EEE

❌whisper-tiny-decoder: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 249, in main
migraphx_arg = migraphx.argument(test_input)
RuntimeError: NumPy type info missing for N8migraphx14version_2_11_013generic_floatILj7ELj8ELj0EEE

❌distilgpt2_fp16: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 249, in main
migraphx_arg = migraphx.argument(test_input)
RuntimeError: NumPy type info missing for N8migraphx14version_2_11_013generic_floatILj7ELj8ELj0EEE

richagadgil added 30 commits October 10, 2024 17:53

first pass at integrating generic float

c51c1ce

fix namespaces

134b408

fix mantissa

d4fa6eb

refactor

0b60841

refactor

7a646f1

add fp

ebe819b

fixed generic float class

379a77a

add fp32 test

174384c

remove import

787b651

update tests

1d1fa1c

fp16 tests that work

1791092

update tests

a2eb005

updated fp16 and fp32 tests

ff8ffc7

half tests

e36fd65

underflow and overflow tests

9ac4e2a

generate map

f05fd31

add more tests

cb4d92d

fix names

0cc1946

update tests

85a761b

remove and

65cf9ae

disable warning

fbabf54

fix tidy warning

549f5e6

migraphx py fix

d302e5d

add increments

8d475e3

fix warnings

a0fd055

disable duplicate branch warning

41379fe

add countzero_std

0c29c7b

ci error

4b012a8

simplify countl

dbaa3a8

fix ci

b2bd2a0

richagadgil and others added 20 commits October 29, 2024 14:17

Update generic_float.hpp

3354c6e

format

6de079b

Merge branch 'develop' into generic_float

7750874

Merge branch 'develop' into generic_float

801f485

fix bug

33e2c8d

Merge branch 'generic_float' of github.com:ROCm/AMDMIGraphX into gene…

9bb7198

…ric_float

fix err

b3c345d

edits

03df6f9

tidy and format

ad817b2

tidy etc

898417b

gf

aa5b9c9

fix tidy errs

6f72370

bf16 changes

0aab1a0

add flag to trace quantization passes (#3571)

7b965c0

bf16

5f5f13d

Update bf16.cpp

d64b124

Update bf16.hpp

a064eaa

Update bf16.hpp

befbd9e

update files with working version

08b9511

generic class for quant

12cafed

richagadgil requested review from a team and causten as code owners November 5, 2024 22:31

richagadgil self-assigned this Nov 5, 2024

richagadgil added 3 commits November 5, 2024 16:39

format

f604146

Merge branch 'develop' into generic_quant_class

edc7ccb

Update quantization.cpp

ffec081

richagadgil changed the base branch from develop to bf16 November 18, 2024 21:39

richagadgil changed the base branch from bf16 to develop November 18, 2024 21:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generic "truncate_float" class for bf16 and fp16 quantization #3591

Generic "truncate_float" class for bf16 and fp16 quantization #3591

richagadgil commented Nov 5, 2024

migraphx-bot commented Nov 9, 2024

migraphx-bot commented Nov 9, 2024

Generic "truncate_float" class for bf16 and fp16 quantization #3591

Are you sure you want to change the base?

Generic "truncate_float" class for bf16 and fp16 quantization #3591

Conversation

richagadgil commented Nov 5, 2024

migraphx-bot commented Nov 9, 2024

migraphx-bot commented Nov 9, 2024