Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[E2E] Torchbench models got fail_accuracy #1222

Open
Tracked by #1223
mengfei25 opened this issue Dec 26, 2024 · 1 comment
Open
Tracked by #1223

[E2E] Torchbench models got fail_accuracy #1222

mengfei25 opened this issue Dec 26, 2024 · 1 comment

Comments

@mengfei25
Copy link
Contributor

🐛 Describe the bug

python benchmarks/dynamo/torchbench.py --accuracy --bfloat16 -d xpu -n10 --inference  --only shufflenet_v2_x1_0 --backend=inductor

============ Summary for torchbench bfloat16 inference accuracy ============
Real failed models: 14 [['timm_nfnet', 'fail_accuracy'], ['shufflenet_v2_x1_0', 'fail_accuracy'], ['mobilenet_v2', 'fail_accuracy'], ['timm_resnest', 'fail_accuracy'], ['mnasnet1_0', 'fail_accuracy'], ['timm_vovnet', 'fail_accuracy'], ['resnet152', 'fail_accuracy']]
============ Summary for torchbench bfloat16 training accuracy ============
Real failed models: 11 [['shufflenet_v2_x1_0', 'fail_accuracy'], ['mnasnet1_0', 'fail_accuracy'], ['timm_nfnet', 'fail_accuracy'], ['mobilenet_v2', 'fail_accuracy'], ['resnet152', 'fail_accuracy'], ['timm_resnest', 'fail_accuracy'], ['timm_vovnet', 'fail_accuracy']]
============ Summary for torchbench float16 training accuracy ============
Real failed models: 5 [['timm_regnet', 'fail_accuracy']
============ Summary for torchbench amp_bf16 inference accuracy ============
Real failed models: 17 [['resnet152', 'fail_accuracy'], ['resnet18', 'fail_accuracy'], ['timm_resnest', 'fail_accuracy'], ['mobilenet_v2', 'fail_accuracy'], ['timm_vovnet', 'fail_accuracy'], ['shufflenet_v2_x1_0', 'fail_accuracy'], ['mnasnet1_0', 'fail_accuracy'], ['timm_regnet', 'fail_accuracy'], ['resnext50_32x4d', 'fail_accuracy'], ['densenet121', 'fail_accuracy']]
============ Summary for torchbench amp_bf16 training accuracy ============
Real failed models: 14 [['resnet18', 'fail_accuracy'], ['shufflenet_v2_x1_0', 'fail_accuracy'], ['resnet50', 'fail_accuracy'], ['timm_resnest', 'fail_accuracy'], ['mnasnet1_0', 'fail_accuracy'], ['densenet121', 'fail_accuracy'], ['timm_regnet', 'fail_accuracy'], ['mobilenet_v2', 'fail_accuracy'], ['resnext50_32x4d', 'fail_accuracy'], ['resnet152', 'fail_accuracy'], ['timm_vovnet', 'fail_accuracy']]
============ Summary for torchbench amp_fp16 training accuracy ============
Real failed models: 5 [['timm_regnet', 'fail_accuracy']]

Versions

env:
python: 3.10
XPU_OPS: 9ed0a1a
TRITON_COMMIT_ID: e98b6fcb8df5b44eb0d0addb6767c573d37ba024
TORCH_COMMIT_ID: 4f8b7c4272db521f7ffc4070ce1bdece513d1183
TORCHBENCH_COMMIT_ID: 03cde49eba0580ed17f9ae2250832fd8af4ed756
TORCHVISION_COMMIT_ID: d23a6e1664d20707c11781299611436e1f0c104f
TORCHAUDIO_COMMIT_ID: a6b0a140cc13216975e8922093459019537bb80a
TRANSFORMERS_VERSION: 243e186efbf7fb93328dd6b34927a4e8c8f24395
TIMM_COMMIT_ID: ac3470188b914c5d7a5058a7e28b9eb685a62427
DRIVER_VERSION: 1.23.10.49.231129.50
KERNEL_VERSION: 5.15.0-73-generic #80-Ubuntu SMP Mon May 15 15:18:26 UTC 2023
BUNDLE_VERSION: 2025.0.1.20241113
OS_PRETTY_NAME: Ubuntu 22.04.2 LTS
GCC_VERSION: 11

@mengfei25
Copy link
Contributor Author

Caused by e035f6b

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant