Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MACE compile mode test failures with InternalTorchDynamoError and Unsupported exceptions in main #639

Open
yixian929 opened this issue Oct 15, 2024 · 1 comment

Comments

@yixian929
Copy link

Describe the bug
I wanted to use the compile mode for my model in MACE, but it failed during testing. The error occurs when running pytest for test_compile.py, causing various InternalTorchDynamoError and Unsupported exceptions. The regular (not compile) mode works without issues.

To Reproduce Steps to reproduce the behavior:

  1. Clone the MACE repository.
  2. Set up the environment using Python 3.10.15 with PyTorch 2.2.2 and CUDA 12.6.
  3. Run pytest on test_compile.py with the following configuration:
    " pytest test_compile.py "
  4. The errors include InternalTorchDynamoError: 'NoneType' object is not subscriptable and torch.dynamo.exc.Unsupported: Tensor.requires_grad.

Expected behavior The model should have compiled successfully using the available GPU with CUDA support. Instead, it failed with errors related to PyTorch Dynamo compilation.

Screenshots : N/A

Additional context Below is the platform and environment setup I used while encountering the bug:
Platform Information
• Operating System: Linux (64-bit, x86_64 architecture)
• Python Version: 3.10.15 (packaged by conda-forge, GCC 13.3.0)
• PyTorch Version: 2.2.2
• CUDA Version: 12.6
• NVIDIA Driver Version: 561.09
• GPU Model: NVIDIA GeForce RTX 3080 (10GB VRAM)
• CUDA Toolkit: nvcc version 12.6, build cuda_12.6.r12.6/compiler.34714021_0
• NVIDIA-SMI Output:
o Driver Version: 561.09
Additionally, torch.cuda.is_available() returns True, indicating CUDA is accessible. The errors persisted across multiple compile modes (default, reduce-overhead, max-autotune) and PyTorch versions.

@ilyes319
Copy link
Contributor

What other Pytorch versions have you tried?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants