Pytorch + cutlass : RuntimeError: CUDA Error cudaError_t.cudaSuccess

[parshakova]

following this https://github.com/NVIDIA/cutlass/blob/main/examples/python/02_pytorch_extension_grouped_gemm.ipynb
notebook, when running

import cutlass
import torch

dtype = torch.float16
plan = cutlass.op.GroupedGemm(element=dtype, layout=cutlass.LayoutType.RowMajor)
import random
random.seed(2023)

# Utility function to initialize A, B, C, and D matrices corresponding to dimensions M, N, and K
def initialize(dtype, M, N, K):
    sizes = [(M, K), (K, N), (M, N), (M, N)]
    return [torch.randint(-3, 3, size, device='cuda').to(dtype) for size in sizes]

# Utility function to generate `problems` GEMMs of random sizes
def generate_problems(problems):
    valid_sizes = [128, 256, 512, 1024]
    As, Bs, Cs, Ds = [], [], [], []
    for _ in range(problems):
        M, N, K = [random.choice(valid_sizes) for _ in range(3)]
        A, B, C, D = initialize(dtype, M, N, K)
        As.append(A)
        Bs.append(B)
        Cs.append(C)
        Ds.append(D)
    return As, Bs, Cs, Ds
As, Bs, Cs, Ds, = generate_problems(50)

plan.run(As, Bs, Cs, Ds, print_module=True)

the last line gives an error RuntimeError: CUDA Error cudaError_t.cudaSuccess. Please see the screenshot.
Do anyone know how I can fix this error?

[jackkosaian]

Hi, @parshakova. Could you please provide the following information:

• GPU being used
• CUDA version being used
• Are you using one of our Docker containers?

[parshakova]

I am on a GPU cluster, so to load CUTLASS I use module load (I have not installed it myself since I am not the root). Please see the attachment for all of the loaded modules.

NVIDIA GeForce RTX 2080 Ti
Cuda 11.5
Cutlass 3.1.0

[jackkosaian]

Thanks. Could you please also print the shared memory available on your device? You can get this via the following in Python:

from cuda import cudart
print(cudart.cudaGetDeviceProperties(0)[1].sharedMemPerBlockOptin)

[parshakova]

The output to print statement is 65536

[jackkosaian]

Thanks, @parshakova.

Taking a closer look at the original screenshot you posted, it appears that the call to cudaDeviceSynchronize() is returning successfully (the printout says that the returned error is cuda.cudaError_t.cudaSuccess), but it is not deemed equal to cudart.CUresult.CUDA_SUCCESS.

I tested out the following in CUDA Python 12.1 and 11.5:

print(cuda.CUresult.CUDA_SUCCESS == cudart.cudaError_t.cudaSuccess)

With 12.1, it prints True, while with 11.5, it prints False. My guess is that equivalence between these enums was not introduced until after CUDA Python 11.5, but I can't say for certain.

In any case, I expect that this could be circumvented by comparing the enum values. The following prints True in both versions of CUDA Python:

print(cuda.CUresult.CUDA_SUCCESS.value == cudart.cudaError_t.cudaSuccess.value)

Would you be willing to try out changing the line here to the following?

         err, = cudart.cudaDeviceSynchronize()
-        if err != cuda.CUresult.CUDA_SUCCESS:
+        if err.value != cuda.CUresult.CUDA_SUCCESS.value:
             raise RuntimeError("CUDA Error %s" % str(err))

You may also need to make a similar change for normal GEMMs here and here.

[parshakova]

Thanks it runs now.

But I run into a new problem RuntimeError: CUDA error: misaligned address from time to time.
Please see attachment

[jackkosaian]

Hmm, I have not experienced that when running on a 2080 Ti.

Could you please add print statements to retrieve the values of M, N, and K that lead to this error?

Once you have this could you also try running the line that is triggering this error in isolation in Python?

import torch
sizes = [(M, K), (K, N), (M, N), (M, N)]
print([torch.randint(-3, 3, size, device='cuda').to(dtype) for size in sizes])