[Build] MoE related unit tests fail for older architectures such as Pascal when building dev debug

[yuslepukhin]

Describe the issue

MoE unit tests fail on older architecture.
The tests have a particular requirement. If that requirement is not met it is pointless to run the tests.

Urgency

No response

Target platform

Windows

Build script

.\build.bat --build_dir .\cuda_build --config Debug --cmake_generator "Visual Studio 17 2022" --use_cuda --cuda_home c:\cuda\cuda --cudnn_home c:\cuda\cudnn --cuda_version 11.8 --build_wheel --build_shared_lib --parallel --skip_submodule_sync

Error / output

[ FAILED ] MoETest.MoETest_Gelu
[ FAILED ] MoETest.MoETest_Relu
[ FAILED ] MoETest.MoETest_Mixtral
Arch unsupported for MoE GEMM

Visual Studio Version

2022 64-bit 17.9.7

GCC / Compiler Version

No response

[tianleiwu]

@wangyems, In MoeGemmRunner::dispatch_to_arch, I saw it dispatch for SM from 70 to 89. We shall skip MOE tests for other GPUs (< 70, and >= 90).

BTW, could you test MOE in V100 to see whether it could run in SM=70? I saw some feature required SM>=80:

onnxruntime/onnxruntime/contrib_ops/cuda/moe/cutlass_extensions/gemm/warp/mma_tensorop_compute_B_with_f16.h

Line 135 in 0996d6e

"MmaTensorOpCvtBToA only supports Fp16 A or Bf16 A on Ampere+");

.

I think the requirements are like:
float: sm 70 to 89 (In theory, it shall support all GPUs, just a limitation of MoeGemmRunner::dispatch_to_arch)
float16: sm 70 to 89
bfloat16: sm 80 to 89

[snnn]

I tested it on V100 on Linux with CUDA 11.8.

[==========] 4 tests from 1 test suite ran. (10758 ms total)
[  PASSED  ] 3 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] MoETest.QMoETest_Mixtral_Int4

[snnn]

Our Linux training GPU machine pools use V100. Why they didn't catch this error?

[wangyems]

checking..