[Build] AllocatorTest.CUDAAllocatorFallbackTest failed

[tianleiwu]

Describe the issue

Unit Test failed in A100 80GB

Urgency

No response

Target platform

ubuntu 20.04

Build script

sh build.sh --config Release --build_dir build/cuda12 --build_shared_lib --parallel --use_cuda
--cuda_version 12.5 --cuda_home /data/cuda12
--cudnn_home /data/cuda12/
--build_wheel --skip_tests
--cmake_generator Ninja
--cmake_extra_defines onnxruntime_BUILD_UNIT_TESTS=ON CMAKE_CUDA_ARCHITECTURES=80 onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON

Error / output

onnxruntime/build/cuda12/Release$ ./onnxruntime_test_all --gtest_filter=CUDA_EP_Unittest*

[ RUN ] AllocatorTest.CUDAAllocatorFallbackTest
unknown file: Failure
C++ exception with description "/home/tlwu/onnxruntime/onnxruntime/core/framework/bfc_arena.cc:376 void* onnxruntime::BFCArena::AllocateRawInternal(size_t, bool, onnxruntime::Stream*, bool, onnxruntime::WaitNotificationFn) Failed to allocate memory for requested buffer of size 42487119872
" thrown in the test body.

[ FAILED ] AllocatorTest.CUDAAllocatorFallbackTest (3 ms)

Visual Studio Version

No response

GCC / Compiler Version

11.4

[tianleiwu]

Test will fail if like 80% GPU memory has been used by other process. It seems that the test has some assumption that might not always pass.

[github-actions]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.