[Feature Request] Support CUDA 12 ?

[Bagashidayat1]

Describe the feature request

Is ONNXRUNTIME support CUDA 12 ?

Describe scenario use case

i got error on google colab using onnxruntime but it doesnt want to use T4 GPU

[karayakar]

I believe it is not supporting CUDA 12, I have tried 12.2 and getting below error.

'[ErrorCode:RuntimeException] D:\a_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1193 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load

[henryruhs]

2023-12-19 07:23:15.717360679 [E:onnxruntime:Default, provider_bridge_ort.cc:1498 TryGetProviderInfo_CUDA]
/onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1195 onnxruntime::Provider&
onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library
libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

Using the nightly builds via

pip install ort-nightly-gpu==1.17.0.dev20231215001 --pre --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple

[feiyuhuahuo]

Does anyone know how many days need to support CUDA12?

[Bagashidayat1]

Does anyone know how many days need to support CUDA12?

no idea

[phineas-pta]

current version 1.16.x support cuda 12, but must build it from source

[m1337v]

Using the nightly builds via
pip install ort-nightly-gpu==1.17.0.dev20231215001 --pre --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple

In my case it didn't help to use the latest nightly (...002). It's still looking for cuda11/libcublasLt.so.11 while cuda12 has libcublasLt.so.12.
I'm already trying to build from source for 2 days but with Google Collab limitations that seems like an impossible endeavor.

[E:onnxruntime:Default, provider_bridge_ort.cc:1498 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1195 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

[phineas-pta]

here my instructions to build onnxruntime with cuda 12 on colab

!apt-get install tensorrt-dev
!git clone \
    --branch=rel-1.16.3 \
    --single-branch \
    --depth=1 \
    --recurse-submodules \
    --shallow-submodules \
    https://github.com/microsoft/onnxruntime
%cd onnxruntime
!python tools/ci_build/build.py \
    --build_dir=build \
    --build \
    --update \
    --parallel \
    --skip_tests \
    --build_wheel \
    --enable_pybind \
    --config=Release \
    --skip_submodule_sync \
    --allow_running_as_root \
    --compile_no_warning_as_error \
    --use_cuda \
    --use_tensorrt \
    --cuda_home=/usr/local/cuda \
    --cudnn_home=/usr/lib/x86_64-linux-gnu \
    --tensorrt_home=/usr/lib/x86_64-linux-gnu \
    --cmake_extra_defines='CMAKE_CUDA_ARCHITECTURES=75' \
    --numpy_version="$(python -c 'import numpy; print(numpy.__version__)')"

take ~2h, no need gpu session (remove tensorrt if not needed)

wheel file created: onnxruntime/build/Release/dist/onnxruntime_gpu-1.16.3-cp310-cp310-linux_x86_64.whl save it elsewhere to use in gpu session

easier solution but only temporary: googlecolab/colabtools#4214 (comment)

[RoM4iK]

For people who will come after me, you can download whl here
https://dev.azure.com/onnxruntime/onnxruntime/_artifacts/feed/onnxruntime-cuda-12/PyPI/onnxruntime-gpu/overview/1.17.0

[netrunner-exe]

For people who will come after me, you can download whl here https://dev.azure.com/onnxruntime/onnxruntime/_artifacts/feed/onnxruntime-cuda-12/PyPI/onnxruntime-gpu/overview/1.17.0

Thank you! Alternatively, you can install the package via pip:
pip install onnxruntime-gpu==1.17.0 --index-url=https://pkgs.dev.azure.com/onnxruntime/onnxruntime/_packaging/onnxruntime-cuda-12/pypi/simple

Previously, I used a slightly different package, maybe someone will find it useful too:
pip install ort-nightly-gpu==1.17.0.dev20231205004 --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ort-cuda-12-nightly/pypi/simple/

[henryruhs]

Nice to see you Oleksandr.

What is the actual difference between both packages? I assume the one on azure is from the onnxruntime team and based on the latest build. The other by one of the onnxruntime devs and locked to a version?

Using a locked version feels more suitable.

[netrunner-exe]

Nice to see you Oleksandr.

What is the actual difference between both packages? I assume the one on azure is from the onnxruntime team and based on the latest build. The other by one of the onnxruntime devs and locked to a version?

Using a locked version feels more suitable.

Hi Henry, good to see you too. In my case, at the moment both packages work in Colab with Cuda 12, so it seems to me that there is no big difference which one to use. At least for now there are no critical bugs in the package based on the latest build.

[jindameias]

Thank you so much for addressing and fixing the existing defects in onnxruntime-gpu. Your efforts are truly appreciated!

[AndreyOrb]

So the nightly onnxruntime-gpu (v 1.17) should work?
@netrunner-exe Oleksandr, what CUDA and cuDNN versions (12.? and 8.9.6?) did you manage to compile with?

[netrunner-exe]

So the nightly onnxruntime-gpu (v 1.17) should work? @netrunner-exe Oleksandr, what CUDA and cuDNN versions (12.? and 8.9.6?) did you manage to compile with?

@AndreyOrb, I tested it on Ubuntu 22.04.3 LTS and CUDA 12.2 (Google Colab). @henryruhs Henry, after small tests of both packages, it turned out that Codeformer converted to .onnx does not work with pip install onnxruntime-gpu --index-url=https://pkgs.dev.azure.com/onnxruntime/onnxruntime/_packaging/onnxruntime-cuda-12/pypi/simple. The second one - works.

------SYSTEM INFO------
Platform: Linux-6.1.58+-x86_64-with-glibc2.35
Chainer: 7.8.1
ChainerX: Not Available
NumPy: 1.23.5
CuPy:
  OS                           : Linux-6.1.58+-x86_64-with-glibc2.35
  Python Version               : 3.10.12
  CuPy Version                 : 12.2.0
  CuPy Platform                : NVIDIA CUDA
  NumPy Version                : 1.23.5
  SciPy Version                : 1.11.4
  Cython Build Version         : 0.29.36
  Cython Runtime Version       : 3.0.7
  CUDA Root                    : /usr/local/cuda
  nvcc PATH                    : /usr/local/cuda/bin/nvcc
  CUDA Build Version           : 12020
  CUDA Driver Version          : 12020
  CUDA Runtime Version         : 12020
  cuBLAS Version               : 120205
  cuFFT Version                : 11008
  cuRAND Version               : 10303
  cuSOLVER Version             : (11, 5, 2)
  cuSPARSE Version             : 12102
  NVRTC Version                : (12, 2)
  Thrust Version               : 200101
  CUB Build Version            : 200101
  Jitify Build Version         : <unknown>
  cuDNN Build Version          : 8801
  cuDNN Version                : 8906
  NCCL Build Version           : 21602
  NCCL Runtime Version         : 21903
  cuTENSOR Version             : None
  cuSPARSELt Build Version     : None
  Device 0 Name                : Tesla T4
  Device 0 Compute Capability  : 75
  Device 0 PCI Bus ID          : 0000:00:04.0
iDeep: Not Available

[henryruhs]

@netrunner-exe I can report have issues with occluder.onnx on google colab using ort-nightly-gpu==1.17.0.dev20231205004.

[AndreyOrb]

@netrunner-exe Thanks for sharing the details.
Windows version has compiled and worked with my ONNX graphs with this command:
.\build.bat --config Release --build_shared_lib --parallel --use_cuda --cuda_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2" --cudnn_home "C:\Program Files\NVIDIA GPU Computing Toolkit\cuDNN\cudnn-windows-x86_64-8.9.6.50_cuda12-archive" --cuda_version 12.2 --skip_tests --use_mimalloc --use_tensorrt --tensorrt_home "C:\Program Files\NVIDIA GPU Computing Toolkit\TensorRT\TensorRT-8.6.1.6.Windows10.x86_64.cuda-12.0\TensorRT-8.6.1.6" --use_tensorrt_oss_parser

[jchen351]

The CUDA 12 support will be official in upcoming Onnxrtime 1.17 realase, in a week. Official instruction on how to download it will be provided then.

[barshag]

Any updates? (Looking for support 12.3 on local machine(ubuntu))

[arcosx]

solution from https://onnxruntime.ai/docs/install/

pip install onnxruntime-gpu --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/