CUDA 12 and session.get_providers() not showing CUDAExecutionProvider

[RomRoc]

Describe the documentation issue

I installed onnxruntime in Colab having T4 gpu and Cuda 12, using the commands the guide:
pip install onnxruntime-gpu --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/

When I run command:
onnxruntime.get_device()
I get GPU

But when I run command:
session = onnxruntime.InferenceSession(model_path, providers=['CUDAExecutionProvider', 'CPUExecutionProvider']) session.get_providers()
I get CPUExecutionProvider

In fact inference doesn't use GPU memory. Why? Should I ran any other steps?
Thanks

Page / URL

https://onnxruntime.ai/docs/install/#install-onnx-runtime-gpu-cuda-12x

[snnn]

The installation command looks wrong. Please uninstall the package then use the following commands:

pip install -r https://raw.githubusercontent.com/microsoft/onnxruntime/main/requirements.txt
pip install onnxruntime-gpu --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/ 

Should be "--index-url" instead of "--extra-index-url".

We will update the doc shortly.

[tianleiwu]

Currently colab uses torch 2.3.0+cu121 by default, which requires nvidia-cudnn-cu12==8.9.2.26 while onnxruntime-gpu 1.18.1 for cuda 12 requires cudnn 9.

You can setup colab like the following:

Choice 1: torch nightly 2.5.0 and onnxruntime-gpu 1.18.1 for cuda 12 and cudnn 9

!pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu124 --force-reinstall
!pip install -r https://raw.githubusercontent.com/microsoft/onnxruntime/main/requirements.txt
!pip install onnxruntime-gpu --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/ 

See example colab job for this combination.

Choice 2: torch 2.3.1+cu121 and onnxruntime-gpu 1.18.0 for cuda 12 and cudnn 8.

!pip install torch torchvision torchaudio
!pip install -r https://raw.githubusercontent.com/microsoft/onnxruntime/main/requirements.txt
!pip install onnxruntime-gpu==1.18.0 --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/ 

See example colab job for this combination.

Choice 3: torch 2.3.1+cu118 and onnxruntime-gpu 1.18.1 for cuda 11.8 and cudnn 8.

!pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 --force-reinstall
!pip install onnxruntime-gpu protobuf==3.20.3  --force-reinstall

See example colab job for this combination.

[huzaifahshamim]

The installation command looks wrong. Please uninstall the package then use the following commands:

pip install -r https://raw.githubusercontent.com/microsoft/onnxruntime/main/requirements.txt
pip install onnxruntime-gpu --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/ 

Should be "--index-url" instead of "--extra-index-url".

We will update the doc shortly.

Hi @snnn , I am attempting to do the following commands via a Dockerfile on a Jetson Xavier Orin. However, I run into the error: "no matching distribution found for onnxruntime-gpu". If you have any advice for me, I would greatly appreciate it.

On my normal computer - I get the following error when I try to access the link:

401 - Uh-oh, you do not have access.
The request requires authentication.
7/15/2024 7:36:45 PM (UTC)
24570155-dc22-4f4c-b823-6d0f732a020f

[tianleiwu]

I am attempting to do the following commands via a Dockerfile on a Jetson Xavier Orin. However, I run into the error: "no matching distribution found for onnxruntime-gpu". If you have any advice for me, I would greatly appreciate it.

@huzaifahshamim, for Jetson, you can download onnxruntim-gpu wheel in https://elinux.org/Jetson_Zoo for JetPack 6.0 (CUDA 12 and cuDNN 8.9).

[tianleiwu]

I also added two notebooks that use onnxruntime-gpu without importing pytorch:

CUDA 11:
https://colab.research.google.com/drive/1iDLX2g1di3FYRO2oKmdOPUuO378O0ITW?usp=sharing

CUDA 12:
https://colab.research.google.com/drive/1OcHA3ie57KD5fIJ9sFhrLR53lr0-6qNQ?usp=sharing

Installation:

pip install onnxruntime-gpu==1.18.1 --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/ 
pip install nvidia-cudnn-cu12==9.2.1.18 nvidia-cuda-runtime-cu12==12.5.82 nvidia-cufft-cu12==11.2.3.61

Then you can run inference like:

from ctypes import *
cudnn_libc = cdll.LoadLibrary("/usr/local/lib/python3.10/dist-packages/nvidia/cudnn/lib/libcudnn.so.9")
cublas_libc = cdll.LoadLibrary("/usr/local/lib/python3.10/dist-packages/nvidia/cublas/lib/libcublas.so.12")
cublaslt_libc = cdll.LoadLibrary("/usr/local/lib/python3.10/dist-packages/nvidia/cublas/lib/libcublasLt.so.12")
cudart_libc = cdll.LoadLibrary("/usr/local/lib/python3.10/dist-packages/nvidia/cuda_runtime/lib/libcudart.so.12")
cufft_libc = cdll.LoadLibrary("/usr/local/lib/python3.10/dist-packages/nvidia/cufft/lib/libcufft.so.11")

import onnxruntime
ort_session = onnxruntime.InferenceSession("model.onnx", providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
print(ort_session.get_providers())

The key here is to preload nvidia DLLs from python package (so that you need not depend on cuda or cudnn installed in system.)

[huzaifahshamim]

I am attempting to do the following commands via a Dockerfile on a Jetson Xavier Orin. However, I run into the error: "no matching distribution found for onnxruntime-gpu". If you have any advice for me, I would greatly appreciate it.

@huzaifahshamim, for Jetson, you can download onnxruntim-gpu wheel in https://elinux.org/Jetson_Zoo for JetPack 6.0 (CUDA 12 and cuDNN 8.9).

Hi @tianleiwu, thank you for your reply. So that is how I was initially attempting to download it onto my Docker container on Orin. I tried both onnxruntime 1.17 and 1.18 with python 3.10 but was running into errors that there was no .so.11 for libcublasLt, libcublas and other CUDA libraries. Which wheel should I use?

[RomRoc]

In my opinion it should be easier to use the most popular inference library (onnxruntime) in the most popular dev tool (Colab).
Maybe you can upgrade the library to be compliant to Colab?
In this way it will be easier even to deploy to VMs. In my case I can't use Cuda even in my T4 Azure instance.
Thanks

[tianleiwu]

@huzaifahshamim, errors that there was no .so.11 for libcublasLt means the wheel is for cuda 11. You will need JetPack 6.0 if you want to use CUDA 12, the wheel for older JetPack are for cuda 11.8. You may either install proper jetpack, cuda and cudnn version for the wheel, or build from source.
Since JetPack issue is not related to this, you can create a new issue for that.

[tianleiwu]

See https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements for cuda 12 requirement.
The root cause of this issue is PyTorch 2.3 use cuDNN 8.x, while onnxruntime-gpu 1.18.1 for cuda 12 requires cuDNN 9.

Note that the coming PyTorch 2.4 will support cuDNN 9.x, so that will help.

Example colab notebooks (ORT 1.18.1 +Torch 2.4, ORT 1.18.1 only) are provided. This issue is closed.

Feel free to create new issue if it is not related to colab.

[RomRoc]

I confirm that upgrading to Pytorch 2.4, it uses Cuda inference.
Thanks