Does onnruntime-gpu support CUDA12.1？

[zhongqiu1245]

Describe the issue

Hello!
I use onnxruntime to run ptq(quant, static), but my cpu threads are always be killed.
So, I want to use gpu.
But, when I set provider=['CUDAExecutionProvider'], the error 'Failed to create cuda provider' was coming.
Does onnxruntime-gpu support CUDA12.1(pip install onnxrumtime)？
And pardon my poor english.

To reproduce

sess = ort.InferenceSession('xxxxxx.onnx', providers=['CUDAExecutionProvider'])

'Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/reference/execution-providers/CUDA-ExecutionProvider.html#requirements to ensure all dependencies are met.'

Urgency

Not really

Platform

Linux

OS Version

ubuntu 22.04

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

onnxrumtime 1.16.2

ONNX Runtime API

Python

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

CUDA12.1

[tianleiwu]

You will need wait for 1.17 release, which will have cuda 12 release package.

You can test release candidate build here:
ort-nightly-gpu : 1.17.0.dev20240118002

[snnn]

See also: #19137

[zhongqiu1245]

Thank you， guys！

[jarredou]

1.17 still generates same error with cuda 12.2 while nightly release was greatly working

[tianleiwu]

See https://onnxruntime.ai/docs/install/#requirements if you encountered issue with CUDA 12.*

[jarredou]

See https://onnxruntime.ai/docs/install/#requirements if you encountered issue with CUDA 12.*

No. That "stable" release has never worked on Google Colab and generates:

2024-03-17 16:17:28.444178554 [E:onnxruntime:Default, provider_bridge_ort.cc:1548 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1209 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

2024-03-17 16:17:28.444221332 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:861 CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirementsto ensure all dependencies are met.

Installing the nighlty release fixes that issue instantly.

[JohnHerry]

Which version can support CUDA12.1+cuDNN8.9?

[tianleiwu]

@JohnHerry,

1.18.0 supports cuda 12 and cudnn 8. See https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#cuda-12x

You can install that version like

pip install onnxruntime-gpu==1.18.0 --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/

[JohnHerry]

https://aiinfra.pkgs.visualstudio.com/PublicPackages/_p

Thank you very much for the help. by the way, I need to install with local package. where can I download this package?

[tianleiwu]

https://aiinfra.pkgs.visualstudio.com/PublicPackages/_p

Thank you very much for the help. by the way, I need to install with local package. where can I download this package?

Yes, you can download the whl file (See https://onnxruntime.ai/docs/install/#python-installs for wheel loculation), then pip install *.whl