onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasnt able to be loaded.

[nitinmukesh]

Describe the issue

Getting the runtime error

(venv) C:\sd\hallo-1.0.0>python scripts/inference.py --source_image input/1.jpg --driving_audio input/1.wav
A matching Triton is not available, some optimizations will not be enabled
Traceback (most recent call last):
  File "C:\sd\hallo-1.0.0\venv\lib\site-packages\xformers\__init__.py", line 55, in _is_triton_available
    from xformers.triton.softmax import softmax as triton_softmax  # noqa
  File "C:\sd\hallo-1.0.0\venv\lib\site-packages\xformers\triton\softmax.py", line 11, in <module>
    import triton
ModuleNotFoundError: No module named 'triton'
2024-07-01 15:05:48.3558955 [E:onnxruntime:Default, provider_bridge_ort.cc:1744 onnxruntime::TryGetProviderInfo_CUDA] C:\a\_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1426 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "C:\sd\hallo-1.0.0\venv\lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

*************** EP Error ***************
EP Error C:\a\_work\1\s\onnxruntime\python\onnxruntime_pybind_state.cc:866 onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasnt able to be loaded. Please install the correct version of CUDA andcuDNN as mentioned in the GPU requirements page  (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements),  make sure they're in the PATH, and that your GPU is supported.
 when using ['CUDAExecutionProvider', 'CPUExecutionProvider']
Falling back to ['CUDAExecutionProvider', 'CPUExecutionProvider'] and retrying.
****************************************
2024-07-01 15:05:48.6544172 [E:onnxruntime:Default, provider_bridge_ort.cc:1744 onnxruntime::TryGetProviderInfo_CUDA] C:\a\_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1426 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "C:\sd\hallo-1.0.0\venv\lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

Traceback (most recent call last):
  File "C:\sd\hallo-1.0.0\venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 419, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "C:\sd\hallo-1.0.0\venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 483, in _create_inference_session
    sess.initialize_session(providers, provider_options, disabled_optimizers)
RuntimeError: C:\a\_work\1\s\onnxruntime\python\onnxruntime_pybind_state.cc:866 onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasnt able to be loaded. Please install the correct version of CUDA andcuDNN as mentioned in the GPU requirements page  (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements),  make sure they're in the PATH, and that your GPU is supported.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\sd\hallo-1.0.0\scripts\inference.py", line 376, in <module>
    inference_process(command_line_args)
  File "C:\sd\hallo-1.0.0\scripts\inference.py", line 157, in inference_process
    with ImageProcessor(img_size, face_analysis_model_path) as image_processor:
  File "c:\sd\hallo-1.0.0\hallo\datasets\image_processor.py", line 100, in __init__
    self.face_analysis = FaceAnalysis(
  File "C:\sd\hallo-1.0.0\venv\lib\site-packages\insightface\app\face_analysis.py", line 31, in __init__
    model = model_zoo.get_model(onnx_file, **kwargs)
  File "C:\sd\hallo-1.0.0\venv\lib\site-packages\insightface\model_zoo\model_zoo.py", line 96, in get_model
    model = router.get_model(providers=providers, provider_options=provider_options)
  File "C:\sd\hallo-1.0.0\venv\lib\site-packages\insightface\model_zoo\model_zoo.py", line 40, in get_model
    session = PickableInferenceSession(self.onnx_file, **kwargs)
  File "C:\sd\hallo-1.0.0\venv\lib\site-packages\insightface\model_zoo\model_zoo.py", line 25, in __init__
    super().__init__(model_path, **kwargs)
  File "C:\sd\hallo-1.0.0\venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 432, in __init__
    raise fallback_error from e
  File "C:\sd\hallo-1.0.0\venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 427, in __init__
    self._create_inference_session(self._fallback_providers, None)
  File "C:\sd\hallo-1.0.0\venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 483, in _create_inference_session
    sess.initialize_session(providers, provider_options, disabled_optimizers)
RuntimeError: C:\a\_work\1\s\onnxruntime\python\onnxruntime_pybind_state.cc:866 onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasnt able to be loaded. Please install the correct version of CUDA andcuDNN as mentioned in the GPU requirements page  (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements),  make sure they're in the PATH, and that your GPU is supported.

(venv) C:\sd\hallo-1.0.0>pip list
Package                            Version      Editable project location
---------------------------------- ------------ -------------------------
absl-py                            2.1.0
accelerate                         0.28.0
aiofiles                           23.2.1
albucore                           0.0.12
albumentations                     1.4.10
alembic                            1.13.2
altair                             5.3.0
aniso8601                          9.0.1
annotated-types                    0.7.0
antlr4-python3-runtime             4.9.3
anyio                              4.4.0
astroid                            3.2.2
attrs                              23.2.0
audio-separator                    0.17.2
audioread                          3.0.1
av                                 12.1.0
beartype                           0.18.5
bitsandbytes                       0.43.1
blinker                            1.8.2
cachetools                         5.3.3
certifi                            2024.6.2
cffi                               1.16.0
cfgv                               3.4.0
charset-normalizer                 3.3.2
click                              8.1.7
cloudpickle                        3.0.0
colorama                           0.4.6
coloredlogs                        15.0.1
contextlib2                        21.6.0
contourpy                          1.2.1
cycler                             0.12.1
Cython                             3.0.10
decorator                          4.4.2
decord                             0.6.0
Deprecated                         1.2.14
diffq                              0.2.4
diffusers                          0.27.2
dill                               0.3.8
distlib                            0.3.8
dnspython                          2.6.1
docker                             7.1.0
easydict                           1.13
einops                             0.8.0
email_validator                    2.2.0
entrypoints                        0.4
exceptiongroup                     1.2.1
fastapi                            0.111.0
fastapi-cli                        0.0.4
ffmpy                              0.3.2
filelock                           3.15.4
Flask                              3.0.3
flatbuffers                        24.3.25
fonttools                          4.53.0
fsspec                             2024.6.1
gitdb                              4.0.11
GitPython                          3.1.43
gradio                             4.36.1
gradio_client                      1.0.1
graphene                           3.3
graphql-core                       3.2.3
graphql-relay                      3.2.0
greenlet                           3.0.3
h11                                0.14.0
hallo                              0.1.0        c:\sd\hallo-1.0.0
httpcore                           1.0.5
httptools                          0.6.1
httpx                              0.27.0
huggingface-hub                    0.23.4
humanfriendly                      10.0
identify                           2.5.36
idna                               3.7
imageio                            2.34.2
imageio-ffmpeg                     0.5.1
importlib_metadata                 7.1.0
importlib_resources                6.4.0
insightface                        0.7.3
isort                              5.13.2
itsdangerous                       2.2.0
jax                                0.4.30
jaxlib                             0.4.30
Jinja2                             3.1.4
joblib                             1.4.2
jsonschema                         4.22.0
jsonschema-specifications          2023.12.1
julius                             0.2.7
kiwisolver                         1.4.5
lazy_loader                        0.4
librosa                            0.10.2.post1
llvmlite                           0.43.0
Mako                               1.3.5
Markdown                           3.6
markdown-it-py                     3.0.0
MarkupSafe                         2.1.5
matplotlib                         3.9.0
mccabe                             0.7.0
mdurl                              0.1.2
mediapipe                          0.10.14
ml-collections                     0.1.1
ml-dtypes                          0.4.0
mlflow                             2.13.1
moviepy                            1.0.3
mpmath                             1.3.0
msgpack                            1.0.8
networkx                           3.3
nodeenv                            1.9.1
numba                              0.60.0
numpy                              1.26.4
omegaconf                          2.3.0
onnx                               1.16.1
onnx2torch                         1.5.14
onnxruntime-gpu                    1.18.0
opencv-contrib-python              4.9.0.80
opencv-python                      4.9.0.80
opencv-python-headless             4.9.0.80
opentelemetry-api                  1.25.0
opentelemetry-sdk                  1.25.0
opentelemetry-semantic-conventions 0.46b0
opt-einsum                         3.3.0
orjson                             3.10.5
packaging                          24.1
pandas                             2.2.2
pillow                             10.3.0
pip                                24.1.1
platformdirs                       4.2.2
pooch                              1.8.2
pre-commit                         3.7.1
prettytable                        3.10.0
proglog                            0.1.10
protobuf                           4.25.3
psutil                             6.0.0
pyarrow                            15.0.2
pycparser                          2.22
pydantic                           2.7.4
pydantic_core                      2.18.4
pydub                              0.25.1
Pygments                           2.18.0
pylint                             3.2.2
pyparsing                          3.1.2
pyreadline3                        3.4.1
python-dateutil                    2.9.0.post0
python-dotenv                      1.0.1
python-multipart                   0.0.9
pytz                               2024.1
pywin32                            306
PyYAML                             6.0.1
querystring-parser                 1.2.4
referencing                        0.35.1
regex                              2024.5.15
requests                           2.32.3
resampy                            0.4.3
rich                               13.7.1
rotary-embedding-torch             0.6.3
rpds-py                            0.18.1
ruff                               0.5.0
safetensors                        0.4.3
samplerate                         0.1.0
scikit-image                       0.24.0
scikit-learn                       1.5.0
scipy                              1.14.0
semantic-version                   2.10.0
setuptools                         70.0.0
shellingham                        1.5.4
six                                1.16.0
smmap                              5.0.1
sniffio                            1.3.1
sounddevice                        0.4.7
soundfile                          0.12.1
soxr                               0.3.7
SQLAlchemy                         2.0.31
sqlparse                           0.5.0
starlette                          0.37.2
sympy                              1.12.1
threadpoolctl                      3.5.0
tifffile                           2024.6.18
tokenizers                         0.15.2
tomli                              2.0.1
tomlkit                            0.12.0
toolz                              0.12.1
torch                              2.2.2+cu121
torchvision                        0.17.2+cu121
tqdm                               4.66.4
transformers                       4.39.2
typer                              0.12.3
typing_extensions                  4.12.2
tzdata                             2024.1
ujson                              5.10.0
urllib3                            2.2.2
uvicorn                            0.30.1
virtualenv                         20.26.3
waitress                           3.0.0
watchfiles                         0.22.0
wcwidth                            0.2.13
websockets                         11.0.3
Werkzeug                           3.0.3
wheel                              0.43.0
wrapt                              1.16.0
xformers                           0.0.25.post1
zipp                               3.19.2

To reproduce

Installed the dependencies for the following project
https://github.com/fudan-generative-vision/hallo/tree/v1.0.0

On starting the inference getting this error

Urgency

Low

Platform

Windows

OS Version

11

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.18.0

ONNX Runtime API

Python

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

cuda_12.5.r12.5/compiler.34177558_0

[nitinmukesh]

C:\Users\nitin>nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Wed_Apr_17_19:36:51_Pacific_Daylight_Time_2024
Cuda compilation tools, release 12.5, V12.5.40
Build cuda_12.5.r12.5/compiler.34177558_0

Also tried installing this way

(venv) C:\sd\hallo-1.0.0>pip install -U torch==2.2.2+cu121 torchvision==0.17.2+cu121 xformers==0.0.25.post1 --index-url https://download.pytorch.org/whl/cu121
Looking in indexes: https://download.pytorch.org/whl/cu121, https://pypi.ngc.nvidia.com
Collecting torch==2.2.2+cu121
  Downloading https://download.pytorch.org/whl/cu121/torch-2.2.2%2Bcu121-cp310-cp310-win_amd64.whl (2454.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 GB 12.8 MB/s eta 0:00:00
Requirement already satisfied: torchvision==0.17.2+cu121 in c:\sd\hallo-1.0.0\venv\lib\site-packages (0.17.2+cu121)
Collecting xformers==0.0.25.post1
  Downloading https://download.pytorch.org/whl/cu121/xformers-0.0.25.post1-cp310-cp310-win_amd64.whl (208.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 208.7/208.7 MB 12.8 MB/s eta 0:00:00
Requirement already satisfied: filelock in c:\sd\hallo-1.0.0\venv\lib\site-packages (from torch==2.2.2+cu121) (3.15.4)
Requirement already satisfied: typing-extensions>=4.8.0 in c:\sd\hallo-1.0.0\venv\lib\site-packages (from torch==2.2.2+cu121) (4.12.2)
Requirement already satisfied: sympy in c:\sd\hallo-1.0.0\venv\lib\site-packages (from torch==2.2.2+cu121) (1.12.1)
Requirement already satisfied: networkx in c:\sd\hallo-1.0.0\venv\lib\site-packages (from torch==2.2.2+cu121) (3.3)
Requirement already satisfied: jinja2 in c:\sd\hallo-1.0.0\venv\lib\site-packages (from torch==2.2.2+cu121) (3.1.4)
Requirement already satisfied: fsspec in c:\sd\hallo-1.0.0\venv\lib\site-packages (from torch==2.2.2+cu121) (2024.6.1)
Requirement already satisfied: numpy in c:\sd\hallo-1.0.0\venv\lib\site-packages (from torchvision==0.17.2+cu121) (1.26.4)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in c:\sd\hallo-1.0.0\venv\lib\site-packages (from torchvision==0.17.2+cu121) (10.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in c:\sd\hallo-1.0.0\venv\lib\site-packages (from jinja2->torch==2.2.2+cu121) (2.1.5)
Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in c:\sd\hallo-1.0.0\venv\lib\site-packages (from sympy->torch==2.2.2+cu121) (1.3.0)
Installing collected packages: torch, xformers
  Attempting uninstall: torch
    Found existing installation: torch 2.2.1+cu121
    Uninstalling torch-2.2.1+cu121:
      Successfully uninstalled torch-2.2.1+cu121
  Attempting uninstall: xformers
    Found existing installation: xformers 0.0.25
    Uninstalling xformers-0.0.25:
      Successfully uninstalled xformers-0.0.25
Successfully installed torch-2.2.2+cu121 xformers-0.0.25.post1

[Programmer-ZX]

how resolve?