From 486b587c6f54ec3064f2e1ab5e1f30ea3b0ad312 Mon Sep 17 00:00:00 2001 From: natke Date: Fri, 28 Jun 2024 17:41:54 -0700 Subject: [PATCH 1/4] Add troubleshooting page --- docs/genai/howto/setup-cuda-env.md | 2 +- docs/genai/howto/troubleshoot.md | 28 ++++++++++++++++++++++++++++ 2 files changed, 29 insertions(+), 1 deletion(-) create mode 100644 docs/genai/howto/troubleshoot.md diff --git a/docs/genai/howto/setup-cuda-env.md b/docs/genai/howto/setup-cuda-env.md index b87c83d8cbf37..354b2c134d215 100644 --- a/docs/genai/howto/setup-cuda-env.md +++ b/docs/genai/howto/setup-cuda-env.md @@ -4,7 +4,7 @@ description: Instructions to setup the CUDA environtment to run onnxruntime-gena has_children: false parent: How to grand_parent: Generate API (Preview) -nav_order: 4 +nav_order: 5 --- # Setup the CUDA Environment diff --git a/docs/genai/howto/troubleshoot.md b/docs/genai/howto/troubleshoot.md new file mode 100644 index 0000000000000..e0d19725ae4cb --- /dev/null +++ b/docs/genai/howto/troubleshoot.md @@ -0,0 +1,28 @@ +--- +title: Troubleshoot +description: How to troubleshoot common problems +has_children: false +parent: How to +grand_parent: Generate API (Preview) +nav_order: 1 +--- + +# Troubleshoot issues with ONNX Runtime generate() API +{: .no_toc } + +* TOC placeholder +{:toc} + +## Installation issues + +``` +ImportError: DLL load failed while importing onnxruntime_genai: A dynamic link library (DLL) initialization routine failed. +``` + +If you see this issue in a Conda environment on Windows, you need to upgrade the `C++ runtime for Visual Studio`. In the conda environment, run the following command: + +```bash +conda install conda-forge::vs2015_runtime +``` + +The onnxruntime-genai Python package should run without error after this extra step. From 92bfb4916384ea3fdd23b40be2c47d9f1e372529 Mon Sep 17 00:00:00 2001 From: natke Date: Mon, 1 Jul 2024 11:54:32 -0700 Subject: [PATCH 2/4] Add CUDA 12 installation instructions --- docs/genai/howto/build-model.md | 2 +- docs/genai/howto/install.md | 30 ++++++++++++++++++++++++++++++ docs/genai/howto/setup-cuda-env.md | 8 -------- docs/genai/howto/troubleshoot.md | 8 +++++++- 4 files changed, 38 insertions(+), 10 deletions(-) diff --git a/docs/genai/howto/build-model.md b/docs/genai/howto/build-model.md index fe8b91e03d1c2..9d048f874c65f 100644 --- a/docs/genai/howto/build-model.md +++ b/docs/genai/howto/build-model.md @@ -4,7 +4,7 @@ description: How to build models with ONNX Runtime generate() API has_children: false parent: How to grand_parent: Generate API (Preview) -nav_order: 2 +nav_order: 3 --- # Generate models using Model Builder diff --git a/docs/genai/howto/install.md b/docs/genai/howto/install.md index 28d97c4712010..2d5f6b9a192ff 100644 --- a/docs/genai/howto/install.md +++ b/docs/genai/howto/install.md @@ -13,12 +13,29 @@ nav_order: 1 * TOC placeholder {:toc} +## Pre-requisites + +### CUDA + +If you are installing the CUDA variant of onnxruntime-genai, the CUDA toolkit must be installed. + +The CUDA toolkit can be downloaded from the [CUDA Toolkit Archive](https://developer.nvidia.com/cuda-toolkit-archive). + +Ensure that the `CUDA_PATH` environment variable is set to the location of your CUDA installation. + ## Python packages +Note: only one of these packages should be installed in your application. + +### CPU + ```bash pip install numpy pip install onnxruntime-genai --pre ``` + +### DirectML + Append `-directml` for the library that is optimized for DirectML on Windows ```bash @@ -26,6 +43,10 @@ pip install numpy pip install onnxruntime-genai-directml --pre ``` +### CUDA + +#### CUDA 11 + Append `-cuda` for the library that is optimized for CUDA environments ```bash @@ -33,8 +54,17 @@ pip install numpy pip install onnxruntime-genai-cuda --pre --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/ ``` +#### CUDA 12 + +```bash +pip install numpy +pip install onnxruntime-genai-cuda --pre --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/ +``` + ## Nuget packages +Note: only one of these packages should be installed in your application. + ```bash dotnet add package Microsoft.ML.OnnxRuntimeGenAI --prerelease ``` diff --git a/docs/genai/howto/setup-cuda-env.md b/docs/genai/howto/setup-cuda-env.md index 354b2c134d215..61d0f2bd67157 100644 --- a/docs/genai/howto/setup-cuda-env.md +++ b/docs/genai/howto/setup-cuda-env.md @@ -13,12 +13,4 @@ nav_order: 5 * TOC placeholder {:toc} -## Install the CUDA Toolkit -On a CUDA capable machine, install the CUDA toolkit. onnxruntime-genai-cuda is built and packaged with CUDA-11.8. - -The CUDA toolkit can be downloaded from the [CUDA Toolkit Archive](https://developer.nvidia.com/cuda-toolkit-archive). - -## Special Instructions on Windows - -After CUDA toolkit installation completed on windows, ensure that the `CUDA_PATH` system environment variable has been set to the path where the toolkit was installed. This variable will be used when importing the onnxruntime_genai python module on windows. Unset or incorrectly set `CUDA_PATH` variable may lead to a `DLL load failed while importing onnxruntime_genai`. diff --git a/docs/genai/howto/troubleshoot.md b/docs/genai/howto/troubleshoot.md index e0d19725ae4cb..9f0fe8c389338 100644 --- a/docs/genai/howto/troubleshoot.md +++ b/docs/genai/howto/troubleshoot.md @@ -4,7 +4,7 @@ description: How to troubleshoot common problems has_children: false parent: How to grand_parent: Generate API (Preview) -nav_order: 1 +nav_order: 4 --- # Troubleshoot issues with ONNX Runtime generate() API @@ -15,6 +15,8 @@ nav_order: 1 ## Installation issues +### Windows Conda import error + ``` ImportError: DLL load failed while importing onnxruntime_genai: A dynamic link library (DLL) initialization routine failed. ``` @@ -26,3 +28,7 @@ conda install conda-forge::vs2015_runtime ``` The onnxruntime-genai Python package should run without error after this extra step. + +### Windows CUDA import error + +After CUDA toolkit installation completed on windows, ensure that the `CUDA_PATH` system environment variable has been set to the path where the toolkit was installed. This variable will be used when importing the onnxruntime_genai python module on Windows. Unset or incorrectly set `CUDA_PATH` variable may lead to a `DLL load failed while importing onnxruntime_genai`. \ No newline at end of file From 77743a8b331d0487ce9278bd0a34f635cba666ba Mon Sep 17 00:00:00 2001 From: natke Date: Mon, 1 Jul 2024 12:04:22 -0700 Subject: [PATCH 3/4] Remove old CUDA doc --- docs/genai/howto/install.md | 4 ++-- docs/genai/howto/setup-cuda-env.md | 16 ---------------- 2 files changed, 2 insertions(+), 18 deletions(-) delete mode 100644 docs/genai/howto/setup-cuda-env.md diff --git a/docs/genai/howto/install.md b/docs/genai/howto/install.md index 2d5f6b9a192ff..c77a9e089cc0b 100644 --- a/docs/genai/howto/install.md +++ b/docs/genai/howto/install.md @@ -45,10 +45,10 @@ pip install onnxruntime-genai-directml --pre ### CUDA -#### CUDA 11 - Append `-cuda` for the library that is optimized for CUDA environments +#### CUDA 11 + ```bash pip install numpy pip install onnxruntime-genai-cuda --pre --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/ diff --git a/docs/genai/howto/setup-cuda-env.md b/docs/genai/howto/setup-cuda-env.md deleted file mode 100644 index 61d0f2bd67157..0000000000000 --- a/docs/genai/howto/setup-cuda-env.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -title: Setup CUDA env -description: Instructions to setup the CUDA environtment to run onnxruntime-genai-cuda -has_children: false -parent: How to -grand_parent: Generate API (Preview) -nav_order: 5 ---- - -# Setup the CUDA Environment -{: .no_toc } - -* TOC placeholder -{:toc} - - From 868a7545430169c081eb559a3fd73ceef8aef48a Mon Sep 17 00:00:00 2001 From: natke Date: Mon, 1 Jul 2024 12:49:57 -0700 Subject: [PATCH 4/4] Update CUDA instructions in Phi3 vision tutorial --- docs/genai/tutorials/phi3-v.md | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/docs/genai/tutorials/phi3-v.md b/docs/genai/tutorials/phi3-v.md index cacdeb3ecbfc6..ee4c70038cd01 100644 --- a/docs/genai/tutorials/phi3-v.md +++ b/docs/genai/tutorials/phi3-v.md @@ -62,17 +62,28 @@ Support for Windows machines with GPUs other than NVIDIA is coming soon! ``` This command downloads the model into a folder called `cuda-int4-rtn-block-32`. -2. Install the generate() API +2. Setup your CUDA environment - ``` + Install the [CUDA toolkit](https://developer.nvidia.com/cuda-toolkit-archive). + + Ensure that the `CUDA_PATH` environment variable is set to the location of your CUDA installation. + + +3. Install the generate() API + + * CUDA 11 + + ```bash pip install numpy pip install --pre onnxruntime-genai-cuda --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/ ``` -3. Setup your CUDA environment - - Please follow the steps [here](../howto/setup-cuda-env.md) to setup the CUDA environment. + * CUDA 12 + ```bash + pip install numpy + pip install onnxruntime-genai-cuda --pre --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/ + ``` 4. Run the model