diff --git a/docs/genai/howto/build-model.md b/docs/genai/howto/build-model.md index fe8b91e03d1c2..9d048f874c65f 100644 --- a/docs/genai/howto/build-model.md +++ b/docs/genai/howto/build-model.md @@ -4,7 +4,7 @@ description: How to build models with ONNX Runtime generate() API has_children: false parent: How to grand_parent: Generate API (Preview) -nav_order: 2 +nav_order: 3 --- # Generate models using Model Builder diff --git a/docs/genai/howto/install.md b/docs/genai/howto/install.md index 28d97c4712010..c77a9e089cc0b 100644 --- a/docs/genai/howto/install.md +++ b/docs/genai/howto/install.md @@ -13,12 +13,29 @@ nav_order: 1 * TOC placeholder {:toc} +## Pre-requisites + +### CUDA + +If you are installing the CUDA variant of onnxruntime-genai, the CUDA toolkit must be installed. + +The CUDA toolkit can be downloaded from the [CUDA Toolkit Archive](https://developer.nvidia.com/cuda-toolkit-archive). + +Ensure that the `CUDA_PATH` environment variable is set to the location of your CUDA installation. + ## Python packages +Note: only one of these packages should be installed in your application. + +### CPU + ```bash pip install numpy pip install onnxruntime-genai --pre ``` + +### DirectML + Append `-directml` for the library that is optimized for DirectML on Windows ```bash @@ -26,15 +43,28 @@ pip install numpy pip install onnxruntime-genai-directml --pre ``` +### CUDA + Append `-cuda` for the library that is optimized for CUDA environments +#### CUDA 11 + ```bash pip install numpy pip install onnxruntime-genai-cuda --pre --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/ ``` +#### CUDA 12 + +```bash +pip install numpy +pip install onnxruntime-genai-cuda --pre --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/ +``` + ## Nuget packages +Note: only one of these packages should be installed in your application. + ```bash dotnet add package Microsoft.ML.OnnxRuntimeGenAI --prerelease ``` diff --git a/docs/genai/howto/setup-cuda-env.md b/docs/genai/howto/setup-cuda-env.md deleted file mode 100644 index b87c83d8cbf37..0000000000000 --- a/docs/genai/howto/setup-cuda-env.md +++ /dev/null @@ -1,24 +0,0 @@ ---- -title: Setup CUDA env -description: Instructions to setup the CUDA environtment to run onnxruntime-genai-cuda -has_children: false -parent: How to -grand_parent: Generate API (Preview) -nav_order: 4 ---- - -# Setup the CUDA Environment -{: .no_toc } - -* TOC placeholder -{:toc} - -## Install the CUDA Toolkit - -On a CUDA capable machine, install the CUDA toolkit. onnxruntime-genai-cuda is built and packaged with CUDA-11.8. - -The CUDA toolkit can be downloaded from the [CUDA Toolkit Archive](https://developer.nvidia.com/cuda-toolkit-archive). - -## Special Instructions on Windows - -After CUDA toolkit installation completed on windows, ensure that the `CUDA_PATH` system environment variable has been set to the path where the toolkit was installed. This variable will be used when importing the onnxruntime_genai python module on windows. Unset or incorrectly set `CUDA_PATH` variable may lead to a `DLL load failed while importing onnxruntime_genai`. diff --git a/docs/genai/howto/troubleshoot.md b/docs/genai/howto/troubleshoot.md new file mode 100644 index 0000000000000..9f0fe8c389338 --- /dev/null +++ b/docs/genai/howto/troubleshoot.md @@ -0,0 +1,34 @@ +--- +title: Troubleshoot +description: How to troubleshoot common problems +has_children: false +parent: How to +grand_parent: Generate API (Preview) +nav_order: 4 +--- + +# Troubleshoot issues with ONNX Runtime generate() API +{: .no_toc } + +* TOC placeholder +{:toc} + +## Installation issues + +### Windows Conda import error + +``` +ImportError: DLL load failed while importing onnxruntime_genai: A dynamic link library (DLL) initialization routine failed. +``` + +If you see this issue in a Conda environment on Windows, you need to upgrade the `C++ runtime for Visual Studio`. In the conda environment, run the following command: + +```bash +conda install conda-forge::vs2015_runtime +``` + +The onnxruntime-genai Python package should run without error after this extra step. + +### Windows CUDA import error + +After CUDA toolkit installation completed on windows, ensure that the `CUDA_PATH` system environment variable has been set to the path where the toolkit was installed. This variable will be used when importing the onnxruntime_genai python module on Windows. Unset or incorrectly set `CUDA_PATH` variable may lead to a `DLL load failed while importing onnxruntime_genai`. \ No newline at end of file diff --git a/docs/genai/tutorials/phi3-v.md b/docs/genai/tutorials/phi3-v.md index cacdeb3ecbfc6..ee4c70038cd01 100644 --- a/docs/genai/tutorials/phi3-v.md +++ b/docs/genai/tutorials/phi3-v.md @@ -62,17 +62,28 @@ Support for Windows machines with GPUs other than NVIDIA is coming soon! ``` This command downloads the model into a folder called `cuda-int4-rtn-block-32`. -2. Install the generate() API +2. Setup your CUDA environment - ``` + Install the [CUDA toolkit](https://developer.nvidia.com/cuda-toolkit-archive). + + Ensure that the `CUDA_PATH` environment variable is set to the location of your CUDA installation. + + +3. Install the generate() API + + * CUDA 11 + + ```bash pip install numpy pip install --pre onnxruntime-genai-cuda --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/ ``` -3. Setup your CUDA environment - - Please follow the steps [here](../howto/setup-cuda-env.md) to setup the CUDA environment. + * CUDA 12 + ```bash + pip install numpy + pip install onnxruntime-genai-cuda --pre --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/ + ``` 4. Run the model