From 9282e66626a8bcc2667dbf8a4649e6f46597d70b Mon Sep 17 00:00:00 2001 From: natke Date: Tue, 6 Aug 2024 16:44:10 -0700 Subject: [PATCH 1/6] Update install instructions for separate packages --- docs/genai/howto/install.md | 88 ++++++++++++++++++++++++++++++++----- 1 file changed, 76 insertions(+), 12 deletions(-) diff --git a/docs/genai/howto/install.md b/docs/genai/howto/install.md index c77a9e089cc0b..3a2eee75a7afb 100644 --- a/docs/genai/howto/install.md +++ b/docs/genai/howto/install.md @@ -15,6 +15,10 @@ nav_order: 1 ## Pre-requisites +### ONNX Runtime + +Previous versions of ONNX Runtime generate() came bundled with the core ONNX Runtime binaries. From version 0.4.0 onwards, the packages are separated to allow a better developer experience. Specific platform instructions are included in each section below. + ### CUDA If you are installing the CUDA variant of onnxruntime-genai, the CUDA toolkit must be installed. @@ -23,61 +27,121 @@ The CUDA toolkit can be downloaded from the [CUDA Toolkit Archive](https://devel Ensure that the `CUDA_PATH` environment variable is set to the location of your CUDA installation. +Versions later than ONNX Runtime release bundle CUDA 12 by default. CUDA 11 is also supported in a separate package. Instructions are given below. + ## Python packages -Note: only one of these packages should be installed in your application. +Note: only one of these set of packages (CPU, DirectML, CUDA) should be installed in your environment. ### CPU +#### Intel CPU + ```bash -pip install numpy +pip install onnxruntime +pip install onnxruntime-genai --pre +``` + +#### Arm CPU + +```bash +pip install onnxruntime-qnn pip install onnxruntime-genai --pre ``` -### DirectML -Append `-directml` for the library that is optimized for DirectML on Windows +### DirectML ```bash pip install numpy +pip install onnxruntime-directml pip install onnxruntime-genai-directml --pre ``` ### CUDA -Append `-cuda` for the library that is optimized for CUDA environments - #### CUDA 11 ```bash -pip install numpy -pip install onnxruntime-genai-cuda --pre --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/ +# Install ORT nightly CUDA 11, change this to released version when it is released +pip install ort-nightly-gpu --index-url +https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ort-cuda-11-nightly/pypi/simple + + +# Install onnxruntime-genai built for CUDA 11 +pip install onnxruntime-genai-cuda-11 --pre --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/ ``` #### CUDA 12 ```bash -pip install numpy -pip install onnxruntime-genai-cuda --pre --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/ +# Install ORT nightly CUDA 12, change this to released version when it is released +pip install onnxruntime-gpu --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/ + +# Install onnxruntime-genai build for CUDA 12 +pip install onnxruntime-genai-cuda --pre --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/ ``` ## Nuget packages -Note: only one of these packages should be installed in your application. +Note: only one of these set of packages (CPU, DirectML, CUDA) should be installed in your application. + +### CPU ```bash +dotnet add package Microsoft.ML.OnnxRuntime dotnet add package Microsoft.ML.OnnxRuntimeGenAI --prerelease ``` -For the package that has been optimized for CUDA: +### CUDA + +#### CUDA 11 + +Add the following lines to a `nuget.config` file in the same folder as your `.csproj` file. + +```xml + + + + + + + +``` ```bash +dotnet add package Microsoft.ML.OnnxRuntime.Gpu dotnet add package Microsoft.ML.OnnxRuntimeGenAI.Cuda --prerelease ``` +#### CUDA 12 + +Add the following lines to a `nuget.config` file in the same folder as your `.csproj` file. + +```xml + + + + + + + +``` + + +```bash +dotnet add package Microsoft.ML.OnnxRuntime.Gpu +dotnet add package Microsoft.ML.OnnxRuntimeGenAI.Cuda --prerelease +``` + +### DirectML + For the package that has been optimized for DirectML: ```bash +dotnet add package Microsoft.ML.OnnxRuntime.DirectML dotnet add package Microsoft.ML.OnnxRuntimeGenAI.DirectML --prerelease ``` From a93c65a8045868f070d063ebb6534160fcd4114e Mon Sep 17 00:00:00 2001 From: natke Date: Tue, 6 Aug 2024 16:56:28 -0700 Subject: [PATCH 2/6] Remove numpy for DML install --- docs/genai/howto/install.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/genai/howto/install.md b/docs/genai/howto/install.md index 3a2eee75a7afb..ba427e4f30535 100644 --- a/docs/genai/howto/install.md +++ b/docs/genai/howto/install.md @@ -53,7 +53,6 @@ pip install onnxruntime-genai --pre ### DirectML ```bash -pip install numpy pip install onnxruntime-directml pip install onnxruntime-genai-directml --pre ``` From eb64a8544783da7847f4b1df9647ea8f77878f9f Mon Sep 17 00:00:00 2001 From: natke Date: Mon, 12 Aug 2024 19:29:56 -0700 Subject: [PATCH 3/6] Add pre-release versions of ONNX Runtime --- docs/genai/howto/install.md | 89 +++++++++++++++++++++++++------------ 1 file changed, 60 insertions(+), 29 deletions(-) diff --git a/docs/genai/howto/install.md b/docs/genai/howto/install.md index ba427e4f30535..cdb1501857d77 100644 --- a/docs/genai/howto/install.md +++ b/docs/genai/howto/install.md @@ -17,7 +17,9 @@ nav_order: 1 ### ONNX Runtime -Previous versions of ONNX Runtime generate() came bundled with the core ONNX Runtime binaries. From version 0.4.0 onwards, the packages are separated to allow a better developer experience. Specific platform instructions are included in each section below. +ONNX Runtime generate() versions 0.3.0 and earlier came bundled with the core ONNX Runtime binaries. From version 0.4.0 onwards, the packages are separated to allow a better developer experience. + +Platform specific instructions to install ONNX Runtime are included in each section below. ### CUDA @@ -27,25 +29,31 @@ The CUDA toolkit can be downloaded from the [CUDA Toolkit Archive](https://devel Ensure that the `CUDA_PATH` environment variable is set to the location of your CUDA installation. -Versions later than ONNX Runtime release bundle CUDA 12 by default. CUDA 11 is also supported in a separate package. Instructions are given below. ## Python packages -Note: only one of these set of packages (CPU, DirectML, CUDA) should be installed in your environment. +Note: only one of these sets of packages (CPU, DirectML, CUDA) should be installed in your environment. ### CPU -#### Intel CPU +#### x64 ```bash -pip install onnxruntime +# Install ONNX Runtime nightly. Update this to the released version when it is available. +pip install ort-nightly --extra-index-url +https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ort-nightly/pypi/simple + + pip install onnxruntime-genai --pre ``` -#### Arm CPU +#### Arm64 ```bash -pip install onnxruntime-qnn +# Install ONNX Runtime nightly QNN version (which supports Arm CPU). Update this to the released version when it is available. +pip install ort-nightly-qnn --extra-index-url +https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ort-nightly/pypi/simple + pip install onnxruntime-genai --pre ``` @@ -53,7 +61,10 @@ pip install onnxruntime-genai --pre ### DirectML ```bash -pip install onnxruntime-directml +# Install the nightly version of ONNX Runtime. Update this to the released version when it is available. +pip install ort-nightly-directml --extra-index-url +https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ort-nightly/pypi/simple + pip install onnxruntime-genai-directml --pre ``` @@ -62,13 +73,13 @@ pip install onnxruntime-genai-directml --pre #### CUDA 11 ```bash -# Install ORT nightly CUDA 11, change this to released version when it is released -pip install ort-nightly-gpu --index-url +# Install the nightly version of ONNX Runtime. Update this to the released version when it is available. +pip install ort-nightly-gpu --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ort-cuda-11-nightly/pypi/simple # Install onnxruntime-genai built for CUDA 11 -pip install onnxruntime-genai-cuda-11 --pre --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/ +pip install onnxruntime-genai-cuda-11 --pre --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/ ``` #### CUDA 12 @@ -78,17 +89,29 @@ pip install onnxruntime-genai-cuda-11 --pre --index-url=https://aiinfra.pkgs.vis pip install onnxruntime-gpu --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/ # Install onnxruntime-genai build for CUDA 12 -pip install onnxruntime-genai-cuda --pre --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/ +pip install onnxruntime-genai-cuda --pre --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/ ``` ## Nuget packages -Note: only one of these set of packages (CPU, DirectML, CUDA) should be installed in your application. +Note: only one of these set of packages (CPU, DirectML, CUDA) should be installed in your project. ### CPU +Add the following lines to a `nuget.config` file in the same folder as your `.csproj` file. + +```xml + + + + + + + +``` + ```bash -dotnet add package Microsoft.ML.OnnxRuntime +dotnet add package Microsoft.ML.OnnxRuntime --source ORT-Nightly --prerelease dotnet add package Microsoft.ML.OnnxRuntimeGenAI --prerelease ``` @@ -101,17 +124,16 @@ Add the following lines to a `nuget.config` file in the same folder as your `.cs ```xml - - - - + + + + ``` ```bash -dotnet add package Microsoft.ML.OnnxRuntime.Gpu -dotnet add package Microsoft.ML.OnnxRuntimeGenAI.Cuda --prerelease +dotnet add package Microsoft.ML.OnnxRuntime.Gpu --source ORT-Nightly --prerelease +dotnet add package Microsoft.ML.OnnxRuntimeGenAI.Cuda --source --prerelease ``` #### CUDA 12 @@ -121,26 +143,35 @@ Add the following lines to a `nuget.config` file in the same folder as your `.cs ```xml - - - - + + + + ``` ```bash -dotnet add package Microsoft.ML.OnnxRuntime.Gpu +dotnet add package Microsoft.ML.OnnxRuntime.Gpu --source ORT-Nightly --prerelease dotnet add package Microsoft.ML.OnnxRuntimeGenAI.Cuda --prerelease ``` ### DirectML -For the package that has been optimized for DirectML: +Add the following lines to a `nuget.config` file in the same folder as your `.csproj` file. + +```xml + + + + + + + +``` ```bash -dotnet add package Microsoft.ML.OnnxRuntime.DirectML +dotnet add package Microsoft.ML.OnnxRuntime.DirectML --source ORT-Nightly --prerelease dotnet add package Microsoft.ML.OnnxRuntimeGenAI.DirectML --prerelease ``` From ed718ac93336248294bf3afd429cd229175baefb Mon Sep 17 00:00:00 2001 From: natke Date: Mon, 12 Aug 2024 19:42:44 -0700 Subject: [PATCH 4/6] Remove extra index for genai cuda python packages --- docs/genai/howto/install.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/genai/howto/install.md b/docs/genai/howto/install.md index cdb1501857d77..a4bb686a01955 100644 --- a/docs/genai/howto/install.md +++ b/docs/genai/howto/install.md @@ -79,7 +79,7 @@ https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ort-cuda-11-nigh # Install onnxruntime-genai built for CUDA 11 -pip install onnxruntime-genai-cuda-11 --pre --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/ +pip install onnxruntime-genai-cuda-11 --pre ``` #### CUDA 12 @@ -89,7 +89,7 @@ pip install onnxruntime-genai-cuda-11 --pre --extra-index-url https://aiinfra.pk pip install onnxruntime-gpu --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/ # Install onnxruntime-genai build for CUDA 12 -pip install onnxruntime-genai-cuda --pre --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/ +pip install onnxruntime-genai-cuda --pre ``` ## Nuget packages From c34f26a6d9ccf24d50312df76deb61fd1ff71ca0 Mon Sep 17 00:00:00 2001 From: natke Date: Tue, 13 Aug 2024 15:34:56 -0700 Subject: [PATCH 5/6] Updated after review --- docs/genai/howto/install.md | 138 ++++++++---------------------------- 1 file changed, 28 insertions(+), 110 deletions(-) diff --git a/docs/genai/howto/install.md b/docs/genai/howto/install.md index a4bb686a01955..cfc87cba7b772 100644 --- a/docs/genai/howto/install.md +++ b/docs/genai/howto/install.md @@ -13,165 +13,83 @@ nav_order: 1 * TOC placeholder {:toc} -## Pre-requisites -### ONNX Runtime - -ONNX Runtime generate() versions 0.3.0 and earlier came bundled with the core ONNX Runtime binaries. From version 0.4.0 onwards, the packages are separated to allow a better developer experience. - -Platform specific instructions to install ONNX Runtime are included in each section below. - -### CUDA - -If you are installing the CUDA variant of onnxruntime-genai, the CUDA toolkit must be installed. - -The CUDA toolkit can be downloaded from the [CUDA Toolkit Archive](https://developer.nvidia.com/cuda-toolkit-archive). - -Ensure that the `CUDA_PATH` environment variable is set to the location of your CUDA installation. - - -## Python packages +## Python package installation Note: only one of these sets of packages (CPU, DirectML, CUDA) should be installed in your environment. ### CPU -#### x64 - ```bash -# Install ONNX Runtime nightly. Update this to the released version when it is available. -pip install ort-nightly --extra-index-url -https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ort-nightly/pypi/simple - - -pip install onnxruntime-genai --pre +pip install numpy +pip install onnxruntime-genai ``` -#### Arm64 +### DirectML ```bash -# Install ONNX Runtime nightly QNN version (which supports Arm CPU). Update this to the released version when it is available. -pip install ort-nightly-qnn --extra-index-url -https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ort-nightly/pypi/simple - -pip install onnxruntime-genai --pre +pip install numpy +pip install onnxruntime-genai-directml ``` +### CUDA -### DirectML - -```bash -# Install the nightly version of ONNX Runtime. Update this to the released version when it is available. -pip install ort-nightly-directml --extra-index-url -https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ort-nightly/pypi/simple +If you are installing the CUDA variant of onnxruntime-genai, the CUDA toolkit must be installed. -pip install onnxruntime-genai-directml --pre -``` +The CUDA toolkit can be downloaded from the [CUDA Toolkit Archive](https://developer.nvidia.com/cuda-toolkit-archive). -### CUDA +Ensure that the `CUDA_PATH` environment variable is set to the location of your CUDA installation. #### CUDA 11 ```bash -# Install the nightly version of ONNX Runtime. Update this to the released version when it is available. -pip install ort-nightly-gpu --extra-index-url -https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ort-cuda-11-nightly/pypi/simple - - -# Install onnxruntime-genai built for CUDA 11 -pip install onnxruntime-genai-cuda-11 --pre +pip install numpy +pip install onnxruntime-genai-cuda --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/ ``` #### CUDA 12 ```bash -# Install ORT nightly CUDA 12, change this to released version when it is released -pip install onnxruntime-gpu --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/ - -# Install onnxruntime-genai build for CUDA 12 -pip install onnxruntime-genai-cuda --pre +pip install numpy +pip install onnxruntime-genai-cuda --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/ ``` -## Nuget packages + +## Nuget package installation Note: only one of these set of packages (CPU, DirectML, CUDA) should be installed in your project. -### CPU +### Pre-requisites -Add the following lines to a `nuget.config` file in the same folder as your `.csproj` file. +#### ONNX Runtime dependency -```xml - - - - - - - -``` +ONNX Runtime generate() versions 0.3.0 and earlier came bundled with the core ONNX Runtime binaries. From version 0.4.0 onwards, the packages are separated to allow a more flexible developer experience. -```bash -dotnet add package Microsoft.ML.OnnxRuntime --source ORT-Nightly --prerelease -dotnet add package Microsoft.ML.OnnxRuntimeGenAI --prerelease -``` - -### CUDA - -#### CUDA 11 +Version 0.4.0-rc1 depends on the ONNX Runtime version 1.19.0 RC. To install 0.4.0-rc1, add the following nuget source *before* installing the ONNX Runtime generate() nuget package. -Add the following lines to a `nuget.config` file in the same folder as your `.csproj` file. - -```xml - - - - - - - ``` - -```bash -dotnet add package Microsoft.ML.OnnxRuntime.Gpu --source ORT-Nightly --prerelease -dotnet add package Microsoft.ML.OnnxRuntimeGenAI.Cuda --source --prerelease +dotnet nuget add source https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/nuget/v3/index.json --name ORT-Nightly ``` -#### CUDA 12 +Install only one of the following packages: CPU, CUDA or DirectML. -Add the following lines to a `nuget.config` file in the same folder as your `.csproj` file. +### CPU -```xml - - - - - - - +```bash +dotnet add package Microsoft.ML.OnnxRuntimeGenAI --prerelease ``` +### CUDA + +Note: only CUDA 11 is supported for versions 0.3.0 and earlier, and only CUDA 12 is supported for versions 0.4.0 and later. ```bash -dotnet add package Microsoft.ML.OnnxRuntime.Gpu --source ORT-Nightly --prerelease dotnet add package Microsoft.ML.OnnxRuntimeGenAI.Cuda --prerelease ``` ### DirectML -Add the following lines to a `nuget.config` file in the same folder as your `.csproj` file. - -```xml - - - - - - - -``` - ```bash -dotnet add package Microsoft.ML.OnnxRuntime.DirectML --source ORT-Nightly --prerelease dotnet add package Microsoft.ML.OnnxRuntimeGenAI.DirectML --prerelease ``` From 1fa71a9e2397316175a8b81856982936342b59a0 Mon Sep 17 00:00:00 2001 From: natke Date: Tue, 13 Aug 2024 15:43:25 -0700 Subject: [PATCH 6/6] Minor edits --- docs/genai/howto/install.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/genai/howto/install.md b/docs/genai/howto/install.md index cfc87cba7b772..86f969c8ccf32 100644 --- a/docs/genai/howto/install.md +++ b/docs/genai/howto/install.md @@ -57,7 +57,7 @@ pip install onnxruntime-genai-cuda --index-url=https://aiinfra.pkgs.visualstudio ## Nuget package installation -Note: only one of these set of packages (CPU, DirectML, CUDA) should be installed in your project. +Note: install only one of these packages (CPU, DirectML, CUDA) in your project. ### Pre-requisites @@ -71,8 +71,6 @@ Version 0.4.0-rc1 depends on the ONNX Runtime version 1.19.0 RC. To install 0.4. dotnet nuget add source https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/nuget/v3/index.json --name ORT-Nightly ``` -Install only one of the following packages: CPU, CUDA or DirectML. - ### CPU ```bash