From ba4fa4cf899df02a6bc78d4229d49ca3502210cb Mon Sep 17 00:00:00 2001 From: Misha Chornyi <99709299+mc-nv@users.noreply.github.com> Date: Tue, 10 Dec 2024 13:28:04 -0800 Subject: [PATCH 1/3] Update README and versions for 1.47.0 / 24.12 (#951) --- Dockerfile | 4 +- README.md | 115 +----------------- VERSION | 2 +- docs/bls_quick_start.md | 4 +- docs/config.md | 2 +- docs/ensemble_quick_start.md | 4 +- docs/kubernetes_deploy.md | 2 +- docs/mm_quick_start.md | 4 +- docs/quick_start.md | 4 +- helm-chart/values.yaml | 2 +- .../config/input/config_defaults.py | 2 +- 11 files changed, 16 insertions(+), 129 deletions(-) diff --git a/Dockerfile b/Dockerfile index 399b285b..fc2f693d 100644 --- a/Dockerfile +++ b/Dockerfile @@ -12,8 +12,8 @@ # See the License for the specific language governing permissions and # limitations under the License. -ARG BASE_IMAGE=nvcr.io/nvidia/tritonserver:24.11-py3 -ARG TRITONSDK_BASE_IMAGE=nvcr.io/nvidia/tritonserver:24.11-py3-sdk +ARG BASE_IMAGE=nvcr.io/nvidia/tritonserver:24.12-py3 +ARG TRITONSDK_BASE_IMAGE=nvcr.io/nvidia/tritonserver:24.12-py3-sdk ARG MODEL_ANALYZER_VERSION=1.47.0dev ARG MODEL_ANALYZER_CONTAINER_VERSION=24.12dev diff --git a/README.md b/README.md index 3a825635..e11612ad 100644 --- a/README.md +++ b/README.md @@ -19,117 +19,4 @@ limitations under the License. # Triton Model Analyzer > [!Warning] -> -> ##### LATEST RELEASE -> -> You are currently on the `main` branch which tracks under-development progress towards the next release.
-> The latest release of the Triton Model Analyzer is 1.46.0 and is available on branch -> [r24.11](https://github.com/triton-inference-server/model_analyzer/tree/r24.11). - -Triton Model Analyzer is a CLI tool which can help you find a more optimal configuration, on a given piece of hardware, for single, multiple, ensemble, or BLS models running on a [Triton Inference Server](https://github.com/triton-inference-server/server/). Model Analyzer will also generate reports to help you better understand the trade-offs of the different configurations along with their compute and memory requirements. -

- -# Features - -### Search Modes - -- [Optuna Search](docs/config_search.md#optuna-search-mode) **_-ALPHA RELEASE-_** allows you to search for every parameter that can be specified in the model configuration, using a hyperparameter optimization framework. Please see the [Optuna](https://optuna.org/) website if you are interested in specific details on how the algorithm functions. - -- [Quick Search](docs/config_search.md#quick-search-mode) will **sparsely** search the [Max Batch Size](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#maximum-batch-size), - [Dynamic Batching](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#dynamic-batcher), and - [Instance Group](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#instance-groups) spaces by utilizing a heuristic hill-climbing algorithm to help you quickly find a more optimal configuration - -- [Automatic Brute Search](docs/config_search.md#automatic-brute-search) will **exhaustively** search the - [Max Batch Size](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#maximum-batch-size), - [Dynamic Batching](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#dynamic-batcher), and - [Instance Group](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#instance-groups) - parameters of your model configuration - -- [Manual Brute Search](docs/config_search.md#manual-brute-search) allows you to create manual sweeps for every parameter that can be specified in the model configuration - -### Model Types - -- [Ensemble](docs/model_types.md#ensemble): Model Analyzer can help you find the optimal - settings when profiling an ensemble model - -- [BLS](docs/model_types.md#bls): Model Analyzer can help you find the optimal - settings when profiling a BLS model - -- [Multi-Model](docs/model_types.md#multi-model): Model Analyzer can help you - find the optimal settings when profiling multiple concurrent models - -- [LLM](docs/model_types.md#llm): Model Analyzer can help you - find the optimal settings when profiling Large Language Models - -### Other Features - -- [Detailed and summary reports](docs/report.md): Model Analyzer is able to generate - summarized and detailed reports that can help you better understand the trade-offs - between different model configurations that can be used for your model. - -- [QoS Constraints](docs/config.md#constraint): Constraints can help you - filter out the Model Analyzer results based on your QoS requirements. For - example, you can specify a latency budget to filter out model configurations - that do not satisfy the specified latency threshold. -

- -# Examples and Tutorials - -### **Single Model** - -See the [Single Model Quick Start](docs/quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on a simple PyTorch model. - -### **Multi Model** - -See the [Multi-model Quick Start](docs/mm_quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on two models running concurrently on the same GPU. - -### **Ensemble Model** - -See the [Ensemble Model Quick Start](docs/ensemble_quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on a simple Ensemble model. - -### **BLS Model** - -See the [BLS Model Quick Start](docs/bls_quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on a simple BLS model. -

- -# Documentation - -- [Installation](docs/install.md) -- [Model Analyzer CLI](docs/cli.md) -- [Launch Modes](docs/launch_modes.md) -- [Configuring Model Analyzer](docs/config.md) -- [Model Analyzer Metrics](docs/metrics.md) -- [Model Config Search](docs/config_search.md) -- [Model Types](docs/model_types.md) -- [Checkpointing](docs/checkpoints.md) -- [Model Analyzer Reports](docs/report.md) -- [Deployment with Kubernetes](docs/kubernetes_deploy.md) -

- -# Terminology - -Below are definitions of some commonly used terms in Model Analyzer: - -- **Model Type** - Category of model being profiled. Examples of this include single, multi, ensemble, BLS, etc.. -- **Search Mode** - How Model Analyzer explores the possible configuration space when profiling. This is either exhaustive (brute) or heuristic (quick/optuna). -- **Model Config Search** - The cross product of model type and search mode. -- **Launch Mode** - How the Triton Server is deployed and used by Model Analyzer. - -# Reporting problems, asking questions - -We appreciate any feedback, questions or bug reporting regarding this -project. When help with code is needed, follow the process outlined in -the Stack Overflow (https://stackoverflow.com/help/mcve) -document. Ensure posted examples are: - -- minimal – use as little code as possible that still produces the - same problem - -- complete – provide all parts needed to reproduce the problem. Check - if you can strip external dependency and still show the problem. The - less time we spend on reproducing problems the more time we have to - fix it - -- verifiable – test the code you're about to provide to make sure it - reproduces the problem. Remove all other problems that are not - related to your request/question. +> You are currently on the `24.12` branch which tracks under-development and unreleased features. \ No newline at end of file diff --git a/VERSION b/VERSION index 81a46d18..21998d3c 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.47.0dev +1.47.0 diff --git a/docs/bls_quick_start.md b/docs/bls_quick_start.md index 39c4c3d5..f95c2ea9 100644 --- a/docs/bls_quick_start.md +++ b/docs/bls_quick_start.md @@ -49,7 +49,7 @@ git pull origin main **1. Pull the SDK container:** ``` -docker pull nvcr.io/nvidia/tritonserver:24.11-py3-sdk +docker pull nvcr.io/nvidia/tritonserver:24.12-py3-sdk ``` **2. Run the SDK container** @@ -59,7 +59,7 @@ docker run -it --gpus 1 \ --shm-size 2G \ -v /var/run/docker.sock:/var/run/docker.sock \ -v $(pwd)/examples/quick-start:$(pwd)/examples/quick-start \ - --net=host nvcr.io/nvidia/tritonserver:24.11-py3-sdk + --net=host nvcr.io/nvidia/tritonserver:24.12-py3-sdk ``` **Important:** The example above uses a single GPU. If you are running on multiple GPUs, you may need to increase the shared memory size accordingly

diff --git a/docs/config.md b/docs/config.md index 350b3f21..8fbddbd0 100644 --- a/docs/config.md +++ b/docs/config.md @@ -153,7 +153,7 @@ cpu_only_composing_models: [ reload_model_disable: | default: false] # Triton Docker image tag used when launching using Docker mode -[ triton_docker_image: | default: nvcr.io/nvidia/tritonserver:24.11-py3 ] +[ triton_docker_image: | default: nvcr.io/nvidia/tritonserver:24.12-py3 ] # Triton Server HTTP endpoint url used by Model Analyzer client" [ triton_http_endpoint: | default: localhost:8000 ] diff --git a/docs/ensemble_quick_start.md b/docs/ensemble_quick_start.md index 2086a0b1..0ac0ba04 100644 --- a/docs/ensemble_quick_start.md +++ b/docs/ensemble_quick_start.md @@ -55,7 +55,7 @@ mkdir examples/quick-start/ensemble_add_sub/1 **1. Pull the SDK container:** ``` -docker pull nvcr.io/nvidia/tritonserver:24.11-py3-sdk +docker pull nvcr.io/nvidia/tritonserver:24.12-py3-sdk ``` **2. Run the SDK container** @@ -65,7 +65,7 @@ docker run -it --gpus 1 \ --shm-size 1G \ -v /var/run/docker.sock:/var/run/docker.sock \ -v $(pwd)/examples/quick-start:$(pwd)/examples/quick-start \ - --net=host nvcr.io/nvidia/tritonserver:24.11-py3-sdk + --net=host nvcr.io/nvidia/tritonserver:24.12-py3-sdk ``` **Important:** The example above uses a single GPU. If you are running on multiple GPUs, you may need to increase the shared memory size accordingly

diff --git a/docs/kubernetes_deploy.md b/docs/kubernetes_deploy.md index d35a1972..8b7b2011 100644 --- a/docs/kubernetes_deploy.md +++ b/docs/kubernetes_deploy.md @@ -79,7 +79,7 @@ images: triton: image: nvcr.io/nvidia/tritonserver - tag: 24.11-py3 + tag: 24.12-py3 ``` The model analyzer executable uses the config file defined in `helm-chart/templates/config-map.yaml`. This config can be modified to supply arguments to model analyzer. Only the content under the `config.yaml` section of the file should be modified. diff --git a/docs/mm_quick_start.md b/docs/mm_quick_start.md index aeb798d6..79abf198 100644 --- a/docs/mm_quick_start.md +++ b/docs/mm_quick_start.md @@ -49,7 +49,7 @@ git pull origin main **1. Pull the SDK container:** ``` -docker pull nvcr.io/nvidia/tritonserver:24.11-py3-sdk +docker pull nvcr.io/nvidia/tritonserver:24.12-py3-sdk ``` **2. Run the SDK container** @@ -58,7 +58,7 @@ docker pull nvcr.io/nvidia/tritonserver:24.11-py3-sdk docker run -it --gpus all \ -v /var/run/docker.sock:/var/run/docker.sock \ -v $(pwd)/examples/quick-start:$(pwd)/examples/quick-start \ - --net=host nvcr.io/nvidia/tritonserver:24.11-py3-sdk + --net=host nvcr.io/nvidia/tritonserver:24.12-py3-sdk ``` ## `Step 3:` Profile both models concurrently diff --git a/docs/quick_start.md b/docs/quick_start.md index 8538a1b7..8f95602c 100644 --- a/docs/quick_start.md +++ b/docs/quick_start.md @@ -49,7 +49,7 @@ git pull origin main **1. Pull the SDK container:** ``` -docker pull nvcr.io/nvidia/tritonserver:24.11-py3-sdk +docker pull nvcr.io/nvidia/tritonserver:24.12-py3-sdk ``` **2. Run the SDK container** @@ -58,7 +58,7 @@ docker pull nvcr.io/nvidia/tritonserver:24.11-py3-sdk docker run -it --gpus all \ -v /var/run/docker.sock:/var/run/docker.sock \ -v $(pwd)/examples/quick-start:$(pwd)/examples/quick-start \ - --net=host nvcr.io/nvidia/tritonserver:24.11-py3-sdk + --net=host nvcr.io/nvidia/tritonserver:24.12-py3-sdk ``` ## `Step 3:` Profile the `add_sub` model diff --git a/helm-chart/values.yaml b/helm-chart/values.yaml index caf10481..24882bf3 100644 --- a/helm-chart/values.yaml +++ b/helm-chart/values.yaml @@ -41,4 +41,4 @@ images: triton: image: nvcr.io/nvidia/tritonserver - tag: 24.11-py3 + tag: 24.12-py3 diff --git a/model_analyzer/config/input/config_defaults.py b/model_analyzer/config/input/config_defaults.py index f5b0272c..4926da85 100755 --- a/model_analyzer/config/input/config_defaults.py +++ b/model_analyzer/config/input/config_defaults.py @@ -64,7 +64,7 @@ DEFAULT_CONCURRENCY_SWEEP_DISABLE = False DEFAULT_DCGM_DISABLE = False DEFAULT_TRITON_LAUNCH_MODE = "local" -DEFAULT_TRITON_DOCKER_IMAGE = "nvcr.io/nvidia/tritonserver:24.11-py3" +DEFAULT_TRITON_DOCKER_IMAGE = "nvcr.io/nvidia/tritonserver:24.12-py3" DEFAULT_TRITON_HTTP_ENDPOINT = "localhost:8000" DEFAULT_TRITON_GRPC_ENDPOINT = "localhost:8001" DEFAULT_TRITON_METRICS_URL = "http://localhost:8002/metrics" From e585baf819d184042d50d5bda9c874421da8700f Mon Sep 17 00:00:00 2001 From: Misha Chornyi <99709299+mc-nv@users.noreply.github.com> Date: Mon, 23 Dec 2024 10:15:32 -0800 Subject: [PATCH 2/3] Update REAMDE.md and version (#953) --- README.md | 109 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 107 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index e11612ad..14b0a068 100644 --- a/README.md +++ b/README.md @@ -18,5 +18,110 @@ limitations under the License. # Triton Model Analyzer -> [!Warning] -> You are currently on the `24.12` branch which tracks under-development and unreleased features. \ No newline at end of file +Triton Model Analyzer is a CLI tool which can help you find a more optimal configuration, on a given piece of hardware, for single, multiple, ensemble, or BLS models running on a [Triton Inference Server](https://github.com/triton-inference-server/server/). Model Analyzer will also generate reports to help you better understand the trade-offs of the different configurations along with their compute and memory requirements. +

+ +# Features + +### Search Modes + +- [Optuna Search](docs/config_search.md#optuna-search-mode) **_-ALPHA RELEASE-_** allows you to search for every parameter that can be specified in the model configuration, using a hyperparameter optimization framework. Please see the [Optuna](https://optuna.org/) website if you are interested in specific details on how the algorithm functions. + +- [Quick Search](docs/config_search.md#quick-search-mode) will **sparsely** search the [Max Batch Size](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#maximum-batch-size), + [Dynamic Batching](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#dynamic-batcher), and + [Instance Group](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#instance-groups) spaces by utilizing a heuristic hill-climbing algorithm to help you quickly find a more optimal configuration + +- [Automatic Brute Search](docs/config_search.md#automatic-brute-search) will **exhaustively** search the + [Max Batch Size](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#maximum-batch-size), + [Dynamic Batching](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#dynamic-batcher), and + [Instance Group](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#instance-groups) + parameters of your model configuration + +- [Manual Brute Search](docs/config_search.md#manual-brute-search) allows you to create manual sweeps for every parameter that can be specified in the model configuration + +### Model Types + +- [Ensemble](docs/model_types.md#ensemble): Model Analyzer can help you find the optimal + settings when profiling an ensemble model + +- [BLS](docs/model_types.md#bls): Model Analyzer can help you find the optimal + settings when profiling a BLS model + +- [Multi-Model](docs/model_types.md#multi-model): Model Analyzer can help you + find the optimal settings when profiling multiple concurrent models + +- [LLM](docs/model_types.md#llm): Model Analyzer can help you + find the optimal settings when profiling Large Language Models + +### Other Features + +- [Detailed and summary reports](docs/report.md): Model Analyzer is able to generate + summarized and detailed reports that can help you better understand the trade-offs + between different model configurations that can be used for your model. + +- [QoS Constraints](docs/config.md#constraint): Constraints can help you + filter out the Model Analyzer results based on your QoS requirements. For + example, you can specify a latency budget to filter out model configurations + that do not satisfy the specified latency threshold. +

+ +# Examples and Tutorials + +### **Single Model** + +See the [Single Model Quick Start](docs/quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on a simple PyTorch model. + +### **Multi Model** + +See the [Multi-model Quick Start](docs/mm_quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on two models running concurrently on the same GPU. + +### **Ensemble Model** + +See the [Ensemble Model Quick Start](docs/ensemble_quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on a simple Ensemble model. + +### **BLS Model** + +See the [BLS Model Quick Start](docs/bls_quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on a simple BLS model. +

+ +# Documentation + +- [Installation](docs/install.md) +- [Model Analyzer CLI](docs/cli.md) +- [Launch Modes](docs/launch_modes.md) +- [Configuring Model Analyzer](docs/config.md) +- [Model Analyzer Metrics](docs/metrics.md) +- [Model Config Search](docs/config_search.md) +- [Model Types](docs/model_types.md) +- [Checkpointing](docs/checkpoints.md) +- [Model Analyzer Reports](docs/report.md) +- [Deployment with Kubernetes](docs/kubernetes_deploy.md) +

+ +# Terminology + +Below are definitions of some commonly used terms in Model Analyzer: + +- **Model Type** - Category of model being profiled. Examples of this include single, multi, ensemble, BLS, etc.. +- **Search Mode** - How Model Analyzer explores the possible configuration space when profiling. This is either exhaustive (brute) or heuristic (quick/optuna). +- **Model Config Search** - The cross product of model type and search mode. +- **Launch Mode** - How the Triton Server is deployed and used by Model Analyzer. + +# Reporting problems, asking questions + +We appreciate any feedback, questions or bug reporting regarding this +project. When help with code is needed, follow the process outlined in +the Stack Overflow (https://stackoverflow.com/help/mcve) +document. Ensure posted examples are: + +- minimal – use as little code as possible that still produces the + same problem + +- complete – provide all parts needed to reproduce the problem. Check + if you can strip external dependency and still show the problem. The + less time we spend on reproducing problems the more time we have to + fix it + +- verifiable – test the code you're about to provide to make sure it + reproduces the problem. Remove all other problems that are not + related to your request/question. From c11b0401c9de0eaa1117d3813d5421bf76054218 Mon Sep 17 00:00:00 2001 From: Misha Chornyi Date: Mon, 6 Jan 2025 09:50:39 -0800 Subject: [PATCH 3/3] Update branch post 24.12 --- Dockerfile | 4 ++-- README.md | 8 ++++++++ VERSION | 2 +- 3 files changed, 11 insertions(+), 3 deletions(-) diff --git a/Dockerfile b/Dockerfile index fc2f693d..fb48cb4b 100644 --- a/Dockerfile +++ b/Dockerfile @@ -15,8 +15,8 @@ ARG BASE_IMAGE=nvcr.io/nvidia/tritonserver:24.12-py3 ARG TRITONSDK_BASE_IMAGE=nvcr.io/nvidia/tritonserver:24.12-py3-sdk -ARG MODEL_ANALYZER_VERSION=1.47.0dev -ARG MODEL_ANALYZER_CONTAINER_VERSION=24.12dev +ARG MODEL_ANALYZER_VERSION=1.48.0dev +ARG MODEL_ANALYZER_CONTAINER_VERSION=25.01dev FROM ${TRITONSDK_BASE_IMAGE} as sdk FROM $BASE_IMAGE diff --git a/README.md b/README.md index 14b0a068..58214f8c 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,14 @@ limitations under the License. # Triton Model Analyzer +> [!Warning] +> +> ##### LATEST RELEASE +> +> You are currently on the `main` branch which tracks under-development progress towards the next release.
+> The latest release of the Triton Model Analyzer is 1.47.0 and is available on branch +> [r24.12](https://github.com/triton-inference-server/model_analyzer/tree/r24.12). + Triton Model Analyzer is a CLI tool which can help you find a more optimal configuration, on a given piece of hardware, for single, multiple, ensemble, or BLS models running on a [Triton Inference Server](https://github.com/triton-inference-server/server/). Model Analyzer will also generate reports to help you better understand the trade-offs of the different configurations along with their compute and memory requirements.

diff --git a/VERSION b/VERSION index 21998d3c..5fd5d860 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.47.0 +1.48.0dev