From 60e921a601dc3788505e686295e1911e840d1a70 Mon Sep 17 00:00:00 2001 From: Misha Chornyi Date: Fri, 24 May 2024 16:52:18 -0700 Subject: [PATCH 1/2] Update README and versions for 1.40.0 / 24.05 --- README.md | 103 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 102 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index a0760fd50..d456e25c5 100644 --- a/README.md +++ b/README.md @@ -19,4 +19,105 @@ limitations under the License. # Triton Model Analyzer > [!Warning] -> You are currently on the `24.05` branch which tracks under-development and unreleased features. \ No newline at end of file +> +> ##### LATEST RELEASE +> +> You are currently on the `main` branch which tracks under-development progress towards the next release.
+> The latest release of the Triton Model Analyzer is 1.39.0 and is available on branch +> [r24.04](https://github.com/triton-inference-server/model_analyzer/tree/r24.04). + +Triton Model Analyzer is a CLI tool which can help you find a more optimal configuration, on a given piece of hardware, for single, multiple, ensemble, or BLS models running on a [Triton Inference Server](https://github.com/triton-inference-server/server/). Model Analyzer will also generate reports to help you better understand the trade-offs of the different configurations along with their compute and memory requirements. +

+ +# Features + +### Search Modes + +- [Quick Search](docs/config_search.md#quick-search-mode) will **sparsely** search the [Max Batch Size](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#maximum-batch-size), + [Dynamic Batching](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#dynamic-batcher), and + [Instance Group](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#instance-groups) spaces by utilizing a heuristic hill-climbing algorithm to help you quickly find a more optimal configuration + +- [Automatic Brute Search](docs/config_search.md#automatic-brute-search) will **exhaustively** search the + [Max Batch Size](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#maximum-batch-size), + [Dynamic Batching](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#dynamic-batcher), and + [Instance Group](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#instance-groups) + parameters of your model configuration + +- [Manual Brute Search](docs/config_search.md#manual-brute-search) allows you to create manual sweeps for every parameter that can be specified in the model configuration + +### Model Types + +- [Ensemble Model Search](docs/config_search.md#ensemble-model-search): Model Analyzer can help you find the optimal + settings when profiling an ensemble model, utilizing the [Quick Search](docs/config_search.md#quick-search-mode) algorithm + +- [BLS Model Search](docs/config_search.md#bls-model-search): Model Analyzer can help you find the optimal + settings when profiling a BLS model, utilizing the [Quick Search](docs/config_search.md#quick-search-mode) algorithm + +- [Multi-Model Search](docs/config_search.md#multi-model-search-mode): Model Analyzer can help you + find the optimal settings when profiling multiple concurrent models, utilizing the [Quick Search](docs/config_search.md#quick-search-mode) algorithm + +- [LLM Search](docs/config_search.md#llm-search-mode): Model Analyzer can help you + find the optimal settings when profiling large language models, utilizing the [Quick Search](docs/config_search.md#quick-search-mode) algorithm + +### Other Features + +- [Detailed and summary reports](docs/report.md): Model Analyzer is able to generate + summarized and detailed reports that can help you better understand the trade-offs + between different model configurations that can be used for your model. + +- [QoS Constraints](docs/config.md#constraint): Constraints can help you + filter out the Model Analyzer results based on your QoS requirements. For + example, you can specify a latency budget to filter out model configurations + that do not satisfy the specified latency threshold. +

+ +# Examples and Tutorials + +### **Single Model** + +See the [Single Model Quick Start](docs/quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on a simple PyTorch model. + +### **Multi Model** + +See the [Multi-model Quick Start](docs/mm_quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on two models running concurrently on the same GPU. + +### **Ensemble Model** + +See the [Ensemble Model Quick Start](docs/ensemble_quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on a simple Ensemble model. + +### **BLS Model** + +See the [BLS Model Quick Start](docs/bls_quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on a simple BLS model. +

+ +# Documentation + +- [Installation](docs/install.md) +- [Model Analyzer CLI](docs/cli.md) +- [Launch Modes](docs/launch_modes.md) +- [Configuring Model Analyzer](docs/config.md) +- [Model Analyzer Metrics](docs/metrics.md) +- [Model Config Search](docs/config_search.md) +- [Checkpointing](docs/checkpoints.md) +- [Model Analyzer Reports](docs/report.md) +- [Deployment with Kubernetes](docs/kubernetes_deploy.md) +

+ +# Reporting problems, asking questions + +We appreciate any feedback, questions or bug reporting regarding this +project. When help with code is needed, follow the process outlined in +the Stack Overflow (https://stackoverflow.com/help/mcve) +document. Ensure posted examples are: + +- minimal – use as little code as possible that still produces the + same problem + +- complete – provide all parts needed to reproduce the problem. Check + if you can strip external dependency and still show the problem. The + less time we spend on reproducing problems the more time we have to + fix it + +- verifiable – test the code you're about to provide to make sure it + reproduces the problem. Remove all other problems that are not + related to your request/question. From d26d606cc56787657545e3137364768008c6162b Mon Sep 17 00:00:00 2001 From: Misha Chornyi Date: Fri, 24 May 2024 16:59:31 -0700 Subject: [PATCH 2/2] Update README and versions for 1.40.0 / 24.05 --- README.md | 22 +++++++--------------- 1 file changed, 7 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index d456e25c5..bf92b33a4 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@