Skip to content

Latest commit

 

History

History
176 lines (134 loc) · 8.51 KB

config_search.md

File metadata and controls

176 lines (134 loc) · 8.51 KB
Raw

Model Config Search

Model Analyzer's profile subcommand supports multiple modes when searching to find the best model configuration.

  • Brute is the default, and will do a brute-force sweep of the cross product of all possible configurations
  • Quick will use heuristics to try to find the optimal configuration much quicker than brute, and can be enabled via --run-config-search-mode quick

Brute Search Mode

Model Analyzer's brute search mode will do a brute-force sweep of the cross product of all possible configurations. You can Manually provide model_config_parameters to tell Model Analyzer what to sweep over, or you can let it Automatically sweep through configurations expected to have the highest impact on performance for Triton models.

Automatic Brute Search

Automatic configuration search is the default behavior when running Model Analyzer without manually specifying what values to search. The parameters that are automatically searched are max_batch_size and instance_group. Additionally, dynamic_batching will be enabled if it is legal to do so.

An example model analyzer config that performs automatic config search looks like below:

model_repository: /path/to/model/repository/

profile_models:
  - model_1
  - model_2

In the default mode, automatic config search will try values 1 through 5 for instance_group. The maximum value can be changed using the run_config_search_max_instance_count key in the Model Analyzer Config. For each instance_group, Model Analyzer will sweep values 1 through 128 increasing exponentially (i.e. 1, 2, 4, 8, ...) for max_batch_size. The start and end values can be changed using run_config_search_min_model_batch_size and run_config_search_max_model_batch_size. Dynamic_batching will be enabled for all model configs generated using automatic search.

For each model config that is generated in automatic search, Model Analyzer will gather data for concurrency values 1 through 1024 increased exponentially (i.e. 1, 2, 4, 8, ...). The maximum value can be configured using the run_config_search_max_concurrency key in the Model Analyzer Config.

An example config that limits the search space used by Model Analyzer is described below:

model_repository: /path/to/model/repository/

run_config_search_max_instance_count: 3
run_config_search_max_concurrency: 8
profile_models:
  - model_1
  - model_2

If any model_config_parameters are specified for a model, it will disable automatic searching of model configs and will only search within the values specified. If concurrency is specified then only those values will be tried instead of the default concurrency sweep. If both concurrency and model_config_parameters are specified, automatic config search will be disabled.

For example, the config specified below will only automatically sweep through the model_config_parameters that was described above:

model_repository: /path/to/model/repository/

profile_models:
  model_1:
    concurrency: 1,2,3,128

The config described below will only sweep through different values for concurrency:

model_repository: /path/to/model/repository/

profile_models:
  model_1:
    model_config_parameters:
      instance_group:
        - kind: KIND_GPU
          count: [1, 2]

Important Note about Remote Mode

In the remote mode, model_config_parameters are always ignored because Model Analyzer has no way of accessing the model repository of the remote Triton Server. In this mode, only concurrency values can be swept.

Manual Brute Search

In addition to the automatic config search, Model Analyzer supports a manual config search mode. To enable this mode, you can specify model_config_parameters to sweep through, or set --run-config-search-disable

Unlike automatic mode, this mode is not limited to max_batch_size, dynamic_batching, and instance_count config parameters. Using manual config search, you can create custom sweeps for every parameter that can be specified in the model configuration. Model Analyzer only checks the syntax of the model_config_parameters that is specified and cannot guarantee that the configuration that is generated is loadable by Triton.

You can also specify concurrency ranges to sweep through. If unspecified, it will automatically sweep concurrency for every model configuration (unless --run-config-search-disable is set, in which case it will only use the concurrency value of 1)

An example Model Analyzer Config that performs manual sweeping looks like below:

model_repository: /path/to/model/repository/

run_config_search_disable: True
profile_models:
  model_1:
    model_config_parameters:
      max_batch_size: [6, 8]
      dynamic_batching:
        max_queue_delay_microseconds: [200, 300]
      instance_group:
        - kind: KIND_GPU
          count: [1, 2]

In this mode, Model Analyzer can sweep through every Triton model configuration parameter available. For a complete list of parameters allowed under model_config_parameters, refer to the Triton Model Configuration. It is your responsibility to make sure that the sweep configuration specified works with your model. For example, in the above config, if we change [6, 8] as the range for the max_batch_size to [1], it will no longer be a valid Triton Model Configuration.

The configuration sweep described above, will sweep through 8 configs = (2 max_batch_size) * (2 max_queue_delay_microseconds) * (2 instance_group) values.

Examples of Additional Model Config Parameters

As mentioned in the previous section, manual configuration search allows you to sweep on every parameter that can be specified in Triton model configuration. In this section, we describe some of the parameters that might be of interest for manual sweep:

Quick Search Mode

Quick search can be enabled by adding the parameter --run-config-search-mode quick to the CLI.

It uses a hill climbing algorithm to search the configuration space, looking for the maximal objective value within the specified constraints. In the majority of cases this will find greater than 95% of the maximum objective value (that could be found using a brute force search), while needing to search less than 10% of the configuration space.

After it has found the best config(s), it will then sweep the top-N configurations found (specified by --num-configs-per-model) over the default concurrency range before generation of the summary reports.