Optuna CI Testing

[nv-braf]

Adds four tests to protect Optuna:

• L0_optuna_search (single model)
• L0_optuna_bls
• L0_optuna_ensemble
• L0_optuna_multi_model

To make the review easier:

• These are all based on their quick search counterparts, with the only differences being:
  • Optuna is called (instead of quick) as the --run-config-search-mode in test.sh
  • Expected number of measurements changed, based on min/max % of search space, in check_results.py

The CI with passing tests can be found at: https://gitlab-master.nvidia.com/dl/dgx/tritonmodelanalyzer/-/pipelines/16546091

[dyastremsky]

Splendid work here! Generally very clean code. Appreciated the clean-up and loved a lot of choices like using safe loading for the config files.

My comments apply across files, so if you address my comments in one place, please check across files to make sure it's fixed everywhere. I think the biggest thing though is that a lot of the files seem like duplicates. If there are slight differences, e.g. in the check_results files' number of trials, those can still be put into a shared file/folder with slightly different parameters that they receive. That'd make updating, maintaining, and documenting them easier.

I think even the test.sh files might be able to combined into one large test.sh file or have a base script which you import or run, just changing the MA command or variables a bit. With those changes, the check_results.py and test_config_generator.py files would be mostly eliminated except for one copy and the test.sh files would be pretty minimal. That'd make reading and updating them easier.

[dyastremsky]

Brian confirmed that this is the current approach for MA testing due to the lengthy CI time. If MA has a faster CI in the future, we can optimize out the shared pieces while still allowing these tests to run in parallel in the CI.