Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding support for concurrency formula as an option in Optuna search #885

Merged
merged 2 commits into from
May 29, 2024

Conversation

nv-braf
Copy link
Contributor

@nv-braf nv-braf commented May 28, 2024

Gives the user the option to use our concurrency formula rather than having Optuna search the full concurrency range.

By default we allow Optuna to search the concurrency space. This gives the user the option of using our concurrency formula (2 * instance_count * batch_size) instead, which will reduce the search space and should improve the search time.

@nv-braf nv-braf changed the title Adding support for concurrency formula as an option in optuna search Adding support for concurrency formula as an option in Optuna search May 28, 2024
@nv-braf nv-braf requested review from debermudez and tgerdesnv May 29, 2024 16:29
@nv-braf nv-braf force-pushed the config-option-for-concurrency-searching branch from c2fb047 to 3447054 Compare May 29, 2024 17:40
tests/test_cli.py Show resolved Hide resolved

self.assertEqual(model_config.to_dict()["name"], self._test_config_dict["name"])

# These values are the result of using a fixed seed of 100
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice note!

analyzer = Analyzer(config, MagicMock(), MagicMock(), MagicMock())
analyzer._populate_search_parameters()

concurrency = analyzer._search_parameters["add_sub"].get_parameter(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This wont error out?
get_parameter is designed to return None when a parameter isnt found?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it will return None when the key is not found.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well done!

@nv-braf nv-braf merged commit 16ef62e into optuna-search-mode May 29, 2024
3 checks passed
nv-braf added a commit that referenced this pull request Jun 6, 2024
…885)

* Fixing merge confilct

* Adding --use-concurrency-formula to unit testing
nv-braf added a commit that referenced this pull request Jun 7, 2024
* Adding cli option for optuna search (#867)

* Adding cli option for optuna search

* Changed RCS description

* Class to hold info about parameters (#868)

* Initial code for ConfigParameters class

* Fixing codeql issue

* Fixes based on review

* Connect up parameter description class (#869)

* Added hooks for creating search parameters with some basic unit testing

* Adding more unit testing

* Cleaning up codeql

* Adding story ref for TODO

* Changes based on review comments

* Refactored ConfigParameters

* Renaming to SearchParameter(s)

* Moving unit testing into SearchParameters test class

* Fix codeql issues

* Creating Optuna RCG factory (#878)

* Creating optuna RCG factory

* fixing codeql issues

* Removing metrics manager

* Fixing mypy failure

* Optuna Search Class (#877)

* Base Optuna class plus unit testing

* codeql fixes

* more codeql fixes

* Removing metrics manager

* Removing metrics manager from Optuna RCG unit test

* Removing client from quick/optuna RCGs

* Changing gpus to gpu_count in quick/optuna RCGs

* Removing magic number

* Fixing codeql issue

* Fixing optuna version

* Adding todo comment about client batch size support

* Using SearchParameters in OptunaRCG (#881)

* Using SearchParameters in OptunaRCG

* Fixing search parameter unit tests

* Removing debug line

* Changes based on PR

* Adding call for default parameters

* Added todo for dynamic batching

* Add Percentage Search Space to Optuna (#882)

* Added method for calculating total possible configurations

* Added min/max percentage of search space to CLI

* Connected up in optuna RCG

* Added in support to cap optuna search based on a strict number of trials (#884)

* Adding support for concurrency formula as an option in Optuna search (#885)

* Fixing merge confilct

* Adding --use-concurrency-formula to unit testing

* Add Debug info to Optuna (#889)

* Adding debug info + bug fixes

* Fixes based on PR

* Optuna Early Exit (#890)

* Add logic to enable early exit along with CLI hooks.

* Changes based on PR

* Check that model supports dynamic batching when creating param_combo (#891)

* Adding option to disable concurrency sweeping (#893)

* Adding support for client batch size (#892)

* Adding support for client batch size

* Fixes based on PR

* Removing redundant keys()

* Fixing codeQL issue

* Attempt to fix unittest issue

* Removing 3.8 testing
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants