Tip
For more details, please check out the following documents as well!
When we run parallel optimization experiments using tabular or surrogate benchmarks, each evaluation must be ordered based on the runtime that each configuration, in reality, takes. However, the evaluation of tabular or surrogate benchmarks, by design, does not take long. For this reason, the timing of each configuration taken into account must be ordered as if we evaluated each configuration.
 |
In this package, we automatically sort out this problem by pending to pass the hyperparameter configurations to be evaluated internally, and in turn, we obtain the right order of each hyperparameter configuration to be evaluated. If the optimizer interface is the ask-and-tell interface, users can pass the optimizer to a simulator directly and the simulator automatically performs the optimization loop as if function calls are run in parallel.
 |
| Arguments | What Wrapper | Function Call | Requirements | Benefits | Downsides |
|---|---|---|---|---|---|
| Default | Function | Parallel | Optimizer spawns child threads or processes (e.g. DEHB and SMAC3) | No need to change the optimizer interface and reproduce exactly how optimizers run | Could be very slow, unstable, and memory-intensive with a large n_workers |
launch_multiple_wrappers_from_user_side=True |
Function | Parallel | Optimizer uses Single-Program Multiple-Data (SPMD) such as MPI, or file-based or server-based synchronization (e.g. NePS and BOHB) | Same above | Same above |
ask_and_tell=True |
Function and Optimizer | *Sequential | Optimizer must take the ask-and-tell interface (see example) | Fast, stable, and memory-efficient even with a large n_workers |
Force the ask-and-tell interface, may unexpectedly ignore the memory bottleneck that could be caused by parallel runs, and not reproduce the random seed effect |
* It runs function call sequentially, but function calls are internally processed as if they are run in parallel.
While users do not have to change the interface of optimizers for ask_and_tell=False and only need to change the interface of objective function, users may need to change the interface of both for ask_and_tell=True.
In principle, ask_and_tell=True requires optimizers to be the ask-and-tell interface.
In exchange for the strict constraint, it stabilizes the simulation.
Note
Our wrapper assumes that none of the workers will not die and any additional workers will not be added after the initialization.
Therefore, if any workers die, our current wrapper hangs and keeps warning except we provide max_waiting_time for the instantiation.
Furthermore, our package cannot be run on Windows OS because the Python module fcntl is not supported on Windows OS.
Although our package supports MacOS, it is advisable to use Linux system.
The installation is easily done by pip-install:
$ pip install mfhpo-simulatorThe requirements are:
- Unix system
- Python 3.8 or later
The dependencies of this package are only numpy and ujson.
Basic usage is available at Usage and the Difference between with or without Our Wrapper and Tests Using Various Open Source Optimizers
Please use the following formats for the citation of this repository:
@article{watanabe2023mfo-simulator,
title = {{P}ython Wrapper for Simulating Multi-Fidelity Optimization on {HPO} Benchmarks without Any Wait},
author = {S. Watanabe},
journal = {arXiv:2305.17595},
year = {2023},
}@article{watanabe2023mfo-simulator,
title = {Fast Benchmarking of Asynchronous Multi-Fidelity Optimization on Zero-Cost Benchmarks},
author = {S. Watanabe, N. Mallik, E. Bergman, F. Hutter},
journal = {arXiv:2403.01888},
year = {2024},
}