Merge pull request #41 from automl/development

Development

.travis.yml

@@ -12,6 +12,7 @@ jobs:
  env:
  - RUN_TESTS="true"
  - USE_SINGULARITY="true"
+ - RUN_CODECOV="true"
  - python: "3.7"
  env: RUN_CODESTYLE="true"
  - python: "3.7"

README.md

@@ -8,7 +8,7 @@ HPOlib2 is a library for hyperparameter optimization and black-box optimization
 
 ## In 4 lines of code
 
-Run a random configuration within a singularity container (requires singularity dependencies: `pip install .[singularity]`)
+Run a random configuration within a singularity container
 ```python
 from hpolib.container.benchmarks.ml.xgboost_benchmark import XGBoostBenchmark
 b = XGBoostBenchmark(task_id=167149, container_source='library://phmueller/automl', rng=1)
@@ -49,7 +49,7 @@ run the following:
 ```
 git clone https://github.com/automl/HPOlib2.git
 cd HPOlib2 
-pip install .[singularity]
+pip install .
 ```
 
 **Note:** This does not install *singularity (version 3.5)*. Please follow the steps described here: [user-guide](https://sylabs.io/guides/3.5/user-guide/quick_start.html#quick-installation-steps). 

changelog.md

@@ -1,3 +1,9 @@
+# 0.0.4
+ * improve test coverage
+ * update HowToAddANewBenchmark.md
+ * Add SVM benchmark working on OpenML data
+ * Add nasbench201 benchmark
+
 # 0.0.3
  * improve forwarding exceptions in containerized benchmarks
  * allow to set debug level with env variable

ci_scripts/script.sh

@@ -1,14 +1,24 @@
 #!/usr/bin/env sh
 
 if [[ "$RUN_TESTS" == "true" ]]; then
- if [[ "$USE_SINGULARITY" == "true" ]]; then
- echo "Run tests with singularity support"
- # Create the coverage report for the singularity example, since it covers more tests.
+ if [[ "$RUN_CODECOV" == "true" ]]; then
+ echo "Run tests with code coverage"
  pytest -sv --cov=hpolib tests/
+ exit_code=$?
+
+ echo "Run code coverage"
  codecov
  else
- echo "Run tests without singularity support"
+ echo "Run tests without code coverage"
  pytest -sv tests/
+ exit_code=$?
+ fi
+
+ if [[ "$exit_code" -eq 0 ]]; then
+ echo "All test have passed."
+ else
+ echo "Some Tests have failed."
+ exit 1
  fi
 fi
 

codecov.yml

@@ -0,0 +1,25 @@
+codecov:
+ require_ci_to_pass: yes
+
+coverage:
+ precision: 2
+ round: down
+ range: "70...100"
+
+parsers:
+ gcov:
+ branch_detection:
+ conditional: yes
+ loop: yes
+ method: no
+ macro: no
+
+comment:
+ layout: "reach,diff,flags,tree"
+ behavior: default
+ require_changes: no
+
+ignore:
+ - "hpolib/benchmarks"
+ - "hpolib/util/dependencies.py"
+ - "hpolib/util/example_utils.py"

examples/container/tabular_benchmark_example.py

@@ -11,7 +11,7 @@
 hpolibrc. A second call, will first look into the data directory, if the container is already available, so it will not
 be downloaded twice.
 
-Please install the necessary dependencies via ``pip install .[singularity]`` and singularity (v3.5).
+Please install the necessary dependencies via ``pip install .`` and singularity (v3.5).
 https://sylabs.io/guides/3.5/user-guide/quick_start.html#quick-installation-steps
 """
 

examples/container/xgboost_with_container.py

@@ -12,7 +12,7 @@
 https://sylabs.io/guides/3.1/user-guide/quick_start.html#quick-installation-steps
 
 Furthermore, make sure to install the right dependencies for the hpolib via:
-``pip3 install .[singularity]``.
+``pip3 install .``.
 """
 
 import argparse

examples/w_optimizer/cartpole_bohb.py

@@ -5,7 +5,7 @@
 This example shows the usage of an Hyperparameter Tuner, such as BOHB on the cartpole benchmark.
 BOHB is a combination of Bayesian optimization and Hyperband.
 
-Please install the necessary dependencies via ``pip install .[singularity]`` and singularity (v3.5).
+Please install the necessary dependencies via ``pip install .`` and singularity (v3.5).
 https://sylabs.io/guides/3.5/user-guide/quick_start.html#quick-installation-steps
 
 """

examples/w_optimizer/cartpole_hyperband.py

@@ -5,7 +5,7 @@
 This example shows the usage of an Hyperparameter Tuner, such as SMAC on the cartpole benchmark.
 We use SMAC with Hyperband.
 
-Please install the necessary dependencies via ``pip install .[singularity]`` and singularity (v3.5).
+Please install the necessary dependencies via ``pip install .`` and singularity (v3.5).
 https://sylabs.io/guides/3.5/user-guide/quick_start.html#quick-installation-steps
 """
 import logging

examples/w_optimizer/cartpole_succesive_halving.py

@@ -5,7 +5,7 @@
 This example shows the usage of an Hyperparameter Tuner, such as SMAC on the cartpole benchmark.
 We use SMAC with Successive Halving.
 
-Please install the necessary dependencies via ``pip install .[singularity]`` and singularity (v3.5).
+Please install the necessary dependencies via ``pip install .`` and singularity (v3.5).
 https://sylabs.io/guides/3.5/user-guide/quick_start.html#quick-installation-steps
 """
 import logging

extra_requirements/svm.json

@@ -0,0 +1,3 @@
+{
+ "svm": ["pandas>=0.22.2,<0.24.2","openml==0.10.2","scikit-learn>=0.18.1"]
+}

extra_requirements/xgboost.json

@@ -1,3 +1,3 @@
 {
- "xgboost": ["xgboost==0.90","pandas>=0.22.2,<0.24.2","openml==0.10.2"]
+ "xgboost": ["xgboost==0.90","pandas>=0.22.2,<0.24.2","openml==0.10.2","scikit-learn>=0.18.1"]
 }

hpolib/HowToAddANewBenchmark.md

@@ -2,9 +2,8 @@
 
 ## Create a local benchmark
 
- 1. Clone Hpolib2, switch to the development branch and create your own branch, then install hpolib2. 
-Note that with `pip install .`
-
+Clone Hpolib2, switch to the development branch and create your own branch, then install hpolib2. 
+with `pip install .`
 ```bash
 git clone https://github.com/automl/HPOlib2.git
 cd HPOlib2
@@ -14,33 +13,59 @@ git checkout newBenchmark
 pip install .
 ```
 
- 2. Implement your new benchmark `hpolib/benchmarks/<type>/<name>` inheriting from the base class 
- `AbstractBenchmark` in `hpolib.abstract_benchmark`
-
- 3. Collect **all additional Python** and **non-Python** dependencies while doing this. 
+Then: 
+
+ 1. Implement your new benchmark `hpolib/benchmarks/<type>/<name>` inheriting from the base class 
+ `AbstractBenchmark` in `hpolib.abstract_benchmark`. Your benchmark should implement `__init__()`, 
+ `get_configuration_space()`, `get_fidelity_space()`, `objective_function()` and `objective_function_test()`.
+ A good example for this can be found in `hpolib/benchmarks/ml/xgboost_benchmark.py`
+ 3. If your benchmarks needs a dataset (e.g. for training a ml model), please also implement a DataManager, see e.g.
+ `hpolib/util/openml_data_manager.py` with a `load()` method that downloads data once and reuses it for further calls.
+ 4. Collect all **additional Python** and **non-Python** dependencies while doing this. 
  Consider fixing the version of each dependency to maintain reproducibility.
- 4. Add dependencies to PiPy in a new file to `/extra_requirements`
- 5. Add the remaining dependencies or steps necessary to run your benchmark in the docstring of your benchmark class.
+ 5. Add dependencies to PIPy in a new file to `/extra_requirements`
+ 6. Add the remaining dependencies or steps necessary to run your benchmark in the docstring of your benchmark class
+ (see, e.g. `hpolib/benchmarks/nas/nasbench_101.py`).
+ 7. Verify that everything works with, e.g.
+
+```python
+from hpolib.benchmarks.<type>.<newbenchmark> import <NewBenchmark>
+b = <NewBenchmark>(<some_args>, rng=1)
+config = b.get_configuration_space(seed=1).sample_configuration()
+result_dict = b.objective_function(configuration=config, rng=1)
+print(result_dict)
+```
+
+**Note:** Ideally, your benchmark behaves deterministic given a seed!
+
+Now, you can create a PR marked as [WIP] and proceed with building a containerized version. 
 
 ## Create a containerized benchmark
 
  1. Create a container benchmark class in `hpolib/container/benchmarks/<type>/<name>` inheriting from the 
- base class `AbstractBenchmarkClient` in `hpolib.container.client_abstract_benchmark` (note: this is just copy/paste from existing classes)
+ base class `AbstractBenchmarkClient` in `hpolib.container.client_abstract_benchmark`. 
+ Note: this are just a few lines of code, see, e.g. `hpolib/container/benchmarks/ml/xgboost_benchmark.py`)
  2. Copy `hpolib/container/recipes/Singularity.template` to `hpolib/container/recipes/<type>/name`
- 3. Create a pull request marked as [WIP]
+ 3. Modify the recipe and add your **additional Python** and **non-Python** dependencies collected above. 
+ 3. Test your container locally (see below)
+
+Now, you can update your PR and let us know, so we can upload the container to Sylabs. Thanks.
 
 ## How to test your container locally
 
- 1. `cd hpolib/container/benchmarks/recipes/<type>` and change to following line in the recipe:
+ 1. `cd hpolib/container/benchmarks/recipes/<type>` and change to following lines in the recipe:
  ```bash
+ && git clone https://github.com/automl/HPOlib2.git \
+ && cd HPOlib2 \
  && git checkout development \
-```
- to point to the branch where your pull request is, e.g. `newBenchmark`
+ ```
+ to point to the branch/repo where your fork is on, e.g. `newBenchmark`.
+
  2. Run `sudo singularity build <newBenchmark> Singularity.<newBenchmark>`
- 3. Verify that everything works with
- ```python
-from hpolib.container.benchmarks.<type>.new_benchmark import newBenchmark
-b = newBenchmark(container_source="./", container_name="newBenchmark")
+ 3. Verify that everything works with:
+
+```python
+from hpolib.container.benchmarks.<type>.<newbenchmark> import <NewBenchmark>
+b = <NewBenchmark>(container_source="./", container_name="newBenchmark")
 res = b.objective_function(configuration=b.get_configuration_space(seed=1).sample_configuration())
 ```
- 4. Finalize your pull request and let us know, so we can upload the container to Sylabs. Thanks.

hpolib/__version__.py

@@ -1 +1 @@
-__version__ = '0.0.3'
+__version__ = '0.0.4'