Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Arxiv paper update for tabular recollection #157

Open
wants to merge 160 commits into
base: develop
Choose a base branch
from
Open

Arxiv paper update for tabular recollection #157

wants to merge 160 commits into from

Conversation

Neeratyoy
Copy link
Collaborator

Main changes/updates:

  • The raw ML benchmarks API now offers the option to record Learning curves for a specified iteration interval
  • The tabular ML benchmarks have been updated (LR also contains LCs)
  • Redundant files removes
  • Container versions updated

@Neeratyoy
Adding sample RF space for tabular collection design
b2155dd
@Neeratyoy
Placeholder SVM benchmark to interface tabular data collection
ce405e6
@Neeratyoy
Writing common ML benchmark class for tabular collection
2ef3af8
@Neeratyoy
Adding placeholder for HistGradientBoostedClassifier
61b6963
@Neeratyoy
Minor code cleaning
a5d0217
@Neeratyoy
Reformatting output dict + option to add more metrics
3def203
@Neeratyoy
Removing redundant import
750cc7d
@Neeratyoy
Decoupling storage of costs for each metric
e7665e6
@Neeratyoy
Including test scores in objective
47fe4cd
@Neeratyoy
Documenting the structure of information in each fn eval.
2d085ec
@Neeratyoy
Some decisions on lower bound for subsample fidelity
2da9d5c
@Neeratyoy
AbstractBenchmark update for fidelity option + including XGBoost
751d2e9
@Neeratyoy
Adding sample RF space for tabular collection design
3f84afb
@Neeratyoy
Placeholder SVM benchmark to interface tabular data collection
09b296a
@Neeratyoy
Writing common ML benchmark class for tabular collection
af4f593
@Neeratyoy
Adding placeholder for HistGradientBoostedClassifier
df2462d
@Neeratyoy
Minor code cleaning
4d1d2d6
@Neeratyoy
Reformatting output dict + option to add more metrics
299e592
@Neeratyoy
Removing redundant import
c46321d
@Neeratyoy
Decoupling storage of costs for each metric
17f6634
@Neeratyoy
Including test scores in objective
7de891f
@Neeratyoy
Documenting the structure of information in each fn eval.
ec316c3
@Neeratyoy
Some decisions on lower bound for subsample fidelity
e7f69b9
@Neeratyoy
AbstractBenchmark update for fidelity option + including XGBoost
edb3e7f
@Neeratyoy
Merge branch 'thesis-paper' of https://github.com/Neeratyoy/HPOBench …
642027b 
…into thesis-paper
@Neeratyoy
Option to load data splits from disk
9e907e6
@Neeratyoy
Reordering data load to work for different cases
f0d4f36
@Neeratyoy
Updating source of SVM HP range
dbeae7c
@Neeratyoy
Adding Tabular Benchmark class
f277a2e
@Neeratyoy
Adding TabularBenchmark interface + easy import
60d5646
PhMueller and others added 15 commits May 5, 2022 15:01
@PhMueller
Merge remote-tracking branch 'fork_philipp/mo_benchmarks' into mo_ben…
eda5936 
…chmarks
@PhMueller
Fix Abstract Benchmark tests
c5f0494
@PhMueller
UPDATE changelog.md
34ccb66
@noorawad
Merge pull request #145 from PhMueller/mo_benchmarks
9c3182f 
Implement AbstractMultiObjectiveBenchmark.
@PhMueller
MO Benchmarks: API enhancement (#146)
9dde397 
* UPDATE Benchmark API to work with Multi-Objective Benchmarks
@PhMueller
Update Github Actions (#151)
812bdd0 
* Update Github Actions Workflow  and drop support for singularity < 3.7
@pfistfl @PhMueller
Add YAHPO Benchmark (#142)
5d3d75e 
* Add yahpo_gym w help from phmueller

Co-authored-by: PhMueller <[email protected]>
@Neeratyoy
Updating tabular benchmark data for ML benchmarks
1544e1f
@Neeratyoy
version update for tabular benchmark
225a82b
@PhMueller
ADD Multi-Objective Nasbench201 (v0.0.6) (#152)
ac9547a 
Update the Nasbench201 benchmark to support Multi-Objective queries.

If you want to use the *single objective* Nasbench201 benchmark, you can query the SO version of this benchmark. 
Although, we have not changed the benchmark logic, you can also use the container v0.0.5 in your experiments to reproduce results from the old version of this benchmark.
@ayushi-3536
Benchmark: Fair Adult from MO-ASHA (#148)
3f08eb2 
We add the benchmark from the MO-ASHA paper by Schmucker et al.

It is a MO benchmark, training an MLP on the Adult data set.
@ayushi-3536 @PhMueller
Multi Objective CNN benchmark: Flowers and Fashion (#147)
4c4f1d9 
Added mo cnn benchmarks from bag of baseline paper

We deviate from the original benchmark in two points: 
* we return as cost only the  training time instead of the total elapsed time
* we return as objective for minimization instead of `-100 * accuracy` now `1 - accuracy` to achieve better output scalings. 

Co-authored-by: ayushi-3536 <[email protected]>
Co-authored-by: Philipp Müller <[email protected]>
@Neeratyoy
Updating container information for ML benches
fd25d57
@Neeratyoy
Bug fix for containers and fidelity space
0ad578f
@Neeratyoy
Updating container version used for experiments
dfa2034
PhMueller and others added 2 commits September 28, 2022 11:13
@PhMueller @dwoiwode
HPOBench with python >= 3.10 (#162)
5889a04 
Make installable for Python 3.10.6
* Add 3.10 to test pipeline
* Remove upper Python version limit
* Change python-versions to strings
* Surround action python-version with quotation marks

* Install paramnet dependencies only for Python <= 3.9
* Disable local paramnet test for Python > 3.9

Co-authored-by: Dominik Woiwode <[email protected]>
@PhMueller
Speed up CI testing (#163)
45b1eb0 
* Speed up tests by disabling slow tests.
@PhMueller PhMueller changed the base branch from master to development December 15, 2022 09:25
PhMueller and others added 7 commits December 15, 2022 11:24
@PhMueller
Refactor PR (#164)
3b2fa7b 
* Refactor: Rename AbstractBenchmark Class
@PhMueller
Nas201 - v0.0.6 (#165)
ebff935 
* Update 201 version 0.0.6

- scale the misclassification rate form 0 to 100 -> 0 and 1
- Update test cases
- link to the correct container version
- Fix Nas201 Test Cases
- remove a Github action trigger. It started all test cases twice. 
- Fix GithubActions: Install correct xgboost version for 3.10
@PhMueller
Nas101 + Nas1shot1 v0.0.5 - Multi Objective (#166)
4414e3d 
* Nas101 v.0.0.5
* Nas1shot1 v.0.0.5
* Test Cases for 101
@PhMueller
MO CNN and MO Adult v0.0.2 (#167)
47f8b5b 
Update to 0.0.2
@PhMueller
Towards compatibility with windows os. (#170)
46f719d
@PhMueller @ayushi-3536 @dwoiwode
Raw YAHPO Benchmarks (#153)
236e542 
* Yahpo Raw: You can run fair/iml/rbv2 experiments from yahpo. 

The installation is quite complicated and described in the recipe. 

* Introduce YAHPO Surrogate V0.0.2
It also goes hand in hand with an update for the yahpo surrogate benchmarks. 

---------

Co-authored-by: ayushi-3536 <[email protected]>
Co-authored-by: ayushi-3536 <[email protected]>
Co-authored-by: Dominik Woiwode <[email protected]>
@KEggensperger
Merge branch 'development' into LCs
36acf0d
@Neeratyoy Neeratyoy changed the base branch from development to develop April 3, 2023 13:13
@Neeratyoy Neeratyoy mentioned this pull request Apr 3, 2023
Open
@Neeratyoy
Copy link
Collaborator Author

#177 is the updated, cleaner PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@KEggensperger KEggensperger Awaiting requested review from KEggensperger

@PhMueller PhMueller Awaiting requested review from PhMueller

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@Neeratyoy @PhMueller @KEggensperger @noorawad @pfistfl @ayushi-3536