Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding public URLs for tabular benchmark #121

Open
wants to merge 102 commits into
base: development
Choose a base branch
from
Open

Adding public URLs for tabular benchmark #121

wants to merge 102 commits into from

Commits on Jun 23, 2021

  1. Adding sample RF space for tabular collection design

    @Neeratyoy
    Neeratyoy committed Jun 23, 2021
    Configuration menu
    Copy the full SHA
    b2155dd View commit details
    Browse the repository at this point in the history

  2. Placeholder SVM benchmark to interface tabular data collection

    @Neeratyoy
    Neeratyoy committed Jun 23, 2021
    Configuration menu
    Copy the full SHA
    ce405e6 View commit details
    Browse the repository at this point in the history

Commits on Jun 24, 2021

  1. Writing common ML benchmark class for tabular collection

    @Neeratyoy
    Neeratyoy committed Jun 24, 2021
    Configuration menu
    Copy the full SHA
    2ef3af8 View commit details
    Browse the repository at this point in the history

  2. Adding placeholder for HistGradientBoostedClassifier

    @Neeratyoy
    Neeratyoy committed Jun 24, 2021
    Configuration menu
    Copy the full SHA
    61b6963 View commit details
    Browse the repository at this point in the history

  3. Minor code cleaning

    @Neeratyoy
    Neeratyoy committed Jun 24, 2021
    Configuration menu
    Copy the full SHA
    a5d0217 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2021

  1. Reformatting output dict + option to add more metrics

    @Neeratyoy
    Neeratyoy committed Jun 26, 2021
    Configuration menu
    Copy the full SHA
    3def203 View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2021

  1. Removing redundant import

    @Neeratyoy
    Neeratyoy committed Jun 28, 2021
    Configuration menu
    Copy the full SHA
    750cc7d View commit details
    Browse the repository at this point in the history

Commits on Jun 30, 2021

  1. Decoupling storage of costs for each metric

    @Neeratyoy
    Neeratyoy committed Jun 30, 2021
    Configuration menu
    Copy the full SHA
    e7665e6 View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2021

  1. Including test scores in objective

    @Neeratyoy
    Neeratyoy committed Jul 1, 2021
    Configuration menu
    Copy the full SHA
    47fe4cd View commit details
    Browse the repository at this point in the history

  2. Documenting the structure of information in each fn eval.

    @Neeratyoy
    Neeratyoy committed Jul 1, 2021
    Configuration menu
    Copy the full SHA
    2d085ec View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2021

  1. Some decisions on lower bound for subsample fidelity

    @Neeratyoy
    Neeratyoy committed Jul 2, 2021
    Configuration menu
    Copy the full SHA
    2da9d5c View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2021

  1. AbstractBenchmark update for fidelity option + including XGBoost

    @Neeratyoy
    Neeratyoy committed Jul 6, 2021
    Configuration menu
    Copy the full SHA
    751d2e9 View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2021

  1. Adding sample RF space for tabular collection design

    @Neeratyoy
    Neeratyoy committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    3f84afb View commit details
    Browse the repository at this point in the history

  2. Placeholder SVM benchmark to interface tabular data collection

    @Neeratyoy
    Neeratyoy committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    09b296a View commit details
    Browse the repository at this point in the history

  3. Writing common ML benchmark class for tabular collection

    @Neeratyoy
    Neeratyoy committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    af4f593 View commit details
    Browse the repository at this point in the history

  4. Adding placeholder for HistGradientBoostedClassifier

    @Neeratyoy
    Neeratyoy committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    df2462d View commit details
    Browse the repository at this point in the history

  5. Minor code cleaning

    @Neeratyoy
    Neeratyoy committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    4d1d2d6 View commit details
    Browse the repository at this point in the history

  6. Reformatting output dict + option to add more metrics

    @Neeratyoy
    Neeratyoy committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    299e592 View commit details
    Browse the repository at this point in the history

  7. Removing redundant import

    @Neeratyoy
    Neeratyoy committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    c46321d View commit details
    Browse the repository at this point in the history

  8. Decoupling storage of costs for each metric

    @Neeratyoy
    Neeratyoy committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    17f6634 View commit details
    Browse the repository at this point in the history

  9. Including test scores in objective

    @Neeratyoy
    Neeratyoy committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    7de891f View commit details
    Browse the repository at this point in the history

  10. Documenting the structure of information in each fn eval.

    @Neeratyoy
    Neeratyoy committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    ec316c3 View commit details
    Browse the repository at this point in the history

  11. Some decisions on lower bound for subsample fidelity

    @Neeratyoy
    Neeratyoy committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    e7f69b9 View commit details
    Browse the repository at this point in the history

  12. AbstractBenchmark update for fidelity option + including XGBoost

    @Neeratyoy
    Neeratyoy committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    edb3e7f View commit details
    Browse the repository at this point in the history

  13. Merge branch 'thesis-paper' of https://github.com/Neeratyoy/HPOBench … 

    …into thesis-paper
    @Neeratyoy
    Neeratyoy committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    642027b View commit details
    Browse the repository at this point in the history

Commits on Jul 8, 2021

  1. Option to load data splits from disk

    @Neeratyoy
    Neeratyoy committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    9e907e6 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2021

  1. Reordering data load to work for different cases

    @Neeratyoy
    Neeratyoy committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    f0d4f36 View commit details
    Browse the repository at this point in the history

Commits on Jul 14, 2021

  1. Updating source of SVM HP range

    @Neeratyoy
    Neeratyoy committed Jul 14, 2021
    Configuration menu
    Copy the full SHA
    dbeae7c View commit details
    Browse the repository at this point in the history

  2. Adding Tabular Benchmark class

    @Neeratyoy
    Neeratyoy committed Jul 14, 2021
    Configuration menu
    Copy the full SHA
    f277a2e View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2021

  1. Adding TabularBenchmark interface + easy import

    @Neeratyoy
    Neeratyoy committed Jul 15, 2021
    Configuration menu
    Copy the full SHA
    60d5646 View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2021

  1. Adding LR space

    @Neeratyoy
    Neeratyoy committed Jul 16, 2021
    Configuration menu
    Copy the full SHA
    c4100fd View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2021

  1. Standardizing fidelity space definitions

    @Neeratyoy
    Neeratyoy committed Jul 19, 2021
    Configuration menu
    Copy the full SHA
    9c6dcdb View commit details
    Browse the repository at this point in the history

  2. Standardizing HPs + Adding NN space

    @Neeratyoy
    Neeratyoy committed Jul 19, 2021
    Configuration menu
    Copy the full SHA
    74b6919 View commit details
    Browse the repository at this point in the history

  3. Small placeholder for testing

    @Neeratyoy
    Neeratyoy committed Jul 19, 2021
    Configuration menu
    Copy the full SHA
    785055e View commit details
    Browse the repository at this point in the history

Commits on Jul 20, 2021

  1. Updating NN HP space + Helper function for TabularBenchmark

    @Neeratyoy
    Neeratyoy committed Jul 20, 2021
    Configuration menu
    Copy the full SHA
    0159a35 View commit details
    Browse the repository at this point in the history

  2. Adding fidelity range retrieval utility to TabularBenchmark

    @Neeratyoy
    Neeratyoy committed Jul 20, 2021
    Configuration menu
    Copy the full SHA
    e9e097a View commit details
    Browse the repository at this point in the history

Commits on Jul 21, 2021

  1. Enforcing subsample lower bound check inside objective

    @Neeratyoy
    Neeratyoy committed Jul 21, 2021
    Configuration menu
    Copy the full SHA
    4797109 View commit details
    Browse the repository at this point in the history

  2. Bug fix + adding precicion as metric

    @Neeratyoy
    Neeratyoy committed Jul 21, 2021
    Configuration menu
    Copy the full SHA
    dbb7327 View commit details
    Browse the repository at this point in the history

Commits on Jul 22, 2021

  1. Fixing param spaces and model building for LR, SVM

    @Neeratyoy
    Neeratyoy committed Jul 22, 2021
    Configuration menu
    Copy the full SHA
    7d5ca57 View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2021

  1. TabularBenchmark edit to read compressed files and query a dataframe

    @Neeratyoy
    Neeratyoy committed Jul 26, 2021
    Configuration menu
    Copy the full SHA
    a6d94bb View commit details
    Browse the repository at this point in the history

Commits on Jul 27, 2021

  1. Not evaluating training set to save time

    @Neeratyoy
    Neeratyoy committed Jul 27, 2021
    Configuration menu
    Copy the full SHA
    93b6908 View commit details
    Browse the repository at this point in the history

  2. Fidelity change for trees + NN space change

    @Neeratyoy
    Neeratyoy committed Jul 27, 2021
    Configuration menu
    Copy the full SHA
    8164eb0 View commit details
    Browse the repository at this point in the history

Commits on Jul 29, 2021

  1. Final RF space

    @Neeratyoy
    Neeratyoy committed Jul 29, 2021
    Configuration menu
    Copy the full SHA
    6916c9c View commit details
    Browse the repository at this point in the history

  2. Final XGB space

    @Neeratyoy
    Neeratyoy committed Jul 29, 2021
    Configuration menu
    Copy the full SHA
    8e5912b View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2021

  1. Final HistGB space

    @Neeratyoy
    Neeratyoy committed Jul 30, 2021
    Configuration menu
    Copy the full SHA
    6968ac3 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2021

  1. Finalizing RF, XGB, NN

    @Neeratyoy
    Neeratyoy committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    79dd1f3 View commit details
    Browse the repository at this point in the history

  2. TabularBenchmark edit to process only table and metadata

    @Neeratyoy
    Neeratyoy committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    ca1e0d4 View commit details
    Browse the repository at this point in the history

Commits on Aug 4, 2021

  1. Merge remote-tracking branch 'origin/development' into PR_Multi-fidel… 

    …ity-tabular-benchmarks
    @PhMueller
    PhMueller committed Aug 4, 2021
    Configuration menu
    Copy the full SHA
    87133ed View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/development' into PR_Multi-fidel… 

    …ity-tabular-benchmarks
    @PhMueller
    PhMueller committed Aug 4, 2021
    Configuration menu
    Copy the full SHA
    6096204 View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2021

  1. TabularBenchmark 

    - Rearrange the benchmark.
- Move data parts to a data manager. It is able to download the data from the web, if it is not present on the local machine.
- Enforce the API structure
    @PhMueller
    PhMueller committed Aug 11, 2021
    Configuration menu
    Copy the full SHA
    0d70d36 View commit details
    Browse the repository at this point in the history

  2. Pycodestyle

    @PhMueller
    PhMueller committed Aug 11, 2021
    Configuration menu
    Copy the full SHA
    12ebce8 View commit details
    Browse the repository at this point in the history

  3. Flake8

    @PhMueller
    PhMueller committed Aug 11, 2021
    Configuration menu
    Copy the full SHA
    873781e View commit details
    Browse the repository at this point in the history

  4. Adapt ML Benchmark Template to fit with current API

    @PhMueller
    PhMueller committed Aug 11, 2021
    Configuration menu
    Copy the full SHA
    532a905 View commit details
    Browse the repository at this point in the history

  5. Corret Datamanager. 

    But how to download the task data to disk?
    @PhMueller
    PhMueller committed Aug 11, 2021
    Configuration menu
    Copy the full SHA
    9dbd61c View commit details
    Browse the repository at this point in the history

  6. Finalize HistGB Benchmarks 

    - split them into multiple benchmarks according to their fidelity spaces.
    @PhMueller
    PhMueller committed Aug 11, 2021
    Configuration menu
    Copy the full SHA
    0304146 View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2021

  1. Write OpenML Datamanager

    @PhMueller
    PhMueller committed Aug 16, 2021
    Configuration menu
    Copy the full SHA
    3e95d19 View commit details
    Browse the repository at this point in the history

  2. Unify interface for the other ml benchmarks.

    @PhMueller
    PhMueller committed Aug 16, 2021
    Configuration menu
    Copy the full SHA
    f3fbd58 View commit details
    Browse the repository at this point in the history

  3. Flake + Pep

    @PhMueller
    PhMueller committed Aug 16, 2021
    Configuration menu
    Copy the full SHA
    e57fbcb View commit details
    Browse the repository at this point in the history

  4. Add Container Interface

    @PhMueller
    PhMueller committed Aug 16, 2021
    Configuration menu
    Copy the full SHA
    f6131ea View commit details
    Browse the repository at this point in the history

  5. Mark task_id as required.

    @PhMueller
    PhMueller committed Aug 16, 2021
    Configuration menu
    Copy the full SHA
    36bc391 View commit details
    Browse the repository at this point in the history

  6. Adapt Interfaces

    @PhMueller
    PhMueller committed Aug 16, 2021
    Configuration menu
    Copy the full SHA
    a5c7d62 View commit details
    Browse the repository at this point in the history

  7. Fix minor errors.

    @PhMueller
    PhMueller committed Aug 16, 2021
    Configuration menu
    Copy the full SHA
    c5f6979 View commit details
    Browse the repository at this point in the history

  8. Fix minor errors.

    @PhMueller
    PhMueller committed Aug 16, 2021
    Configuration menu
    Copy the full SHA
    48af58d View commit details
    Browse the repository at this point in the history

  9. Pylint

    @PhMueller
    PhMueller committed Aug 16, 2021
    Configuration menu
    Copy the full SHA
    cf24488 View commit details
    Browse the repository at this point in the history

Commits on Aug 17, 2021

  1. Init Model can handle now Configurations

    @PhMueller
    PhMueller committed Aug 17, 2021
    Configuration menu
    Copy the full SHA
    528dde1 View commit details
    Browse the repository at this point in the history

  2. PR Requests: Rename Classes

    @PhMueller
    PhMueller committed Aug 17, 2021
    Configuration menu
    Copy the full SHA
    6bdf5c0 View commit details
    Browse the repository at this point in the history

  3. PR Requests: Move dependencies to correct directory

    @PhMueller
    PhMueller committed Aug 17, 2021
    Configuration menu
    Copy the full SHA
    b8b30a5 View commit details
    Browse the repository at this point in the history

  4. PR Requests: Tabular Benchmarks - Remove unnecessary class definition

    @PhMueller
    PhMueller committed Aug 17, 2021
    Configuration menu
    Copy the full SHA
    875c594 View commit details
    Browse the repository at this point in the history

  5. PR Requests: Minor improvments

    @PhMueller
    PhMueller committed Aug 17, 2021
    Configuration menu
    Copy the full SHA
    8891e33 View commit details
    Browse the repository at this point in the history

  6. PR Requests: Update upper bounds of the fidelities

    @PhMueller
    PhMueller committed Aug 17, 2021
    Configuration menu
    Copy the full SHA
    75f345d View commit details
    Browse the repository at this point in the history

  7. PR Requests: Remove OriginalTabBenchmarks

    @PhMueller
    PhMueller committed Aug 17, 2021
    Configuration menu
    Copy the full SHA
    8c2ab6c View commit details
    Browse the repository at this point in the history

  8. PR Requests: Revert the query function

    @PhMueller
    PhMueller committed Aug 17, 2021
    Configuration menu
    Copy the full SHA
    e24d537 View commit details
    Browse the repository at this point in the history

  9. PR Requests: Minor improvements

    @PhMueller
    PhMueller committed Aug 17, 2021
    Configuration menu
    Copy the full SHA
    3c4f375 View commit details
    Browse the repository at this point in the history

  10. Pycodestyle

    @PhMueller
    PhMueller committed Aug 17, 2021
    Configuration menu
    Copy the full SHA
    6fc7f57 View commit details
    Browse the repository at this point in the history

  11. Add missing requirements

    @PhMueller
    PhMueller committed Aug 17, 2021
    Configuration menu
    Copy the full SHA
    0430c68 View commit details
    Browse the repository at this point in the history

  12. Minor Improvements 

    - cast return values to float
- improve the __all__ vars
    @PhMueller
    PhMueller committed Aug 17, 2021
    Configuration menu
    Copy the full SHA
    3eb3a2d View commit details
    Browse the repository at this point in the history

  13. ADD container recipes

    @PhMueller
    PhMueller committed Aug 17, 2021
    Configuration menu
    Copy the full SHA
    fa691f7 View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2021

  1. PR: Fix path in tabular data loader

    @PhMueller
    PhMueller committed Aug 19, 2021
    Configuration menu
    Copy the full SHA
    f64917e View commit details
    Browse the repository at this point in the history

  2. PR: Remove casting configspace to np.floats

    @PhMueller
    PhMueller committed Aug 19, 2021
    Configuration menu
    Copy the full SHA
    b95d2a5 View commit details
    Browse the repository at this point in the history

  3. PR: Move everything back from ml_mmfb/ to ml/

    @PhMueller
    PhMueller committed Aug 19, 2021
    Configuration menu
    Copy the full SHA
    d7d7a2d View commit details
    Browse the repository at this point in the history

  4. PR: Remove pybnn from the init. 

    This would cause an error since importing stuff for the tab benchmarks would require the pybnn stuff.
    @PhMueller
    PhMueller committed Aug 19, 2021
    Configuration menu
    Copy the full SHA
    be641f8 View commit details
    Browse the repository at this point in the history

  5. PR: Cleanup

    @PhMueller
    PhMueller committed Aug 19, 2021
    Configuration menu
    Copy the full SHA
    7bc25bc View commit details
    Browse the repository at this point in the history

  6. PR: Fix Tests

    @PhMueller
    PhMueller committed Aug 19, 2021
    Configuration menu
    Copy the full SHA
    b0d9b7f View commit details
    Browse the repository at this point in the history

  7. Adding public URLs for tabular benchmark

    @Neeratyoy
    Neeratyoy committed Aug 19, 2021
    Configuration menu
    Copy the full SHA
    59bd905 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'PR_Multi-fidelity-tabular-benchmarks' into add-tabular-… 

    …urls
    @Neeratyoy
    Neeratyoy committed Aug 19, 2021
    Configuration menu
    Copy the full SHA
    6576e99 View commit details
    Browse the repository at this point in the history

  9. Adding more models

    @Neeratyoy
    Neeratyoy committed Aug 19, 2021
    Configuration menu
    Copy the full SHA
    f576fb3 View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2021

  1. Updating figshare URLs with new public ones

    @Neeratyoy
    Neeratyoy committed Aug 20, 2021
    Configuration menu
    Copy the full SHA
    63f5177 View commit details
    Browse the repository at this point in the history

  2. PR Fix URLs and dependencies

    @PhMueller
    PhMueller committed Aug 20, 2021
    Configuration menu
    Copy the full SHA
    5335831 View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2021

  1. Updating URL for SVM data

    @Neeratyoy
    Neeratyoy committed Aug 21, 2021
    Configuration menu
    Copy the full SHA
    cf9b4ef View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2021

  1. Updating Tabular bench URLs

    @Neeratyoy
    Neeratyoy committed Aug 23, 2021
    Configuration menu
    Copy the full SHA
    ed7d23e View commit details
    Browse the repository at this point in the history

Commits on Aug 25, 2021

  1. PR Fix URLs and dependencies

    @PhMueller
    PhMueller committed Aug 25, 2021
    Configuration menu
    Copy the full SHA
    9181bbb View commit details
    Browse the repository at this point in the history

  2. PR Fix URLs and dependencies

    @PhMueller
    PhMueller committed Aug 25, 2021
    Configuration menu
    Copy the full SHA
    451ff08 View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2021

  1. Updating RF benchmark URL

    @Neeratyoy
    Neeratyoy authored Aug 26, 2021
    Configuration menu
    Copy the full SHA
    310b11e View commit details
    Browse the repository at this point in the history

  2. Updating XGB URL

    @Neeratyoy
    Neeratyoy authored Aug 26, 2021
    Configuration menu
    Copy the full SHA
    f01286b View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2021

  1. PR Fix tests

    @PhMueller
    PhMueller committed Aug 27, 2021
    Configuration menu
    Copy the full SHA
    12b72b1 View commit details
    Browse the repository at this point in the history

  2. New Urls

    @PhMueller
    PhMueller committed Aug 27, 2021
    Configuration menu
    Copy the full SHA
    41aa96b View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2021

  1. Trigger Rebuild.

    @PhMueller
    PhMueller committed Aug 30, 2021
    Configuration menu
    Copy the full SHA
    c23e354 View commit details
    Browse the repository at this point in the history

  2. Fix Dataloader Assertion

    @PhMueller
    PhMueller committed Aug 30, 2021
    Configuration menu
    Copy the full SHA
    1fa684c View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2021

  1. Merge branch 'PR_Multi-fidelity-tabular-benchmarks' into add-tabular-… 

    …urls
    @Neeratyoy
    Neeratyoy authored Aug 31, 2021
    Configuration menu
    Copy the full SHA
    5f015a6 View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2021

  1. Merge branch 'development' into add-tabular-urls

    @Neeratyoy
    Neeratyoy authored Sep 1, 2021
    Configuration menu
    Copy the full SHA
    11c57bf View commit details
    Browse the repository at this point in the history

Commits on Oct 6, 2021

  1. Merge branch 'development' into add-tabular-urls

    @Neeratyoy
    Neeratyoy committed Oct 6, 2021
    Configuration menu
    Copy the full SHA
    8d7ea97 View commit details
    Browse the repository at this point in the history

  2. inference cost key fix

    @Neeratyoy
    Neeratyoy committed Oct 6, 2021
    Configuration menu
    Copy the full SHA
    6394521 View commit details
    Browse the repository at this point in the history