Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Basic benchmarking structure #413

Merged
merged 121 commits into from
Nov 13, 2024
Merged

Conversation

fabianliebig
Copy link
Collaborator

@fabianliebig fabianliebig commented Oct 22, 2024

Since #410 added too many lines, lets start over with a single basic structure, which includes:

  • Benchmarking classes that execute a custom-defined callable which makes use of parametrizable setting input.
  • Classes to Create and develop further settings, used in the callable.
  • Results containing the DataFrame and relevant metadata based on the execution.
  • One benchmark function to demonstrate the module's synergy.

After this PR is reviewed, I plan to create follow up PRs which may run in parallel:

  • Refactoring of the classes and add a possibility to encapsulate the domain.
  • Add the respective parts to persist the results in an S3 Bucket.

Please feel free to adjust or extend this list as necessary. I appreciate your feedback.

AdrianSosic

This comment was marked as resolved.

@fabianliebig

This comment was marked as resolved.

AdrianSosic

This comment was marked as resolved.

AVHopp

This comment was marked as resolved.

benchmark/result/result.py Outdated Show resolved Hide resolved
benchmark/result/result.py Outdated Show resolved Hide resolved
benchmark/result/result.py Outdated Show resolved Hide resolved
benchmark/result/result.py Outdated Show resolved Hide resolved
benchmark/result/result.py Outdated Show resolved Hide resolved
benchmark/definition/benchmarking.py Outdated Show resolved Hide resolved
benchmark/domain/synthetic_3.py Outdated Show resolved Hide resolved
benchmark/domain/synthetic_3.py Outdated Show resolved Hide resolved
benchmark/__main__.py Outdated Show resolved Hide resolved
benchmark/__main__.py Outdated Show resolved Hide resolved
benchmark/domain/__init__.py Outdated Show resolved Hide resolved
benchmark/result/utils.py Outdated Show resolved Hide resolved
benchmark/result/utils.py Outdated Show resolved Hide resolved
benchmark/domain/synthetic_3.py Outdated Show resolved Hide resolved
benchmark/definition/benchmarking.py Outdated Show resolved Hide resolved
benchmark/definition/benchmarking.py Outdated Show resolved Hide resolved
benchmark/definition/benchmarking.py Outdated Show resolved Hide resolved
benchmark/domain/synthetic_3.py Outdated Show resolved Hide resolved
@fabianliebig

This comment was marked as resolved.

AdrianSosic

This comment was marked as resolved.

@fabianliebig

This comment was marked as resolved.

benchmark/__init__.py Outdated Show resolved Hide resolved
benchmark/definition/benchmarking.py Outdated Show resolved Hide resolved
benchmark/definition/benchmarking.py Outdated Show resolved Hide resolved
benchmark/__init__.py Outdated Show resolved Hide resolved
benchmark/definition/benchmarking.py Outdated Show resolved Hide resolved
benchmark/result/metadata_class.py Outdated Show resolved Hide resolved
benchmark/result/result.py Outdated Show resolved Hide resolved
benchmark/result/result.py Outdated Show resolved Hide resolved
benchmark/result/result.py Outdated Show resolved Hide resolved
@AdrianSosic AdrianSosic changed the title Feature/basic benchmarking structure Basic benchmarking structure Nov 5, 2024
@AdrianSosic AdrianSosic added the new feature New functionality label Nov 5, 2024
benchmarks/definition/core.py Outdated Show resolved Hide resolved
benchmarks/definition/core.py Outdated Show resolved Hide resolved
benchmarks/definition/core.py Outdated Show resolved Hide resolved
benchmarks/domains/synthetic_3.py Outdated Show resolved Hide resolved
benchmarks/result/metadata_class.py Outdated Show resolved Hide resolved
AdrianSosic

This comment was marked as resolved.

benchmarks/__main__.py Show resolved Hide resolved
benchmarks/domains/synthetic_2C1D_1C.py Outdated Show resolved Hide resolved
benchmarks/domains/synthetic_2C1D_1C.py Outdated Show resolved Hide resolved
benchmarks/definition/config.py Outdated Show resolved Hide resolved
benchmarks/result/result.py Outdated Show resolved Hide resolved
pyproject.toml Outdated Show resolved Hide resolved
benchmarks/domains/synthetic_2C1D_1C.py Outdated Show resolved Hide resolved
benchmarks/domains/synthetic_2C1D_1C.py Outdated Show resolved Hide resolved
benchmarks/domains/synthetic_2C1D_1C.py Outdated Show resolved Hide resolved
benchmarks/domains/synthetic_2C1D_1C.py Outdated Show resolved Hide resolved
benchmarks/definition/core.py Outdated Show resolved Hide resolved
baybe/serialization/core.py Outdated Show resolved Hide resolved
benchmarks/definition/config.py Outdated Show resolved Hide resolved
benchmarks/definition/core.py Outdated Show resolved Hide resolved
benchmarks/domains/synthetic_2C1D_1C.py Show resolved Hide resolved
benchmarks/result/metadata.py Outdated Show resolved Hide resolved
pyproject.toml Outdated Show resolved Hide resolved
benchmarks/domains/synthetic_2C1D_1C.py Outdated Show resolved Hide resolved
benchmarks/domains/synthetic_2C1D_1C.py Outdated Show resolved Hide resolved
benchmarks/domains/synthetic_2C1D_1C.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@AVHopp AVHopp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is the first approve :) Thank you very much for all of the work and for including all of the comments :)
Only thing I'd like to see removed is the plotting, but I'll leave that up to you and @AdrianSosic whether or not that should be in here.

@fabianliebig fabianliebig force-pushed the feature/basic_benchmarking_structure branch from b3a151b to bae86d1 Compare November 11, 2024 15:50
@AdrianSosic AdrianSosic force-pushed the feature/basic_benchmarking_structure branch from 46301ba to 27893fe Compare November 13, 2024 10:33
Copy link
Collaborator

@AdrianSosic AdrianSosic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's update the PR description and we are ready to go 🚀
As agreed, a follow-up PR will take care of refactoring the class responsibilities but as a first version, this will fulfill its purpose 👍🏼

CHANGELOG.md Outdated Show resolved Hide resolved
benchmarks/__init__.py Show resolved Hide resolved
benchmarks/domains/synthetic_2C1D_1C.py Show resolved Hide resolved
benchmarks/definition/config.py Outdated Show resolved Hide resolved
@fabianliebig fabianliebig force-pushed the feature/basic_benchmarking_structure branch from dff35ba to d902170 Compare November 13, 2024 12:49
@fabianliebig fabianliebig merged commit 00e1488 into main Nov 13, 2024
10 of 11 checks passed
@fabianliebig fabianliebig deleted the feature/basic_benchmarking_structure branch November 13, 2024 15:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants