Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MAINT: Add benchmarking files #55

Merged
merged 1 commit into from
Jan 20, 2025

Conversation

jhlegarreta
Copy link
Contributor

Add benchmarking files so that nifreeze can be benchmarked using asv:

  • Add the actual files that allow to benchmark different nifreeze capabilities.
  • Add a new benchmark optional dependencies section to pyproject.toml.
  • Add the asv configuration file.
  • Add a README.rst file to explain how to run the benchmarking.
  • Add a GitHub Actions workflow file to run the benchmarks for every PR.

@jhlegarreta
Copy link
Contributor Author

jhlegarreta commented Jan 19, 2025

Thought this would be relevant for when we will be needing to measure/benchmark nifreeze. This is a shy attempt to it. A few thoughts:

  • Do we prefer/want a benchmark section to be added to tox.ini?
  • Maybe a smaller testing/benchmarking dataset should be used. Especially taking into account that the GHA runners have very limited resources, and that I had to increase the default_benchmark_timeout value.
  • This PR is by no means meant to be comprehensive; it is meant to be an initial step: further abstraction and sophistication so that we avoid boilerplate code, etc. (e.g. the scikit-learn way could serve as an inspiration) are left for a separate PR.
  • The asv.conf.json file is placed at the benchmarks folder. Let me know if we want to move this to the project root.
  • Eventually we will be willing to compare across different commits/over time so that we avoid introducing changes that are detrimental to performance.
  • May need to pin dependencies, which will also require maintenance, since at some point we will want to measure the performance with modern, faster versions of packages.

An issue could be opened to keep track of the relevant comments above.

@jhlegarreta jhlegarreta force-pushed the AddAsvBenchmarking branch 7 times, most recently from c68d7e4 to ec78a80 Compare January 19, 2025 23:44
Copy link

codecov bot commented Jan 20, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 65.85%. Comparing base (a78af6c) to head (ec78a80).
Report is 8 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #55      +/-   ##
==========================================
+ Coverage   65.74%   65.85%   +0.10%     
==========================================
  Files          19       19              
  Lines         943      943              
  Branches      121      121              
==========================================
+ Hits          620      621       +1     
+ Misses        278      277       -1     
  Partials       45       45              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@oesteban oesteban left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not fully sure what's ASV about, but it looks useful :) Happy to see forward-looking initiatives @jhlegarreta.

benchmarks/README.rst Outdated Show resolved Hide resolved
benchmarks/README.rst Outdated Show resolved Hide resolved
benchmarks/README.rst Outdated Show resolved Hide resolved
benchmarks/README.rst Outdated Show resolved Hide resolved
benchmarks/README.rst Outdated Show resolved Hide resolved
benchmarks/README.rst Outdated Show resolved Hide resolved
benchmarks/asv.conf.json Outdated Show resolved Hide resolved
@oesteban oesteban changed the title ENH: Add benchmarking files MAINT: Add benchmarking files Jan 20, 2025
@jhlegarreta jhlegarreta force-pushed the AddAsvBenchmarking branch 2 times, most recently from 0cd2e19 to 64ac5f6 Compare January 20, 2025 15:26
Add benchmarking files so that `nifreeze` can be benchmarked using
`asv`:
- Add the actual files that allow to benchmark different `nifreeze`
  capabilities.
- Add a new `benchmark` optional dependencies section to
  `pyproject.toml`.
- Add the `asv` configuration file.
- Add a `README.rst` file to explain how to run the benchmarking.
- Add a GitHub Actions workflow file to run the benchmarks for every PR.

Co-authored-by: Oscar Esteban <[email protected]>
@oesteban oesteban merged commit 856a00e into nipreps:main Jan 20, 2025
8 checks passed
@jhlegarreta jhlegarreta deleted the AddAsvBenchmarking branch January 20, 2025 17:04
@jhlegarreta jhlegarreta mentioned this pull request Jan 20, 2025
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants