Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable rerunning modules on all existing submissions #470

Open
JuliaS92 opened this issue Dec 3, 2024 · 2 comments
Open

Enable rerunning modules on all existing submissions #470

JuliaS92 opened this issue Dec 3, 2024 · 2 comments

Comments

@JuliaS92
Copy link
Contributor

JuliaS92 commented Dec 3, 2024

Is your feature request related to a problem? Please describe.
We need to be able to rerun a module on all existing submission in case we ever make changes that affect the benchmarking results.

Describe the solution you'd like
A script that can be run by a maintainer to recreate results.json for submission to the Proteobench/module repository.
This can either be a standalone srcipt (preferable) or a wrapper around streamlit page tests.
If possible the script should be generic and take the module name, results.json and file location as input.

Additional context
Here are some prerequisites:

  • all data required for rerunning a module are stored
  • method names are standardized enough to write one generic runner

Please add any additional prerequisites, hurdles or goals you see for this.

@JuliaS92
Copy link
Contributor Author

JuliaS92 commented Dec 4, 2024

Would you rather see this as a jupyter notebook (react with eyes), or as a command line script (react with rocket)?
A notebook makes it easier to see what's happening, while a command line script could more easily be run via gh actions.

@RobbinBouwmeester
Copy link
Contributor

I am OK not making a GH actions out of this. As this should be done only very rarely and should be avoided if possible.

I like your suggestion of a jupyter notebook.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants