Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added DVC, teamcity-messages and Julia TC integration #1915

Merged
merged 11 commits into from
Oct 22, 2024
Merged

Added DVC, teamcity-messages and Julia TC integration #1915

merged 11 commits into from
Oct 22, 2024

Conversation

evetion
Copy link
Member

@evetion evetion commented Oct 21, 2024

Fixes #1879

This does three things

  • Adds teamcity-messages as a Python test dependency, enabling TC to keep track of pytest tests (number of tests, number of failures, linking to specific logs), making debugging much easier.
Screenshot 2024-10-21 at 22 35 54
  • Adds teamcity specific logging in Julia (much like teamcity-messages) to keep track of the integration test time and differences with a previous model. These parameters are tracked by TC (and can be failed on), and can be plotted over time.
Screenshot 2024-10-21 at 22 38 31
  • Finally, and biggest, this adds DVC to our project, with the minio repo tracked, and the integration test as stage. It tracks a new default toml for that model (models/integration.toml), and like TC in 2, the time and differences. Once someone changes either the hws model, or the core code, or the integration.toml, the results are outdated and dvc repro will rerun the test. We can save these experiments over time, and compare results with the VS Code plugin:
Screenshot 2024-10-21 at 22 09 27

This disables the threshold test in the integration test, and opts for monitoring results (could be done automatically) instead. I've also opted for keeping track of results and settings threshold outside of Julia.

This is a backwards compatible setup PR. Changes/additions for benchmark tuning can be made in another PR.

@evetion evetion requested a review from visr October 21, 2024 20:31
@visr
Copy link
Member

visr commented Oct 21, 2024

Wonderful!

@evetion
Copy link
Member Author

evetion commented Oct 22, 2024

Wonderful!

This is now good to go.

Copy link
Member

@visr visr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Left some small suggestions.

.dvcignore Outdated Show resolved Hide resolved
core/test/utils.jl Outdated Show resolved Hide resolved
pixi.toml Show resolved Hide resolved
@visr visr merged commit d7fb4bb into main Oct 22, 2024
27 checks passed
@visr visr deleted the feat/dvc-exp branch October 22, 2024 21:26
@evetion
Copy link
Member Author

evetion commented Oct 23, 2024

Note that I have reset the benchmarks to the 1.0 release, before #1904, so any changes are currently from that PR and any followups that touch the core.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Save benchmark results over time in readable/plottable format
2 participants