-
Notifications
You must be signed in to change notification settings - Fork 92
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Faster CI: Sparse run matrix, cancel previous jobs on the same branch #635
Conversation
…cel previous runs on the same branch
Great! Thanks a lot! This should help to speed up our CI. One note, though. clang-19 may still be broken (llvm/llvm-project#110231) |
Ah right, I observed the same locally, and I thought I'd give it a shot in CI. I'll remove Clang 19 then for now. |
70ee158
to
3474844
Compare
3474844
to
e464677
Compare
…d; also, fix freestanding
3c52347
to
33c82fc
Compare
33c82fc
to
b35e241
Compare
Finally, CI passes! As always, took me a lot of trial an error until it fully worked online in CI - do you want me to squash the history here? Overall, the changes here seem to be extraordinarily effective: With the exception of the tasks on macos runners (none of which have been picked up yet), all checks passed within less than 15 mins of the commit here. While, as of now, on @mpusz's and on @chiphogg's branches, there are still queued workflows from 10 hours ago and earlier. The immediate cancelling of earlier workflows on the branch obviously helps a lot, but it leads to commits being marked as "failed" even if no job actually failed. This would only affect commits made in rapid succession, for which we usually don't care if intermediate ones don't go green. So I believe this fixes #633 and is ready to merge. Unfortunately, workflows started on branches prior to this PR will not get cancelled, because those workflows are not associated with any "concurrency group", but its just a couple more hours until those will timeout :-). |
@burnpanck, thanks a lot! Do you think that it is ready to merge? Maybe we should disable Mac OS-based CI for a while? |
I think we can merge. I was trying to find information about some site-wide outage on GitHub's hosted macos runners, but I couldn't find anything on the GitHub status page nor on reddit. If they are limiting the number of active jobs on macos per repo, then hopefully, in a couple hours when all the past ones timeout, we might finally catch up to the new commits (where we now only have ~8 jobs on macos as it seems). If tomorrow things still don't progress, I think we should assume severe performance issue in GitHub and disable the macos runners. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should go a HUGE way towards restoring sanity in terms of workload, without meaningfully sacrificing our coverage. Thank you @burnpanck!
(I didn't review the code changes, but @mpusz has already improved, and I'm excited to see this change.)
with: | ||
python-version: 3.x | ||
- id: set-matrix | ||
run: python .github/generate-job-matrix.py --preset clang-tidy --seed 42 --debug combinations counts |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is some GitHub Actions wizardry I had not seen before. Cool!
@burnpanck, is there a way to ensure that there is at least one configuration that tests |
Also, if other compilers will get this configuration it should not enable |
Absolutely. The specific configuration is generated in The code here should be exactly compatible with previous versions of the GitHub Action scripts (with the exception of the renaming from Note also that this script always uses a fixed random seed (can be changed using |
I think
Yes, it was needed before but not anymore. I left the option for the future, though.
Do you mean that we can repro a specific CI run, or does it mean that every CI is the same and we are not testing other configurations? |
Currently, thats part of the
Done: #637.
Every CI is the same; we're not testing other configurations. I considered that preferrable for reproducibility reasons, but it would be easy to change. |
We can rename |
This PR starts using a dynamic run matrix in the four heavier workflows.
It uses a small python script to generate a subset of all possible combinations. For the two biggest workflows (Conan and CMake), it currently employs the following strategy:
std::format
where supportedgsl-lite
This currently generates 26 different combinations, out of 261 total acceptable combinations (I added Clang 19 to the test-set).
Hopefully, the python code to generate the run matrix is readable enough.
Another feature I added is to use concurrency groups to cancel previous runs on the same branch that haven't completed yet