ci: Test more targets, move the testing logic to a script

[vadorovsky]

Cover all possible targets. The ones which fail to build, are disabled with an appropriate comment explaining the error.

To make the tests easier to reproduce locally, provide the container_test.sh script and move the most of logic there.

[vadorovsky]

The last red build on main took 40 minutes, which means that's the time for the toolchain to be ready. Workflows can run for 3-6 hours, so I will try to build the whole system here. If that takes too long, I will fall back to installing zstd (or some other package).

[thesamesam]

I would like to cover a lot more targets so maybe we could only bother with the full @system test for selected ones?

[thesamesam]

There is no sparc* port for musl (or musl port for sparc*), unfortunately.

[vadorovsky]

Good point, removed

[vadorovsky]

Now that we have some jobs failing, I realized that the CI jobs are hard to debug, because the outputs of emerge are redirected to log files. Would you be open to having an option to print the failure logs to terminal (like --log-errors or --log-failures)? I could add it in a separate commit.

I will also make the whole script fail realy in the next push.

[thesamesam]

Now that we have some jobs failing, I realized that the CI jobs are hard to debug, because the outputs of emerge are redirected to log files. Would you be open to having an option to print the failure logs to terminal (like --log-errors or --log-failures)? I could add it in a separate commit.

I will also make the whole script fail realy in the next push.

I suppose, not sure if I love the name though. --capture-errors maybe.

[thesamesam]

This needs to go too.

[vadorovsky]

Now that we have some jobs failing, I realized that the CI jobs are hard to debug, because the outputs of emerge are redirected to log files. Would you be open to having an option to print the failure logs to terminal (like --log-errors or --log-failures)? I could add it in a separate commit.
I will also make the whole script fail realy in the next push.

I suppose, not sure if I love the name though. --capture-errors maybe.

Nevermind, there is --show-fail-log already.

[thesamesam]

I had no idea!

[vadorovsky]

@thesamesam Are you by any chance able to restart single jobs? These two failed just because of flakyness of emerge --sync:

https://github.com/gentoo/crossdev/actions/runs/12005351093/job/33461826634?pr=28
https://github.com/gentoo/crossdev/actions/runs/12005351093/job/33461827068?pr=28

These targets worked before.

If not, I will force push, but I'm wondering if we can avoid that. I'm usually able to restart jobs in repositories I own.

In general, I think we're almost there. Once everything is green, I think we are good to go.

[thesamesam]

Unfortunately, I can't (the permissions most people have are very limited on the gentoo github repos). But maybe it's a sign of a real bug, actually (or a real risk of flakiness): let's change it to emerge-webrsync instead so we get the nightly snapshots and we're guaranteed all jobs used the same tree state?

[thesamesam]

Looks like we're good to go? Thanks a lot for your work. I'll wait until you switch to ready before merging.

I have always been a bit uneasy about crossdev changes because of the lack of such automated testing.

[vadorovsky]

I marked it as ready. Thank you for your review! Looking forward to test my LLVM-related changes with it.

BTW, would you be open for using Github Actions in the main gentoo repo as well? I know that there is Tinderbox and https://qa-reports.gentoo.org/. But from my understanding, Tinderbox tests mostly already merged changes and the QA reports on pull requests are mostly syntax linters? I think a Github Action which tries to ebuild the given package in containers would prevent issues like https://bugs.gentoo.org/943129.

[thesamesam]

I think that's a more controversial question that would require broader discussion. It would be nice for PRs at least to help catch errors like that though, I agree. There have been a lot of different efforts in the past to have tinderboxing-setup-as-regular-CI like Zorry is doing at the moment with buildbot, but relying on github actions would kind of be advantageous here anyway for untrusted/arbitrary PRs anyway...