test: integration tests #348
Conversation
|
Thank you for this! I appreciate the consideration to other distros, but I think focusing on Fedora/CentOS/RHEL is fine, as integration tests are by their nature tied to the target distros. We would of course welcome contributions of integration tests for other distros if others have an interest in that. The structure looks good to me. I think fuzzing systemd1 makes sense as we do parse all these messages and it will get good coverage for our parser. A natural extension would be to fuzz also the dbus control interface as that's what dbus-broker itself implements, but that could be done in a follow up. Once you are happy with this PR and mark it ready for review I'd be happy to merge it. @dvdhrm ? |
Let's follow-up on 0a0b0fa and introduce a second Packit job that runs integration tests on the just built RPMs in Testing Farm [0]. The test organization and execution is handled by tmt [1]. To not unnecessarily pollute the dbus-broker's git root, the tmt root is initialized in the test/integration/ subdirectory (meaning that for the tmt commands to work correctly you need to be somewhere below this directory). Since there are no pre-existing integration tests, add a simple one that runs dfuzzer on the org.freedesktop.systemd1 D-Bus interface to exercise dbus-broker a bit. A short README file is provided as well with instructions on how to easily add a new test case, and with some links where to find more information about tmt. [0] https://packit.dev/docs/configuration/upstream/tests [1] https://tmt.readthedocs.io/en/stable/overview.html
I beefed up the commit message a bit, so it should be ready to go. As discussed internally, I'll look into running the reference tests next once I have a bit of spare time. |
|
The integration tests appear to be stuck? |
There was an unexpected outage caused by a wild workaround (https://status.testing-farm.io/issues/2024-03-12-long-queue/). The issue should be resolved, but there's still a long queue Testing Farm has to chew through, so it might take a bit (but the runners seem to have already picked our jobs as I'm writing this). |
| # dbus-broker integration test suite | ||
|
|
||
| dbus-broker's integration test suite uses tmt (Test Management Tool [0]) to organize and run tests. Since tmt | ||
| offers a _lot_ of features, this document pinpoints the most _interesting_ ones to get stuff up and running |
There was a problem hiding this comment.
@mrc0mmand just out of curiosity I wonder if it's possible to build things with ASan/UBSan and install that to the test environment somehow? Valgrind would be nice too.
(As far as I can remember it came up recently elsewhere in the context of moving stuff like that from GitHub Actions to the Packit infrastructure)
There was a problem hiding this comment.
That's definitely on my TODO list (i.e. running dbus-broker with ASan/UBSan). I'm not sure how I'll do that, but that's something for my future me to figure out :)
Let's follow-up on 0a0b0fa and introduce a second Packit job that runs integration tests on the just built RPMs in Testing Farm [0].
Since there are no pre-existing integration tests, add a simple one that runs dfuzzer on the org.freedesktop.systemd1 D-Bus interface to exercise dbus-broker a bit.
[0] https://packit.dev/docs/configuration/upstream/tests
This is the continuation of #347 which adds an integration test "infrastructure" with a single simple test, and some notes on how to add new tests. I followed the original intention from #279 and used tmt here, but separated the tmt tree into a subdirectory, so the fmf metadata don't pollute the project root (just a personal preference, feel free to rearrange).
I'm opening this as a draft, since there's one issue with this approach - tmt, in its entirety, works properly only on RHEL, CentOS, and Fedora, since it uses
dnf/yumto install packages. AFAIU this doesn't affect local runs (see https://tmt.readthedocs.io/en/stable/questions.html#using-tmt-outside-of-fedora-centos-and-rhel-distribution), so it should still be possible to run the tests to some degree, but I personally have no experience with tmt outside of the RH ecosystem. This may or may not be a an issue; on one hand there is no pre-existing integration test suite, so nothing will suddenly break, and as tmt is fully supported by the Fedora CI infra the integration with is then very simple. However, other distributions might run into some obstacles when/if they try to run it on their side. So treat this PR as a PoC that shows how it could look like, but feel free to discard if the limitations are not acceptable.