Run tests via qemu on multiple distros/kernels

[geofft]

#34 (comment) had a proposal for running tests via qemu, which we punted on, deciding to just use the host kernel. It turns out that Travis's kernel is pretty old (4.15 even on Bionic), see #229. While we could resolve that by just switching to some other CI with a newer kernel (e.g., GitHub Actions?), I'd like to also think about actually doing testing on multiple distros and multiple kernels per distro via qemu.

What I'm thinking, loosely, is that we do the build inside Docker, install/download a kernel inside the Docker container, and then spin up qemu using 9p or NFS rootfs pointed at the container. But it's possible that if we boot a real VM using hardware-assisted virtualization, that will be fast enough that just doing normal builds inside the VM would work fine.

Here's some distros we'd want to support, based on tickets people have filed. Ideally we should pin to specific snapshot or at least stable versions of the distro so we don't have to chase CI failing just because something got upgraded. It may be enough to do that by just building our Docker images separately from CI and only updating them manually.

• Debian
• Debian backports (hello-world build and run on Debian 5.6.14 kernel 5.6.0-2-amd64 #219)
• Ubuntu
• Fedora
• Perhaps Mint (Error with kernel-cflags-finder on make #225) but maybe that's close enough to Ubuntu
• Arch (rolling, but if we need snapshots there's https://wiki.archlinux.org/index.php/Arch_Linux_Archive)
• Manjaro (rolling, and unfortunately does not have anything like the Arch Linux Archive)

Note that Manjaro currently won't work because their kernel has structleak (#152). We should add it in as soon as we get it working.

Also, we should find some way of testing very recent LLVM and very recent kernels in a cron-style CI, that is, not blocking merges, but using the same scripts to run the tests. Probably Debian unstable would work.

Some things we should look at for inspiration:

• https://people.kernel.org/mcgrof/kdevops-a-devops-framework-for-linux-kernel-development (https://github.com/mcgrof/kdevops)
• https://kernelci.org/
• https://github.com/python-trio/trio/blob/master/ci.sh , which runs qemu in Travis in roughly the way we'd want, and has solved a couple of the annoying problems (getting /dev/kvm access, transferring files with 9P, etc. - note they don't use 9P root, which is vaguely nice because you don't have to repack the initramfs)
• https://github.com/ClangBuiltLinux/continuous-integration , which builds the kernel from scratch - we should have some coverage of LLVM-built kernels, and I'm not sure there's a better way (unless we can steal an Android x86 nightly or something???)

[geofft]

tuxmake/tuxbuild https://linuxplumbersconf.org/event/7/contributions/797/ are also worth looking at.