Add systemd-sysext development install mode

[martinpitt]

Now that we don't install any suid root stuff any more and use fully
dynamic system users, we can build our git tree straight into a
systemd-sysext [1]. With that we can test all parts of cockpit
(including tls, ws, session, bridge, the login page, and systemd units)
in a safe manner on the host system. Nothing ever hits the disk (it's
installed to /run/extensions) and allows fast iteration.

This is primarily aimed at our documented development workflow of
building in a cockpit/tasks toolbox, but written in a generic way so
that it should work on Fedora, CentOS/RHEL, Debian/Ubuntu, and Arch, and
also directly from the host system. Use the "extension-release"
declaration to ensure compatibility between the build and host systems.

[1] https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html

https://issues.redhat.com/browse/COCKPIT-1208

---

This works really nicely for me -- testing and feedback appreciated!

documentation preview

Development: Add systemd-sysext install mode

If you want to develop Cockpit, you can now build and install cockpit's source code as a systemd-sysext. This allows you to test all parts of cockpit including the web server, login page, systemd units, and the bridge, in a safe manner on the host system. It is installed to /run/extensions, so does not change anything on disk. This provides very fast iteration without the need to use a VM.

See the documentation for details.

[martinpitt]

@mvollmer @allisonkarlitskaya that may be interesting for you. @garrett perhaps for you as well, as you occasionally work on the login page. @jelly I'm afraid this currently doesn't support an Arch host, but of course I'm happy to work with you to fix that 😉

[allisonkarlitskaya]

@martinpitt the selinux thing is a bit of a non-starter for me. I wonder if @travier has some ideas...

[martinpitt]

He actually does, see https://github.com/travier/fedora-sysexts?tab=readme-ov-file#know-issues . I have no idea how bad that is, so far it's a "works for me" thing. But I'll look into the workaround if that fixes the contexts. But not today, today is "everything exploded" all-hands piloting.

[martinpitt]

Actually our case isn't covered by any of the three items in https://github.com/travier/fedora-sysexts?tab=readme-ov-file#know-issues. cockpit ships its own SELinux policy, and there's no magic to apply that when installing into the sysext dir in an "upstream" way. This is surely fixable somehow, but TBH I have zero interest in this. This is supposed to replace the manual bind mounts, which would suffer from the exact same SELinux issues.

I'll update the documentation accordingly, and if not accepted, just close this and put it into my own personal script collection instead. (really, no hard feelings -- this is a debugging/development tool)

[jelly]

After testing the new anaconda installer I have second laptop with Fedora, so happy to give it a try.

[travier]

The way I create sysexts with correct SELinux labels is by installing the SELinux modules in a temporarily container and then relabeling the files in the sysext rootfs: https://github.com/travier/fedora-sysexts/blob/13703d1934f71b7f240d69102edd4a29798dd9af/sysext.just#L314

Any Fedora container within the same major release should do for that part, you don't need to strictly run it on the same content as the host.

Actually our case isn't covered by any of the three items in travier/fedora-sysexts#know-issues. cockpit ships its own SELinux policy, and there's no magic to apply that when installing into the sysext dir in an "upstream" way.

But indeed, once the policy modules are added on the system via the sysext, you still need to rebuild/reload the policy which is not done automatically right now. I have not yet started looking into this.

[martinpitt]

@jelly Reworked so that it should support Arch natively now. It should work both with "build on host" and also "build in some Arch container", but not "build in cockpit/tasks and run on arch" (that can't work). The main thing I'm unsure about is the extension-release.d/ file. $VERSION_ID must match between that file and the host, but Arch as rolling release has no VERSION_ID. So I'm not sure if "both are empty" works, or if that needs special treatment.

Are you interested in this feature, i.e. would it help you with development? Then I'm happy to work further on this.

[martinpitt]

Reworked with some fixes from live testing/debugging with @jelly . It now cleans up more robustly, handles a missing /run/extensions, and has a proper "stop" mode.

[jelly]

This is cool!