Hook up blueprint filesystem customizations

[bcl]

Really simple change :) The testing feels a bit klunky, suggestions welcome. I thought about checking the manifest, but it boots the image anyway so checking the output of df seemed to be the most straight forward way.

[achilleas-k]

One tricky part here is that we have rules in osbuild/images about what partitions a user can (not) add to avoid making unbootable images. Unfortunately those aren't exposed (yet) so we can't apply them here.

Maybe we can only allow cutomizing the size of / for now to unblock issues with images failing to build when the base container is too big and then work on bringing in the rules from images.

[bcl]

One tricky part here is that we have rules in osbuild/images about what partitions a user can (not) add to avoid making unbootable images. Unfortunately those aren't exposed (yet) so we can't apply them here.

It feels weird to essentially be partially re-implenting things here (and will be totally confusing to users) so I think exposing more of osbuild/images is the right way to go.

[bcl]

PR to move the policies into the public pkg - osbuild/images#395

[bcl]

NOTE: CI tests will fail until images is updated

[github-actions]

This PR is stale because it has been open 30 days with no activity. Remove "Stale" label or comment or this will be closed in 7 days.

[bcl]

I've lost track of the status of this, so if you want it don't wait on me :)

[cgwalters]

This looks pretty small and sane to me, I think we also just need to update the docs, right? (Can be done post-merge)

[cgwalters]

Hm, testing farm was failing with

OSError: [Errno 30] Read-only file system: 'boot.1'
But I'm not totally sure that was the root cause?

[cgwalters]

I've rebased this on git main, just fixing up the (textual, but not logical) conflict with the new "double rootfs size default") code. Not tested locally beyond building, but let's see what CI says!

[mvo5]

I've rebased this on git main, just fixing up the (textual, but not logical) conflict with the new "double rootfs size default") code. Not tested locally beyond building, but let's see what CI says!

Once we move to the bootc install to-filesystem flow (which is hopefully soon) we will no longer write a custom /etc/fstab which is crucial for this feature to work in the general case. I think we have some options:

1. add policy so that we only allow to customize the size of "/", "/boot" and disallow adding extra partitions (probably a good starting point)
2. support writing a custom /etc/fstab from images, I managed to find a bit of time today and looked over the ostree docs and played with guestfish with a fresh image that never booted. The only way to customize /etc/fstab I found was to write into the deployment root /ostree/deploy/default/deploy/d3d806...0/etc/fstab of the unbooted disk. Any other path ("/", "/sysroot", "/ostree/deploy/default/etc/fstab") I experiented with got "shadowed" or was wrong on the first boot. We do have support for this in osbuild, so adding this way of customizing is straightforward to add but I feel uneasy about it because it seems to me that bootc should be in control of it's content and just "side-loading" like this does break any integrity checking.
3. with (2) solved, it seems we also need some code to deal with custom mountpoints(?). I.e. it seems to support e.g. /data we also need to create the mountpoint in the deploy root?

Please let me know if I'm missing something here or thinking too complicated.

[cgwalters]

but I feel uneasy about it because it seems to me that bootc should be in control of it's content and just "side-loading" like this does break any integrity checking.

I feel like we keep re-treading this, not sure how to solve the logical disconnect. I'll try to write some more about this in the bootc docs. Now for sure I would like to support better things that just "unmanaged regular files" in /etc (configmaps perhaps or other things).

But for broad/generic use cases, we simply have to support machine-local state - and for maximum compatibility we can't just require everything to learn to do configmaps or sysexts or something that doesn't boil down to supporting vi /etc/somefile.conf.

For example Anaconda today is writing an /etc/fstab always...which tangentially I needed to do an elaborate workaround for a corner case and hopefully as we align things here when Anaconda learns to use bootc install to-filesystem too we can centralize this logic more.

The only way to customize /etc/fstab I found was to write into the deployment root /ostree/deploy/default/deploy/d3d806...0/etc/fstab of the unbooted disk.

The configuration definitely needs to be in the deployment root, yes.

Also BTW let's not get too hung up on /etc/fstab specifically as e.g. .mount units also make sense - but all of this is really just a special case of "config files in /etc/ and kernel arguments" which both can form machine-local state necessary for storage, networking etc.

[mvo5]

[..]

The configuration definitely needs to be in the deployment root, yes.

Also BTW let's not get too hung up on /etc/fstab specifically as e.g. .mount units also make sense - but all of this is really just a special case of "config files in /etc/ and kernel arguments" which both can form machine-local state necessary for storage, networking etc.

Thank you! Sorry that we keep coming back to this, but hopefully this was the disconnect I had. In my (wrong) mental model I assumed that there is

1. /ostree/deploy/default/deploy/ which shouldn't be touched
2. extra directory with overrides to the above dir to add local state like /etc/fstab (this is why I messed around in /etc on the not-yet-booted-disk)

It seems that ostree is smart enough to know if files in (1) get changed and tracks them as local state. I hope I got this right now and sorry that it took me a while.

[cgwalters]

It seems that ostree is smart enough to know if files in (1) get changed and tracks them as local state.

Yes, https://containers.github.io/bootc/filesystem.html#etc ➡️ https://ostreedev.github.io/ostree/atomic-upgrades/#assembling-a-new-deployment-directory (I'll expand this more over there)

ostree's semantics here are more arguably more "package system" like (vs "simple image system" as is emphasized by e.g. systemd's push for empty /etc) - and this definitely has tradeoffs, but again in a nutshell we support dropping arbitrary files there and they persist.

[cgwalters]

I rebased this again...I think it still makes sense. (Sorry about being initially semi-against...I had just really hoped we also started with the container-based configuration but that is just messier than I thought)

[mvo5]

I rebased this again...I think it still makes sense. (Sorry about being initially semi-against...I had just really hoped we also started with the container-based configuration but that is just messier than I thought)

Once we move to bootc install to-filesystem (hopefully really soon) this will need something like osbuild/images#597 first.

[github-actions]

This PR is stale because it has been open 30 days with no activity. Remove "Stale" label or comment or this will be closed in 7 days.

[mvo5]

I rebased this again (and slightly tweaked tiny bits). Unfortunately it needs more work because:

...
Installing image: docker://quay.io/centos-bootc/centos-bootc:stream9
ERROR Installing to filesystem: Verifying empty rootfs: Non-empty root filesystem; found "var"
Traceback (most recent call last):
  File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 53, in <module>
    r = main(args["options"], args["inputs"], args["paths"])
  File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 48, in main
    subprocess.run(pargs, env=env, check=True)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['bootc', 'install', 'to-filesystem', '--source-imgref', 'containers-storage:[overlay@/run/osbuild/containers/storage+/run/containers/storage]3b612dd1fae2437c00ae3187d0e63daa7a94711560fb1712389edd4121668c96', '--skip-fetch-check', '--generic-image', '--karg', 'rw', '--karg', 'console=tty0', '--karg', 'console=ttyS0', '--karg', 'systemd.journald.forward_to_console=1', '--target-imgref', 'quay.io/centos-bootc/centos-bootc:stream9', '/run/osbuild/mounts']' returned non-zero exit status 1.

I attach the generated osbuild manifest with the exiting code:
fs-customizations.json

The issue is that we run "org.osbuild.bootc.install-to-filesystem" with all the mounts, i.e. "/", "/boot" but also "/var/log" now.
So AFAICT "images" need sto be changed so that:

1. the "install-to-filesystem" stage only takes the "essential" mounts (/, /boot/, /boot/efi)
2. after "install-to-filesystem" ran we need a "org.osbuild.mkdir" stage for the extra mount points that also only mounts the "essential" mounts

That should be enough (but is a bit of extra work).

[mvo5]

As suggested by @ochosi I tweaked the PR now so that we initially only allow to customize "/" and "/boot" which will already cover many use-cases. Then we can continue on this once we have updated images to support what bib needs.

[mvo5]

Fwiw, this looks good to me now but please count me as only a a +0.5 because I tweaked/rebased this code a bit.

[achilleas-k]

I agree, let's enable / and /boot and enable more later.