Read-only /sysroot prevents pulling down new commits via ostree CLI

[jlebon]

A local build with git master rpm-ostree and git master ostree (which has ostreedev/ostree#1767) errors on:

[root@coreos ~]# ostree pull --commit-metadata-only fedora:fedora/x86_64/coreos/stable
error: Cannot write to repository: Read-only file system

Seems like we need to remount rw in the pull paths too. Hmm, or maybe it's something that ostree and rpm-ostreed should just do upfront instead of only in specific paths?

Note the testing release 31.20191217.2.0 already has sysroot.readonly=true (coreos/coreos-assembler#736). So we'll want to fix this before the ostree code drops (because otherwise, nodes won't be able to update).

[jlebon]

Ahh actually, rpm-ostree rebase does work, so this shouldn't break auto-updates. So I think this is just scoped to the ostree CLI then.

[cgwalters]

We probably should change ostree pull to do that indeed. (This is a bit unfortunate as it layer-crosses the otherwise clean separation between sysroot and repo, but eh)

But we're just using the regular rpm-ostree upgrade path which remounts the sysroot before it does much else. IOW, this only breaks people who manually invoke ostree pull right?

[jlebon]

IOW, this only breaks people who manually invoke ostree pull right?

Yes, that's my conclusion as well. Much less worrying!
We can move this issue to the ostree repo if you prefer.

There's another use case, which is ostree checkout to somewhere under /ostree/repo to make sure one gets hardlinks (which is actually how I hit this first).

[jlebon]

Now that 2020.2 has hit the stable repos, it's entered FCOS bodhi-updates, and kola is reporting more issues there:

--- FAIL: rpmostree.upgrade-rollback (25.76s)
    --- FAIL: rpmostree.upgrade-rollback/upgrade (0.17s)
            cluster.go:122: error: open(O_TMPFILE): Read-only file system
            cluster.go:143: "sudo ostree refs --create local-branch 62eabc11696c3744b9301db0e3a3235d5f0a0c54b73767a0046bd0ad1eb6f334" failed: output , status Process exited with status 1
    --- SKIP: rpmostree.upgrade-rollback/rollback (0.00s)
            cluster.go:50: A previous test has already failed

--- FAIL: ostree.remote (29.57s)
    --- PASS: ostree.remote/add (0.13s)
    --- PASS: ostree.remote/list (0.11s)
    --- PASS: ostree.remote/show-url (0.10s)
    --- PASS: ostree.remote/refs (1.09s)
    --- PASS: ostree.remote/summary (2.05s)
    --- FAIL: ostree.remote/delete (0.29s)
            cluster.go:122: error: unlink(custom.trustedkeys.gpg): Read-only file system
            cluster.go:143: "sudo ostree remote delete custom" failed: output , status Process exited with status 1

--- FAIL: rpmostree.install-uninstall (26.64s)
    --- FAIL: rpmostree.install-uninstall/install (0.53s)
            cluster.go:122: error: Read-only file system
            cluster.go:143: "sudo rpm-ostree install bird" failed: output , status Process exited with status 1
    --- FAIL: rpmostree.install-uninstall/uninstall (0.48s)
            cluster.go:122: error: Read-only file system
            cluster.go:143: "sudo rpm-ostree uninstall bird" failed: output , status Process exited with status 1

Let's look into those, though for now let's unblock lockfile bumps: coreos/fedora-coreos-config#300

[cgwalters]

Yeah, working on this in ostreedev/ostree#2027 it's just ugly.

[cgwalters]

PR in ostreedev/ostree#2027 is tested and working.

[jlebon]

coreos/fedora-coreos-config#302
coreos/fedora-coreos-config#303

I think we can close this issue after those are merged!

[dustymabe]

merged... should we close?

[jlebon]

Yup, let's close this!

(There's still some work ongoing in coreos/rpm-ostree#2023 to adapt the rpm-ostree testsuite, but that's clearly not an FCOS problem.)