issue again likely with hardlinks

[cgwalters]

Deploying quay.io/flightctl/flightctl-agent-centos:bootstrap with bib fails like this:

ostree container image deploy --imgref=ostree-unverified-image:dir:/tmp/tmpl16zdc_g/image --stateroot=default --target-imgref=ostree-unverified-registry:quay.io/flightctl/flightctl-agent-centos:bootstrap --karg=rw --karg=console=tty0 --karg=console=ttyS0 --karg=root=LABEL=root --sysroot=/run/osbuild/tree
error: Performing deployment: Importing: Parsing layer blob sha256:9de5a4ff2cc05f1dc30ea793a9a10bca5935ea3f60965962ac68c5b655a68b32: : Processing tar via ostree: No such file or directory (os error 2)

At first, I thought this was another echo of coreos/rpm-ostree#3827 (comment) ➡️ ostreedev/ostree-rs-ext@883a79e

Digging into the relevant layer, it's the one corresponding to this run invocation:

RUN dnf install -y flightctl-agent && \
    systemctl enable flightctl-agent.service

Digging in a bit...the first thing I notice here is that building that with podman here works fine. And yes...it looks like that same issue, because in the remotely built image we have:

hrw-r--r-- 0/0               0 usr/lib/sysimage/rpm-ostree-base-db/rpmdb.sqlite link to sysroot/ostree/repo/objects/cd/3e1735c45d8013153d7808e4efb834b0478016266aaf5f740c1ff0e8262119.file
hrw-r--r-- 0/0               0 usr/lib/sysimage/rpm-ostree-base-db/rpmdb.sqlite-shm link to sysroot/ostree/repo/objects/bf/13d8fe63612425b89f526a906f06359a1a71ac71902caef8a7160558d91187.file

But I thought we'd fixed this one....ah yes, in ostreedev/ostree-rs-ext@43e1648 and also in ostreedev/ostree-rs-ext#558

[cgwalters]

This is claimed to be a recent regression (rotating tags would help prove/disprove).

[cgwalters]

What's really weird is that so far I can't reproduce this with bootc switch from a running host, or even directly doing an ostree container deploy. It somehow only reproduces in bib so far which is really weird because it should literally be doing the same thing.

[oglok]

I’m struggling to understand something. If I reference by digest a centos bootc image from a week ago, that worked at that time, why shouldn’t it work today? is there something during the build process that gets downloaded on the go and pulling the latest?

Let's say my Containerfile is now:

FROM quay.io/centos-bootc/centos-bootc:stream9-1709003520
ADD etc etc

RUN dnf install -y flightctl-agent && \
    systemctl enable flightctl-agent.service

That digest is from 3 days ago.
Podman build works, but BIB disk creation doesn't with the same failure.

[cgwalters]

It's definitely possible that if you're using a new bib image that that is somehow involved in the failure.

Also

stream9-1709003520

Oh wow, I had no idea this existed, it must be something that the tekton release flow is doing automagically? Will comment over in #189

[oglok]

BIB latest image is from 7 days ago, and this was working 3 days ago...

[cgwalters]

After a bunch of debugging, ostreedev/ostree-rs-ext#605 pointed out that the problem is that osbuild is not creating a /var/tmp in its bwrap stages.

[cgwalters]

Closing in favor of osbuild/bootc-image-builder#223

[cgwalters]

@oglok for my own sanity...did you end up rebuilding that image with podman? I no longer reproduce the failure.

[oglok]

@oglok for my own sanity...did you end up rebuilding that image with podman? I no longer reproduce the failure.

Yes, it works now @cgwalters
Thanks a lot!