Build and fetch errors with clux/muslrust:1.60.0

[Nutomic]

We are suddenly getting some weird failures in our build using clux/muslrust:1.60.0 on cloud.drone.io. First one is this:

+ git fetch --tags

fatal: unable to access 'https://github.com/LemmyNet/lemmy.git/': getaddrinfo() thread failed to start

And second, this:

+ cargo check -p lemmy_api_common
error: could not execute process `rustc -vV` (never executed)

Caused by:
  Operation not permitted (os error 1)

A few days ago the same ci build worked fine, was the image updated to a broken one in the meantime? For now i downgraded to muslrust:1.59.0 and it works fine.

[clux]

Ohh. That might be related to the os upgrade yeah. Didn't test that git worked. I'll have a quick look later.

[charlesndalton]

Also randomly getting "error: could not execute process rustc -vV (never executed)"

[Nutomic]

FYI it seems the error with getaddrinfo() thread failed to start only happens when running in drone.io (and happens every time). But local builds work fine.

[clux]

Ah, thanks a lot for this update. I was struggling to reproduce anything myself.

For anyone with issues with the new jammy version it would be helpful to have a Dockerfile and a link to a how it's built (e.g. github action invocations).

[Nutomic]

We are using this config for drone.io: https://github.com/LemmyNet/lemmy/blob/main/.drone.yml

You should be able to reproduce by copying it into a new project (or forking), and changing the Docker image version. Then enable cloud.drone.io CI for the repo. The error already happens in the first step at git fetch --tags (example output).

I can also test things if you tell me how to get useful info out of it.

[xD3CODER]

Same issue for me

[clux]

@xD3CODER : you also on drone.io?

not able to do a full test against drone myself since they are no longer accepting new registrations, but can see if i can reproduce via a test case on github actions by emulating what lemmy is doing in gha.

[xD3CODER]

@xD3CODER : you also on drone.io?

not able to do a full test against drone myself since they are no longer accepting new registrations, but can see if i can reproduce via a test case on github actions by emulating what lemmy is doing in gha.

For me it happens on local Docker build

[clux]

@xD3CODER : oh can actually be very helpful. Do you have something public (or a Dockerfile) and what command that is failing (if it's not a git fetch error)?

[clux]

Ultimately tests manage to run git fetch fine, both locally and in CI (#100). Possibly this is permission related. The users and groups haven't changed, but maybe caches from a previous container is interacting badly 🤔 maybe cargo clean first? (saw a similar error via rust-lang/cargo#2305)

For the git error, I'm not sure what's exactly is happening there. git was upgraded. Maybe something to do with the git config? People are seeing the same git error in different CI systems. This post suggests setting export SYSTEMD_SECCOMP=0.

[Nutomic]

@clux I also have a selfhosted drone installation which gave the same errors. Just added your username to allowlist, so you should be able to sign in using Github at https://drone.yerbamate.ml and enable it for your repo.

[xD3CODER]

@clux The following Dockerfile fails at cargo build with

error: could not execute process `rustc -vV` (never executed)

Caused by:
  Operation not permitted (os error 1)

FROM clux/muslrust:latest as builder
WORKDIR /app
RUN cargo new --bin api
WORKDIR /app/api
RUN cargo build --release

[clux]

Thanks both. I have added this trivial dockerfile as a test case in #100 although it is passing both on github actions and locally. My hunch is that this is a security restriction somewhere.

Can't look more until tonight, but any info about setups (id (showing group membership), os setup (selinux / distro)) is helpful in the meantime. Will try the drone setup tonight/tomorrow.

[clux]

So... drone actually works fine for me with clux/muslrust:1.61.0; https://drone.yerbamate.ml/clux/muslrust/1/1/3 using simplified drone ci config via 43287ae

[Nutomic]

Its strange, drone.yerbamate.ml seems to have fixed itself. Dont think i changed anything there except install system updates and reboot. But the same error is still happening on cloud.drone.io.

I opened a thread about this in the Drone Support forum.

[jrd-rocks]

@xD3CODER I had the same problem, after searching a bit (apt-get upgrade also failed, eg https://askubuntu.com/questions/1408528/apt-update-on-ubuntu22-04-results-in-error-100-on-some-docker-hosts) i found that docker apparently was bugged. I upgraded docker 2.10.7 to 2.10.12 (was docker-ce, i installed docker.io which is in the ubuntu repo) and the problem went away.

[xD3CODER]

Thank's for your help @jrd-rocks .
I think my problem is due to the fact that I updated my Ubuntu distribution (focal -> jammy) without upgrading Docker

[clux]

Given the updates on this, going to close this issue. Seems like the new ubuntu incidentally stumbled upon something weird, that's not really our fault.