sentry-cli >= 2.21.3 does not work with rootless docker

[Snack-X]

Environment

sentry-cli >= 2.21.3 + rootless docker

Steps to Reproduce

1. Install Docker in rootless mode
2. Pull some image (eg: python:3.12) and start new container (docker run --rm -it python:3.12 bash)
3. Install sentry-cli >= 2.21.3 (curl -sL https://sentry.io/get-cli/ | SENTRY_CLI_VERSION=2.21.3 bash)
4. Run sentry-cli info

Expected Result

No error

Actual Result

(failure on authentication: API request failed)

Logs

$ docker version
Client:
 Version:           25.0.2
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        29cf629
 Built:             Thu Feb  1 00:22:06 2024
 OS/Arch:           linux/amd64
 Context:           rootless

$ docker pull python:3.12
$ docker run --rm -it python:3.12 bash

# curl -sL https://sentry.io/get-cli/ | SENTRY_CLI_VERSION=2.21.3 bash
(snipped)
Installed sentry-cli 2.21.3
Done!

# sentry-cli --version
sentry-cli 2.21.3

# SENTRY_LOG_LEVEL=debug sentry-cli info
  DEBUG   2024-02-06 --:--:-- sentry-cli version: 2.21.3, platform: "linux", architecture: "x86_64"
  INFO    2024-02-06 --:--:-- sentry-cli was invoked with the following command line: "sentry-cli" "info"
  DEBUG   2024-02-06 --:--:-- request GET https://sentry.io/api/0/
  DEBUG   2024-02-06 --:--:-- retry number 0, max retries: 0

Sentry Server: https://sentry.io
Default Organization: -
Default Project: -

Authentication Info:
  Method: Unauthorized
  (failure on authentication: API request failed)

  INFO    2024-02-06 --:--:-- Running update nagger update check
  DEBUG   2024-02-06 --:--:-- request GET https://release-registry.services.sentry.io/apps/sentry-cli/latest
  DEBUG   2024-02-06 --:--:-- retry number 0, max retries: 0

---

• Related with: [vite] Couldn't resolve host name (Could not resolve host: sentry.io) #1843

---

While updating our nuxt web app's sentry module(@nuxtjs/sentry), we've also updated @sentry/webpack-plugin from 1.x.x to 2.x.x.

After this change, we started to see our CI/CD pipeline failing to publish releases and sourcemaps. The only error message we could see was error: API request failed caused by: [6] Couldn't resolve host name (Could not resolve host: sentry.io).

Few hours of tests and bisection narrowed down to combination of rootless docker (used in our Actions runner) and sentry-cli >= 2.21.3. Standard docker installation works just fine, and older version of sentry-cli also has no problem -- which is the fix we chose.

We tested multiple configurations of OS and architectures, which all had same result:

• Ubuntu 22.04 on AWS EC2 t3 (x86_64)
• Ubuntu 22.04 on AWS EC2 t4g (arm64)
• Amazon Linux 2023 on AWS EC2 t3 (x86_64)
• Ubuntu 22.04 on local VM (x86_64)
• (plus Docker 23.x, 24.x, 25.x)

I think some low level change made between 2.21.2 and 2.21.3 affected how DNS resolution works and created some kind of incompatibility.

It could be an issue of docker, it could be both, I don't know at this point. I hope this information is helpful.

[szokeasaurusrex]

@Snack-X thank you for the very detailed issue report!

It appears that we did make some changes to the compiler optimizations from version 2.21.2 to 2.21.3. However, we reverted one of the changes in 2.23.0 (commit 59fcf92) because it was causing problems when Sentry CLI was run in Xcode Cloud.

Have you tried running sentry-cli ≥2.23.0 in rootless Docker yet? If not, please try again with sentry-cli ≥2.23.0, since it is possible that the fix from 59fcf92 will fix your problem, too. Otherwise, if you still have trouble with sentry-cli ≥2.23.0, please let me know, and I will investigate further.

[Snack-X]

Thanks for the response.

That version was found with bisection from 2.0.0 to 2.28.0, so I think I've already tried >= 2.23.0. But I'll check it again just in case.

[szokeasaurusrex]

Okay, please let me know once you have tried ≥2.23.0 again and have confirmed whether the issue is still occurring in those versions

[Snack-X]

I can confirm both 2.23.0 and latest (2.28.0) still have same issue.

# curl -sL https://sentry.io/get-cli/ | SENTRY_CLI_VERSION=2.23.0 bash
Installed sentry-cli 2.23.0
Done!

# sentry-cli --version
sentry-cli 2.23.0

# sentry-cli info
Sentry Server: https://sentry.io
Default Organization: -
Default Project: -

Authentication Info:
  Method: Unauthorized
  (failure on authentication: API request failed)

# curl -sL https://sentry.io/get-cli/ | SENTRY_CLI_VERSION=2.28.0 bash
Installed sentry-cli 2.28.0
Done!

# sentry-cli --version
sentry-cli 2.28.0

# sentry-cli info
Sentry Server: https://sentry.io
Default Organization: -
Default Project: -

Authentication Info:
  Method: Unauthorized
  (failure on authentication: API request failed)

[Snack-X]

• DNS errors in binaries built with muslrust when run in rootless Docker clux/muslrust#122
• github.com not reachable from alpine image alpinelinux/docker-alpine#155 (also linked in issue above)

I've also found these issues which be related to this one, since sentry-cli is built with musl.

Looks like musl has problem with DNS resolving.

[szokeasaurusrex]

Thank you for the information and for linking those issues! I will investigate further to see whether we can fix this somehow, or whether this is something that needs to be fixed in musl

[darthf1]

Thanks @Snack-X for debugging and linking to issue #1843, I can confirm my issue is the same :) The error message for me is thrown in a Gitlab pipeline, using the docker:dind-rootless image to build my docker image. I have pinned my sentry-cli version to 2.21.2 currently.

[szokeasaurusrex]

Also I see that DNS over TCP was added to musl recently. @szokeasaurusrex do you know how to track down what version of musl goes into a given sentry-cli version?

@jrr I am unsure; a codebase-wide search for "musl" does not yield any results containing a version number

[Snack-X]

Other than f226a6e, I don't see any other commits related to musl. Although it doesn't make sense to me, but I'm not an expert in this area.

Previous docker image, getsentry/rust-musl-cross (repo) appears to be built in 2023-04-something with latest musl version, which would be 1.2.3 (released 2022-04-07).

2.21.3 was released in 2023-11-09, and I think implies it was built with latest build of new docker image, messense/rust-musl-cross (repo), at that time.

Although latest musl version at that time is 1.2.4 (released in 2023-05-01) which could have caused this issue, that build also appears to use 1.2.3, what?

I have no idea.

[jrr]

Good finds, @Snack-X . Summarizing my understanding:

• sentry-cli Linux binaries are built with a Docker container (CI build.yml), which bakes musl into the resulting sentry-cli binary.
• Sentry used to maintain their own getsentry/rust-musl-cross Docker image for this (docker hub, git repo)
• In Nov. 2023, the project switched to using a third-party Docker image (sentry-cli PR, messense/rust-musl-cross docker hub and git repo)
• As of today, that third-party docker image is a couple versions behind on musl:
  • messense/rust-musl-cross is on musl 1.2.3 from 2022 (ref)
  • The latest musl is 1.2.5 from Feb 2024, which notably includes improvements to DNS resolution.

A couple ideas for how Sentry can get get musl's DNS improvements into sentry-cli binaries:

• contribute to the upstream image, updating their musl
• switch back to your own sentry-maintained docker image, and this time keep up on maintenance :)

[szokeasaurusrex]

Yes, thank you both for this information! I wonder whether simply reverting f226a6e could solve this problem – perhaps we could try this before we implement one of the other solutions you proposed, @jrr, so that we can verify that this commit was indeed the cause?

I am unaware of the reasoning behind the change in f226a6e though, so I will first confirm with @loewenheim (who authored the change) whether there is any reason we should not revert that commit.

[loewenheim]

I think pushing for a musl update in the upstream image (PR by @jrr here) is a good idea.

[joekeilty-oub]

Thank you all for your work in investigating this issue!

I have been experiencing the exact same symptoms when running GitLab CI jobs within a docker container (based on debian slim) that runs sentry-cli via the npm package https://www.npmjs.com/package/@sentry/cli

Would we be able to experience the problem described by jrr in #1929 (comment) under these circumstances?

I had spent a lot of time looking into docker DNS resolution, the DNS set up on the build runner server, EDNS etc.etc. but is the fundamental issue how sentry-cli binary itself is performing the DNS resolution?

Happy to test a solution and report back if so 🫡

[szokeasaurusrex]

@joekeilty-oub From my understanding of the situation, it seems like it could be possible that the cause of your problem is the same since you are running the CLI inside a Linux Docker image. Let's see whether we are able to get this PR merged and whether that change ends up solving your issue

[mzavattaro]

@szokeasaurusrex Is this issue related to the error thrown when using sentry-cli in a Docker container? I've got it installed as a dependency as part of @sentry/nextjs but because it's a macOS binary trying to run on Alpine Linux, it fails. Is there a solution to this if it's not related?

0.775    Creating an optimized production build ...
10.20 Failed to compile.
10.20
10.20 Sentry CLI Plugin: Command failed: /app/node_modules/@sentry/cli/sentry-cli releases new IALP9dRCjJokMal0t774p
10.20 Cannot run macOS (Mach-O) executable in isolated machine: Exec format error
10.20
10.20 Sentry CLI Plugin: Command failed: /app/node_modules/@sentry/cli/sentry-cli releases new IALP9dRCjJokMal0t774p
10.20 Cannot run macOS (Mach-O) executable in isolated machine: Exec format error
10.20
10.20
10.20 > Build failed because of webpack errors
------
Dockerfile:15
--------------------
  13 |     COPY . .
  14 |     # Build your Next.js app
  15 | >>> RUN npm run build
  16 |
  17 |     # Stage 2: Running the app
--------------------
ERROR: failed to solve: process "/bin/sh -c npm run build" did not complete successfully: exit code: 1

[szokeasaurusrex]

@mzavattaro as far as I can tell, this looks like a separate issue. I don't recognize any of the error messages, and to me it looks like quite possibly the problem is occurring before Sentry CLI even gets called. Although, it's hard for me to tell since I'm not familiar with your particular setup.

[szokeasaurusrex]

One thing folks still experiencing this issue might be able to try, while we wait for an upstream fix in musl, would be to try compiling Sentry CLI from source locally, and running the produced binary on your system.

If the command succeeds on the locally-compiled binary, this would confirm the problem likely lies with musl