wget Connection reset by peer

[cw-20021351]

When using version 1.36 in eks environment, wget request fails.

I attach busybox.yaml file and wget command.

It doesn't seem to be a problem with the eks environment because it works normally in versions below 1.35.

apiVersion: v1
kind: Pod
metadata:
  name: busybox
spec:
  containers:
  - name: busybox
    image: busybox
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

# wget https://dtdg.co/latest-java-tracer
# Connecting to dtdg.co (67.199.248.12:443)
# wget: note: TLS certificate validation not implemented
# wget: error getting response: Connection reset by peer

Is there any change to version 1.36 that I should consider?

[fatz]

from https://busybox.net/

3 January 2023 -- BusyBox 1.36.0 (unstable)

there is no 1_36_0 release tag. Just the branch and according to the release notes its marked unstable.

Please rollback the latest tag change of the busybox so it points again on 1.35 as that just works fine

[yosifkit]

I am unable to reproduce; the current latest (aka 1.36.0) works fine:

$ docker run -it --rm busybox
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
205dae5015e7: Pull complete 
Digest: sha256:7b3ccabffc97de872a30dfd234fd972a66d247c8cfc69b0550f276481852627c
Status: Downloaded newer image for busybox:latest
/ # 
/ # wget https://dtdg.co/latest-java-tracer
Connecting to dtdg.co (67.199.248.13:443)
wget: note: TLS certificate validation not implemented
Connecting to github.com (140.82.114.4:443)
Connecting to objects.githubusercontent.com (185.199.109.133:443)
saving to 'latest-java-tracer'
latest-java-tracer   100% |********************************| 19.4M  0:00:00 ETA
'latest-java-tracer' saved
/ # / # busybox --help
BusyBox v1.36.0 (2023-01-03 22:42:57 UTC) multi-call binary.
...

---

The version bump for latest was not accidental: #161. Both 1.35.0 and 1.36.0 are still marked as unstable on https://busybox.net/. The stable tag will still point to the most recent release that was specifically marked as stable by BusyBox (even if it is not within the last two releases).

[a-schild]

I see the same behaviour in gks environment with the 1.36.0 release

With 1.34.1

BusyBox v1.34.1 (2022-12-21 18:28:04 UTC) multi-call binary.
wget --no-check-certificate https://my.nextcloud.com/cron.php
Connecting to my.nextcloud.com (xx.xx.xx.xx:443)
saving to 'cron.php'
cron.php             100% |************************************************************************|    20  0:00:00 ETA
'cron.php' saved

With 1.36.0

BusyBox v1.36.0 (2023-01-03 22:42:57 UTC) multi-call binary.
wget --no-check-certificate https://my.nextcloud.com
Connecting to my.nextcloud.com (xx.xx.xx.xx:443)
wget: error getting response: Connection reset by peer

Something to do with https/ssl handling?

[PylotLight]

@yosifkit
This issue appears to have cause us major production issues as :latest is still pulling 1.36.0 which when run on aks nodes results in failure to complete the wget call which is reproduceable.
Running this on docker desktop locally however runs fine, so not sure if it's also related to kernel configuration of some kind.
Any chance on getting :latest sent back to latest stable which is 1.34.1 as I noticed the :unstable and :latest tags are both on 1.36.0 currently?

[tianon]

I'd suggest pinning to busybox:1.35 (or busybox:1.34 or busybox:stable) explicitly if the latest release isn't working for your use case.

[PylotLight]

That's not my team and I did make that recommendation, but would be nice to have the tag fixed anyway.

[endriu0]

Just to add to that and narrow it down just a little. I just run into this and noticed that the issue only appears on the latest but also only on amd64 version , same image but arm64 version works fine.

amd64:

wget -O /tmp/newrelic.jar https://download.newrelic.com/newrelic/java-agent/newrelic-agent/7.10.0/newrelic-agent-7.10.0.jar
Connecting to download.newrelic.com (151.101.2.137:443)
wget: error getting response: Connection reset by peer

arm64:

wget -O /tmp/newrelic.jar https://download.newrelic.com/newrelic/java-agent/newrelic-agent/7.9.0/newrelic-agent-7.9.0.jar
Connecting to download.newrelic.com (151.101.2.137:443)
wget: note: TLS certificate validation not implemented
saving to '/tmp/newrelic.jar'
new-relic.jar         20% |*********************************************

[modelbitjason]

This is still broken in the musl version although the glibc version seems fixed.
Any chance of getting the musl-latest-1 build pushed back as busybox:musl?

The error only happens for some domains -- github.com works but amazon.com does not (Nor does any S3 URL).

This is from a fresh build off master

# musl
$ docker run --rm busybox:latest-musl-test wget https://amazon.com
Connecting to amazon.com (52.94.236.248:443)
wget: note: TLS certificate validation not implemented
wget: TLS error from peer (alert code 40): handshake failure
wget: error getting response: Connection reset by peer

# glibc
$ docker run --rm busybox:latest-glibc-test wget https://amazon.com
Connecting to amazon.com (52.94.236.248:443)
wget: note: TLS certificate validation not implemented
Connecting to www.amazon.com (104.78.177.85:443)
wget: server returned error: HTTP/1.1 503 Service Unavailable

# previous musl
$ docker run --rm busybox:latest-1-musl-test wget https://amazon.com
Connecting to amazon.com (52.94.236.248:443)
wget: note: TLS certificate validation not implemented
Connecting to www.amazon.com (13.224.247.127:443)
wget: server returned error: HTTP/1.1 503 Service Unavailable

# musl, works with github.com
$ docker run --rm busybox:latest-musl-test wget https://github.com                   
Connecting to github.com (140.82.114.4:443)
wget: note: TLS certificate validation not implemented
saving to 'index.html'
index.html           100% |********************************|  220k  0:00:00 ETA

[kassane]

Tested from version 1.34 to the latest.

x86_64/amd64: ✅
aarch64/arm64: ❌
riscv64: ❌
powerpc64le: ❌

0.221 wget: error getting response: Connection reset by peer

[sbp]

Bug 15679 ("wget with amazon.com fails with TLS handshake failure on Debian 12") appears to be upstream's own tracking of this issue. Filed 4 July 2023, with no contributors to date other than the original submitter.

[JoonaHa]

Tested latest stable 1.36.1 based on https://busybox.net/ on amd64. Still reproducible:

$ docker run --rm -it busybox:stable
wget https://dtdg.co/latest-java-tracer
Connecting to dtdg.co (67.199.248.13:443)
wget: note: TLS certificate validation not implemented
wget: TLS error from peer (alert code 40): handshake failure
wget: error getting response: Connection reset by peer

With previous stable 1.34.1 wget produces different errors based on the address:

$ docker run --rm -it busybox:1.34.1
wget https://download.geonames.org/export/dump/FI.zip
Connecting to download.geonames.org (5.9.152.54:443)
wget: note: TLS certificate validation not implemented
wget: TLS error from peer (alert code 40): handshake failure
wget: error getting response: Connection reset by peer

wget https://dtdg.co/latest-java-tracer
Connecting to dtdg.co (67.199.248.13:443)
wget: note: TLS certificate validation not implemented
Connecting to github.com (140.82.121.4:443)
wget: TLS error from peer (alert code 80): 80
wget: error getting response: Connection reset by peer

On 1.29.1 https://download.geonames.org/export/dump/FI.zip works without issues but https://dtdg.co/latest-java-tracer does not.

$ docker run --rm -it busybox:1.29.1
wget https://download.geonames.org/export/dump/FI.zip
Connecting to download.geonames.org (5.9.152.54:443)
wget: note: TLS certificate validation not implemented
FI.zip               100% |***********************************************************************| 4542k  0:00:00 ETA

wget https://dtdg.co/latest-java-tracer
Connecting to dtdg.co (67.199.248.13:443)
wget: note: TLS certificate validation not implemented
Connecting to github.com (140.82.121.4:443)
wget: note: TLS certificate validation not implemented
wget: TLS error from peer (alert code 80): 80
wget: error getting response: Connection reset by peer

[discapes]

Also seeing this on an embedded system with just busybox 1.36.1, with musl 1.2.5.
I cannot connect to any domain, even https://github.com or https://google.com.
Sometimes the output is just

/ # wget -o- https://github.com
Connecting to github.com (140.82.121.4:443)
#

Sometimes it's

Connecting to github.com (140.82.121.4:443)
wget: error getting response: Connection reset by peer

[jblom]

Got the same (using the stable version of this moment) as @discapes (when connecting to a path within github.com):

Connecting to github.com (140.82.121.3:443)
wget: note: TLS certificate validation not implemented
Connecting to github.com (140.82.121.3:443)
wget: got bad TLS record (len:0) while expecting switch to encrypted traffic
wget: error getting response: Connection reset by peer

Apparently it does not always occur, so it might have been just a couple of hiccups when connecting to github.com (and it has nothing to do with busybox after all...)

[tianon]

GitHub did apparently have a bunch of hiccups over the weekend (as evidenced by the pile of CI failure emails I've got from this weekend 😄)