Overrun stack and heap OTP-26.0

[vshev4enko]

Describe the bug
Application down few seconds after the run release in AWS eks

To Reproduce
Unfortunately i have no idea how to reproduce.

Affected versions
26.0

Additional context
AWS EKS
erlang 26.0
alpine 3.18.0

Application logs

hend=0x00007f4f6f34c7f0
stop=0x00007f4f6f34c670
htop=0x00007f4f6f34c678
heap=0x00007f4f6f349600
beam/erl_gc.c, line 735: <0.3141.0>: Overrun stack and heap

[benoawfu]

Same thing here, alpine 3.18.0, OTP 26.0 but on AWS ECS Fargate.

hend=0x00007f5329e030c8
stop=0x00007f5329e02db8
htop=0x00007f5329e02dc0
heap=0x00007f5329df5d28
beam/erl_gc.c, line 735: <0.190015.0>: Overrun stack and heap
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
[os_mon] memory supervisor port (memsup): Erlang has closed

[jhogberg]

Thanks for your report, could you provide a core dump together with the beam.smp binary?

[benoawfu]

Sorry I can't retrieve that off ECS.

[jhogberg]

That's unfortunate :-/

Can you reproduce this with image that isn't based on alpine (or more importantly, doesn't use musl)?

[benoawfu]

It's production stuff, sorry I can't do experiments on this. I haven't had any issue on the pre-production/test servers.

[jhogberg]

musl makes the JIT run in a weird configuration that isn't as well-tested as the one under GNU libc, so it would be nice if you could try it out some time. If it only happens under musl it'll be much easier for me to narrow things down.

As an added bonus you'll get 10-15% better performance on linear code. :-)

[vshev4enko]

You mean run the same application under ubuntu instead of alpine on environment where the latter crashed?

[jhogberg]

Yeah, that'd work. As long as the environment uses GNU libc instead of musl.

[vshev4enko]

Replaced base image with debian-bullseye-20230227-slim and it works. I can't tell about performance but no crashes.

[jhogberg]

Thank you, then the crash may be related to that configuration. If you can provide a core dump (+beam.smp) the next time this crashes we'd be very thankful.

[wkirschbaum]

@jhogberg would you recommend as a general rule that we avoid musl for running erlang in production?

I am getting the same issue on alpine:

hend=0x00007f0a7e8fd830
stop=0x00007f0a7e8fd630
htop=0x00007f0a7e8fd638
heap=0x00007f0a7e8f0490
beam/erl_gc.c, line 735: <0.2688.0>: Overrun stack and heap

And can reproduce it on a build server. I don't see a erl_crash.dump file, so if someone can maybe help me get the dump and beam.smp files I can post it.

[jhogberg]

@jhogberg would you recommend as a general rule that we avoid musl for running erlang in production?

I'd avoid it for now, this only seems to happen with this configuration.

And can reproduce it on a build server. I don't see a erl_crash.dump file, so if someone can maybe help me get the dump and beam.smp files I can post it.

That's great! It ought to have dumped core and not generated a crash dump, so that's why you're not seeing one. Please send us the core dump together with beam.smp if you can. :-)

[ziopio]

FLy.io machine
erlang 26.0
alpine 3.18.0

I want to signal that this also appeared on under the same conditions when I was working on a Fly.io machine in the cloud.

I was spawning 1000 docker containers all under a single supervisor, so there was an erlang process for each one of them.
Then all stdout of all containers was stored in a single erlang process. At the same time the same process was also indirectly doing work for all containers. So that one was the trigger i think.
It did not happen with 100 containers, for example.

[wkirschbaum]

@jhogberg beam.smp and core dump: https://git.sr.ht/~whk/erlang-26-crash-reports/tree

Please let me know if there is anything which will help.

Docker image: hexpm/elixir:1.15.0-erlang-26.0.1-alpine-3.18.2

[jhogberg]

That beam.smp is stripped, do you happen to have the symbols saved somewhere?

[wkirschbaum]

@jhogberg i can't even pretend to understand what you mean, since I have very little erlang or gdb experience. If its somewhere on the instance I can upload it, but not sure where to look.

[wkirschbaum]

I just copied what was in this container, but have a feeling its not there: hexpm/elixir:1.15.0-erlang-26.0.1-alpine-3.18.2

cp /usr/local/lib/erlang/erts-14.0.1/bin/beam.smp build-archive

[jhogberg]

That's really annoying, digging into the images they seem based on https://hub.docker.com/_/erlang which forcibly strips all symbols without having the common decency to save them anywhere. It would've been nice if they didn't strip them in that layer but left it to the final one.

We're basically stuck until we have the symbols, I'll try to get in touch with the folks maintaining those images but I'm not sure how long that will take. In the meantime I suppose we can make our own nearly-identical images by removing the lines that strip symbols (the ones that look like this), but if you want to wait until the upstream images contain symbols I'm fine with that.

[wkirschbaum]

@jhogberg thanks for having a look. I can try the suggested change on our build server somewhere this week and will post. There is no urgency for me, but sure more people will run into this soon.

[princemaple]

Probably worthless information but here it is anyway: it seems to only happen on apps that use large chunks of memory. Out of 3 apps that I'm running with OTP 26 on alpine, only one that uses large chunk of memory had issues and crashed multiple times a day. The other 2 had been running fairly smoothly.

[mudssrali]

Probably worthless information but here it is anyway: it seems to only happen on apps that use large chunks of memory. Out of 3 apps that I'm running with OTP 26 on alpine, only one that uses large chunk of memory had issues and crashed multiple times a day. The other 2 had been running fairly smoothly.

I don't think so it's memory issue -- I've 16GB machine on cloud and locally -- works fine on local however deployment on cloud is crashing -- fairly simple Elixir application that runs bunch of GenServers -- with 1MB payload. I'm using alpine -- guessing by default wiith OTP-26

FROM elixir:1.15.1-alpine

Error

hend=0x00007fdd10cc0e18
stop=0x00007fdd10cc0db8
htop=0x00007fdd10cc0dc0
heap=0x00007fdd10cc0258
beam/erl_gc.c, line 735: <0.1616.0>: Overrun stack and heap

Previously the same application was working fine on elixir:1-8-alpine

[Harrisonl]

Don't have much to add to this thread, other then that it was working fine on OTP-25 and Elixir 1.14 alpine linux (ECS-Fargate) for us, and only seen it today for the first time on OTP-26 and Elixir 1.15, will try the debian image tomorrow and see how that get's on.

[davidye]

Also started happening to me when I went updated elixir:1.15.4-alpine to elixir:1.14.0-alpine.

[Lankester]

I have just got this error in a Debian docker container running off of the elixir:1.15-slim Dockerfile.

user@host:/app# cat /etc/issue
Debian GNU/Linux 11 \n \l

Error:

hend=0x00007fb33de2c288
stop=0x00007fb33de2c080
htop=0x00007fb33de2c088
heap=0x00007fb33de2b6c8
beam/erl_gc.c, line 735: <0.253.0>: Overrun stack and heap
Aborted (core dumped)

It's a relatively simple application consisting of 3 GenServers.

I have a core dump but can't share due to sensitive information within and I have been unable to replicate the issue in a test app that omits the sensitive info.

I've attached a gdb backtrace in case that's useful though and I am open to suggestions and happy to help further.

backtrace.txt

[vshev4enko]

I am not running Mint.Websocket.
Also i stopped getting crashes after switching the image from alpine to debian.

[Lankester]

It seems to crash almost every time in the same place — receiving and decoding a websocket message. Also, possibly worth pointing out hardware is Intel Mac.

Mint is version 1.5.1.

Another gdb log attached from another crash.

gdb_crash_2.txt

[Lankester]

Moving to OTP 25 via elixir:1.15-otp-25-slim seems to have resolved the issue for me, no further crashes so far.

[nathany-copia]

Seeing this on OTP 26.0.2 and Alpine 3.17.4

We're using this Docker image -- hexpm/elixir:1.15.4-erlang-26.0.2-alpine-3.17.4 for the build and running under alpine:3.17.4 with Erlang bundled via mix releases.

Not sure if the core dump will have symbols for this Docker image, or where to find the core dump, but I'll follow up if we find it.

htop=0x00007fe486d4ad48
stop=0x00007fe486d4ad40
heap=0x00007fe486d47ba8
beam/erl_gc.c, line 735: <0.8138.0>: Overrun stack and heap
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
[os_mon] memory supervisor port (memsup): Erlang has closed
Aborted (core dumped)

UPDATE: I poked around and but didn't find a core dump -- I don't know the PID it was running under, and I think Kubernetes restarted containers so the evidence is gone. For now I'm downgrading to hexpm/elixir:1.15.4-erlang-25.3.2.5-alpine-3.17.4, which is a little easier than switching to Debian or another base image.

https://hub.docker.com/r/hexpm/elixir

[ramyma]

I'm getting a similar crash as well with erlang 26.0.2 and elixir 1.15.4-otp-26 with mint_web_socket following their genserver example (https://github.com/elixir-mint/mint_web_socket/blob/main/examples/genserver.exs):

hend=0x00007f79dba1b8c0
stop=0x00007f79dba1b7a8
htop=0x00007f79dba1b7b0
heap=0x00007f79dba186d0
beam/erl_gc.c, line 735: <0.777.0>: Overrun stack and heap
[1]    48048 IOT instruction (core dumped)  iex -S mix phx.server

The crash happens after receiving some messages (and while receiving more messages) over the websocket.

I'm using Ubuntu 22.04.3

[akoutmos]

I am seeing this issue too. I am also running the hexpm/elixir:1.15.4-erlang-26.0.2-alpine-3.17.4 image and have a mix release Phoenix app running in the container. The only thing that I can correlate this too without much of a deep dive was that this happened right after our app was DoS attacked.

I will try and get more information around this, but for now will revert to hexpm/elixir:1.15.4-erlang-25.3.2.5-alpine-3.17.4.

[bernardo-martinez]

I am seeing this issue too:

hend=0x00007ff8c845fb90
stop=0x00007ff8c845fa60
htop=0x00007ff8c845fa68
heap=0x00007ff8c845c9a0
beam/erl_gc.c, line 735: <0.67[52](https://gitlab.otters.xyz/product/business/experience/app/-/jobs/12068617#L52).0>: Overrun stack and heap

building image myself with: elixir 1.15.3 and erlang 26.0.2
running on a gitlab alpine image

[sverker]

It seems a lot of people are able to reproduce this.
It would be really helpful if someone could give detailed instructions of how to reproduce the crash. Preferably a "minimal" example that is easy and quick to run.

[ggalan87]

I reached this issue after some rabbitmq server crashes.

Setup: plain LXC container, alpine 1.18 image, erlang-26.0.2, elixir-1.15.4

Log:

  Starting broker... completed with 5 plugins.      
                                                                                                         
hend=0x00007fe234eb34b0                                                                                  
stop=0x00007fe234eb3458                                                                                  
htop=0x00007fe234eb3460                                                                                                                                                                                            
heap=0x00007fe234eb21a0                                                                                  
beam/erl_gc.c, line 735: <0.15407.0>: Overrun stack and heap                                             
Aborted (core dumped)                                                       

[JamesLavin]

In case my context might help anyone debug this, I hit this bug hard yesterday after building (on Apple Silicon) Ubuntu images (plural because I used a variety of base images with various versions of Elixir & Erlang, hoping that one of them might work) to run an Elixir script. Was able to build the images fine, but when I booted the container and ran my .exs file, Mix.install never managed to compile the Jason library (https://github.com/michalmuskala/jason). It always blew up with this error message during Jason library compilation.

(Compiling on Apple Silicon adds a layer of complexity, see: https://pythonspeed.com/articles/docker-build-problems-mac/, but this may be a red herring, given that others seem to have hit this issue elsewhere.)

FWIW, I wound up spinning up an EC2 instance and getting the same .exs file to run just fine there.

[ramyma]

It seems a lot of people are able to reproduce this. It would be really helpful if someone could give detailed instructions of how to reproduce the crash. Preferably a "minimal" example that is easy and quick to run.

I created a simple repo where I'm able to reproduce the issue consistently: https://github.com/ramyma/otp_crash

You'll just have to run iex -S mix phx.server, and within a couple of minutes it will crash with:

iex(1)> hend=0x00007f4f237fbdb0
stop=0x00007f4f237fbc98
htop=0x00007f4f237fbca0
heap=0x00007f4f237f6ce0
beam/erl_gc.c, line 735: <0.561.0>: Overrun stack and heap
[1]    97851 IOT instruction (core dumped)  iex -S mix phx.server

Note: I'm using asdf, you'll find a .tool-versions, so you can run asdf install to get the required Erlang and Elixir versions.
Tested on Ubuntu 22.04.3

[Lankester]

I've put together a minimal repo based on the elixir 1.15-slim docker image. https://github.com/Lankester/Otp26Crash

Run docker compose up and it should crash within between 10 and 20 minutes.

otp-26-crash-7292  | 00:38:00.255 [info] Broadcast 73898124 bytes on WebSocket.
otp-26-crash-7292  | 00:38:01.108 [info] Received message. 73898124 bytes.
otp-26-crash-7292  | 00:38:02.749 [info] Broadcast 77805516 bytes on WebSocket.
otp-26-crash-7292  | 00:38:03.670 [info] Received message. 77805516 bytes.
otp-26-crash-7292  | 00:38:05.133 [info] Broadcast 60002824 bytes on WebSocket.
otp-26-crash-7292  | 00:38:05.703 [info] Received message. 60002824 bytes.
otp-26-crash-7292  | 00:38:07.487 [info] Broadcast 54965496 bytes on WebSocket.
otp-26-crash-7292  | 00:38:08.008 [info] Received message. 54965496 bytes.
otp-26-crash-7292  | hend=0x00007f52f487d520
otp-26-crash-7292  | stop=0x00007f52f487d318
otp-26-crash-7292  | htop=0x00007f52f487d320
otp-26-crash-7292  | heap=0x00007f52f487a330
otp-26-crash-7292  | beam/erl_gc.c, line 735: <0
otp-26-crash-7292  | .2402.0>: Overrun stack and heap

[sverker]

Thanks, @ramyma and @Lankester for the "crash repos".

@ramyma Just to be clear; you ran it directly on Ubuntu without any docker image?

[ramyma]

Thanks, @ramyma and @Lankester for the "crash repos".

@ramyma Just to be clear; you ran it directly on Ubuntu without any docker image?

@sverker yes, directly on Ubuntu.

[sverker]

We think we found the bug. Here is a quick fix if anyone wants to try it out:

diff --git a/erts/emulator/beam/jit/x86/instr_bs.cpp b/erts/emulator/beam/jit/x86/instr_bs.cpp
index 39dfb64f8f..f8d8cebbdf 100644
--- a/erts/emulator/beam/jit/x86/instr_bs.cpp
+++ b/erts/emulator/beam/jit/x86/instr_bs.cpp
@@ -3829,7 +3829,7 @@ static std::vector<BsmSegment> opt_bsm_segments(
             }
             break;
         case BsmSegment::action::GET_BINARY:
-            heap_need += heap_bin_size((seg.size + 7) / 8);
+            heap_need += std::max(ERL_SUB_BIN_SIZE, heap_bin_size((seg.size + 7) / 8));
             break;
         case BsmSegment::action::GET_TAIL:
             heap_need += EXTRACT_SUB_BIN_HEAP_NEED;

The same fix can be done for ARM in erts/emulator/beam/jit/arm/instr_bs.cpp.

The bug exists since OTP 26.0. The root cause has nothing to do with elixir, alpine or musl libc.
It takes matching small byte-unaligned bitstrings from a larger binary, as done by Elixir.Mint.WebSocket.Frame:decode_raw/3, plus some bad luck with being almost out of process heap space.

We will probably soon release a fix in OTP 26.0.3.