Segfault on gnome-shell Wayland

[pv]

Gnome-shell on Wayland crashes after enabling shellshape, when trying to map a new window. Seen on gnome-shell 3.18.4-0ubuntu2, and iirc also on 3.16 on ubuntu 15.10.

Backtrace:

Core was generated by `/usr/bin/gnome-shell --wayland --display-server'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  fill_states (window=0x0, states=0x7ffec493f640) at wayland/meta-wayland-surface.c:2255
[Current thread is 1 (Thread 0x7f4e0a1c7a80 (LWP 2241))]
(gdb) bt
#0  0x00007f4e08ccbfb4 in meta_wayland_surface_configure_notify (window=0x0, states=0x7ffec493f640) at wayland/meta-wayland-surface.c:2255
#1  0x00007f4e08ccbfb4 in meta_wayland_surface_configure_notify (surface=0x26574b0 [MetaWaylandSurface], new_width=new_width@entry=1366, new_height=new_height@entry=741, sent_serial=sent_serial@entry=0x4c4eb60) at wayland/meta-wayland-surface.c:2291
#2  0x00007f4e08ccd64f in meta_window_wayland_move_resize_internal (window=0x4c4e8a0 [MetaWindowWayland], gravity=<optimized out>, unconstrained_rect=..., constrained_rect=..., flags=(META_MOVE_RESIZE_USER_ACTION | META_MOVE_RESIZE_MOVE_ACTION | META_MOVE_RESIZE_RESIZE_ACTION), result=0x7ffec493f72c) at wayland/meta-window-wayland.c:236
#3  0x00007f4e08ca5555 in meta_window_move_resize_internal (window=0x4c4e8a0 [MetaWindowWayland], flags=(META_MOVE_RESIZE_USER_ACTION | META_MOVE_RESIZE_MOVE_ACTION | META_MOVE_RESIZE_RESIZE_ACTION), gravity=gravity@entry=1, frame_rect=...) at core/window.c:3707
#4  0x00007f4e08ca5959 in meta_window_move_resize_frame (window=<optimized out>, user_op=<optimized out>, root_x_nw=<optimized out>, root_y_nw=<optimized out>, w=<optimized out>, h=<optimized out>) at core/window.c:3820
#5  0x00007f4e02099e40 in ffi_call_unix64 () at /usr/lib/x86_64-linux-gnu/libffi.so.6
#6  0x00007f4e020998ab in ffi_call () at /usr/lib/x86_64-linux-gnu/libffi.so.6
#7  0x00007f4e095a947b in  () at /usr/lib/libgjs.so.0
#8  0x00007f4e095aa95f in  () at /usr/lib/libgjs.so.0
#9  0x00007f4e02a3a98c in  () at /usr/lib/x86_64-linux-gnu/libmozjs-24.so.0
#10 0x00007f4e02a3b90e in  () at /usr/lib/x86_64-linux-gnu/libmozjs-24.so.0
#11 0x00007f4e02a439d8 in  () at /usr/lib/x86_64-linux-gnu/libmozjs-24.so.0
#12 0x00007f4e02a3a87d in  () at /usr/lib/x86_64-linux-gnu/libmozjs-24.so.0
#13 0x00007f4e02a3ad5b in  () at /usr/lib/x86_64-linux-gnu/libmozjs-24.so.0
#14 0x00007f4e02c959dd in  () at /usr/lib/x86_64-linux-gnu/libmozjs-24.so.0
#15 0x00007f4df0009aa2 in  ()

gnome-session logs:

gnome-session[2230]: INFO : initting schemas
gnome-session[2230]: WARN : loading schema from: /home/pauli/.local/share/gnome-shell/extensions/[email protected]/data/glib-2.0/schemas
gnome-session[2230]: WARN : Layout.tile_for(null)
gnome-session[2230]: INFO : shellshape enabled
gnome-session[2230]: INFO : Garbage collecting 1 windows
gnome-session[2230]: WARN : Layout.tile_for(null)
gnome-session[2230]: (EE)
gnome-session[2230]: Fatal server error:
gnome-session[2230]: (EE) failed to dispatch Wayland events: Broken pipe
gnome-session[2230]: gnome-session-binary[2230]: WARNING: Application 'gnome-shell-wayland.desktop' killed by signal 11
gnome-session-binary[2230]: WARNING: Application 'gnome-shell-wayland.desktop' killed by signal 11

[timbertson]

Damn :(

This is definitely a bug in gnome-shell / mutter and not shellshape (shellshape is JS only, so any segfaults are the problem of the host application). But I've not had much luck getting segfaults fixed in mutter unless they can be reliably reproduced. If you're able to find a minimal snippet that can be pasted into the looking-glass console (alt-f2; "lg") or a minimal extension, that might make it easier for the mutter devs to locate the bug.

[pv]

The repro is 1) enable shellshape only, 2) open a new window of any kind (apparently).
Reducing to a simpler test case would probably be useful, would require learning how to debug gjs first though :/

[pv]

https://bugzilla.gnome.org/show_bug.cgi?id=763887

[rhofour]

@pv Did you make any progress on this? I'm just starting to dig into it.

[pv]

@rhofour: No, sorry --- I did not try to debug it in detail.

[rhofour]

So, I've been digging into this and it looks like this is caused by a segfault in the wayland move and resize code. I'm totally new to both the shellshape and gnome codebases, but I'm slowly investigating this.

[timbertson]

I should have said something about this earlier, but thanks @rhofour for investigating! I haven't had much time to devote to shellshape lately and haven't actually used it on wayland yet, so I'm very appreciative.

[simpoir]

I've been doing a bit of debugging and it seems gnome receives info about windows that are 0,0 sized. My guess is those windows can't be placed because they are still being created.
FWIW I've hacked the following patch which seems to get rid of the segfaults, although there seems to be a couple more issues with wayland and sizing:

diff --git a/src/gjs/mutter_window.ts b/src/gjs/mutter_window.ts
index 6a0e6f2..431601a 100644
--- a/src/gjs/mutter_window.ts
+++ b/src/gjs/mutter_window.ts
@@ -178,6 +178,11 @@ module MutterWindow {
                        }
                        var pos = r.pos;
                        var size = r.size;
+                       var old_rect = this.meta_window.get_frame_rect();
+                       if (old_rect.width == 0 || old_rect.height == 0) {
+                               this.log.warn("move_resize() with a zero_sized frame");
+                               return;
+                       }
                        this.meta_window.move_resize_frame(true, pos.x, pos.y, size.x, size.y);
                }
                set_tile_preference(new_pref) {

[jubalfh]

I can confirm that @simpoir's patch indeed makes the issue disappear on Wayland. As this affects all Wayland installations and a number of open issues (this one, #180, #184 and #190), it would be grand if the fix could be integrated into master as soon as possible. :-)

[timbertson]

Thanks all for your help tracking this down everyone! Could someone confirm whether the latest master branch (9b7b4f7) fixes the issue? It should be a more efficient and equivalent fix, as long as the zero-sized window happens at the start of the window's lifetime (i.e. it only goes from zero -> nonzero, never back to zero)

[jubalfh]

So far, so good, my regular workflow is now stable.

I'll try to play a bit more with random layout changes (and random operations like window creation, closing, moving between tiled and untiled workspaces etc.) and will report back shortly.

[timbertson]

I've released a new version with this fix, so I'm going to call this fixed until it happens again. Thanks to everyone who helped out!