Clamp mask fill selection merging to layer bounds

To avoid accessing nonexistent pixels and crashing because of it.
drawpile · Nov 22, 2024 · 3d6ddd2 · 3d6ddd2
1 parent bbe919a
commit 3d6ddd2
Show file tree

Hide file tree

Showing 2 changed files with 14 additions and 3 deletions.
diff --git a/ChangeLog b/ChangeLog
@@ -14,6 +14,7 @@ Unreleased Version 2.2.2-pre
  * Fix: Work around modifier keys not registering when using a tablet on Wayland. Thanks Absolute Goober for reporting.
  * Server Fix: Don't pointlessly reload server certificates if they didn't change. Thanks Liz for reporting.
  * Server Fix: Work around too many authenticated users in a session, which could cause a crash. Thanks Liz for reporting.
+ * Fix: Don't crash when filling with feathering and a canvas-filling selection. Thanks Meru for reporting.
 
 2024-11-06 Version 2.2.2-beta.4
  * Fix: Solve rendering glitches with selection outlines that happen on some systems. Thanks xxxx for reporting.

diff --git a/src/drawdance/libengine/dpengine/flood_fill.c b/src/drawdance/libengine/dpengine/flood_fill.c
@@ -1042,14 +1042,24 @@ static void merge_mask_with_selection(float *mask, int img_x, int img_y,
                                       DP_Selection *sel)
 {
     DP_LayerContent *lc = DP_selection_content_noinc(sel);
+    int layer_width = DP_layer_content_width(lc);
+    int layer_height = DP_layer_content_height(lc);
     for (int y = 0; y < img_height; ++y) {
         for (int x = 0; x < img_width; ++x) {
             int index = y * img_width + x;
             float value = mask[index];
             if (value > 0.0f) {
-                DP_Pixel15 pixel =
-                    DP_layer_content_pixel_at(lc, x + img_x, y + img_y);
-                mask[index] = value * DP_channel15_to_float(pixel.a);
+                int layer_x = x + img_x;
+                int layer_y = y + img_y;
+                if (layer_x >= 0 && layer_x < layer_width && layer_y >= 0
+                    && layer_y < layer_height) {
+                    DP_Pixel15 pixel =
+                        DP_layer_content_pixel_at(lc, x + img_x, y + img_y);
+                    mask[index] = value * DP_channel15_to_float(pixel.a);
+                }
+                else {
+                    mask[index] = 0.0f;
+                }
             }
         }
     }