"worker process {id} exited on signal 9" on very large file upload

[Matthias-vdE]

When uploading really big files (talking 5GB and larger) to Immich via NPM, the upload always gets cancelled at the very end. The only thing that is showing in the log is

2024-10-17T11:44:37.096387678Z 2024/10/17 13:44:37 [warn] 1991#1991: *1725 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000002 while reading request body, client: 94.X.X.X, server: immich.X.X, request: "POST /api/assets HTTP/1.1", host: "immich.X.X"

2024-10-17T12:18:08.594066739Z 2024/10/17 14:18:08 [alert] 1693#1693: worker process 1991 exited on signal 9

Smaller files are no problem. I have disabled AppSec because it was causing issues as well (#1123) but this has now been resolved (simply by disabling AppSec alltogether.)

My compose file looks like this:

services:
  npmplus:
    container_name: npmplus
    image: zoeyvid/npmplus:latest
    restart: always
    network_mode: host
    volumes:
      - "/opt/npm:/data"
    environment:
      - "TZ=Europe/Brussels" 
      - "NGINX_LOG_NOT_FOUND=true" 
      - "LOGROTATE=true" 
      - "LOGROTATIONS=7"
      - "GOA=true"
      - "CRT=36"
      - "IPRT=3"
      - "NGINX_DISABLE_PROXY_BUFFERING=true"

  crowdsec:
    container_name: crowdsec
    image: crowdsecurity/crowdsec:latest
    restart: always
    network_mode: bridge
    ports:
      - "127.0.0.1:7422:7422"
      - "127.0.0.1:8080:8080"
    environment:
      - "TZ=Europe/Brussels"
      - "COLLECTIONS=ZoeyVid/npmplus"
      - "LEVEL_FATAL=true"
      - "LEVEL_ERROR=true"
      - "LEVEL_WARN=true"
      - "LEVEL_INFO=false"
      - "LEVEL_DEBUG=false"
      - "LEVEL_TRACE=false"
    volumes:
      - "/opt/crowdsec/conf:/etc/crowdsec"
      - "/opt/crowdsec/data:/var/lib/crowdsec/data"
      - "/opt/npm/nginx:/opt/npm/nginx:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  geoipupdate:
    container_name: geoipupdate
    image: maxmindinc/geoipupdate:latest
    restart: always
    network_mode: bridge
    environment:
      - "TZ=Europe/Brussels"
      - "GEOIPUPDATE_EDITION_IDS=GeoLite2-Country GeoLite2-City GeoLite2-ASN"
      - "GEOIPUPDATE_ACCOUNT_ID=REDACTED"
      - "GEOIPUPDATE_LICENSE_KEY=REDACTED"
      - "GEOIPUPDATE_FREQUENCY=24"
    volumes:
      - "/opt/npm/etc/goaccess/geoip:/usr/share/GeoIP"

I haved tested with both NGINX_DISABLE_PROXY_BUFFERING true and false but the result is the same.

My Host configuration is this:

The proxy timeout should be plenty. I've also disable gzip and Brotli because I thought that might cause an issue but it makes no difference in having this on or off.

I'm not hitting the memory limit of my server either (I have 16GB)

Any ideas, suggestions, or other logs I can gather in order to troubleshoot would be great. Or some other advanced Host options that may help with uploading large files.

Thanks.

[Zoey2936]

can please try to add:
proxy_request_buffering off;

[Matthias-vdE]

Sure, here is the full config file:

server {
    set $forward_scheme http;
    set $server "127.0.0.1";
    set $port 2283;

    listen unix:/run/nginx-21.sock;

    listen 0.0.0.0:80;
    listen [::]:80;

    listen 0.0.0.0:443 ssl;
    listen [::]:443 ssl;

    listen 0.0.0.0:443 quic;
    listen [::]:443 quic;

    more_set_headers 'Alt-Svc: h3=":443"; ma=86400';

    server_name immich.mydomain.org;

    # Certbot TLS
    include conf.d/include/tls-ciphers.conf;
    ssl_certificate /data/tls/certbot/live/npm-36/fullchain.pem;
    ssl_certificate_key /data/tls/certbot/live/npm-36/privkey.pem;

    more_clear_headers "Expect-CT";
    include conf.d/include/hsts.conf;

    # Force TLS
    include conf.d/include/force-tls.conf;

    modsecurity on;

    modsecurity_rules_file /usr/local/nginx/conf/conf.d/include/modsecurity.conf;

    include conf.d/include/always.conf;

    client_max_body_size 0;

    proxy_read_timeout 3200s;
    proxy_send_timeout 3200s;
    send_timeout 3200s;

    proxy_request_buffering off;
    gzip off;

    # custom locations
    location / {
        include conf.d/include/always.conf;

        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;

        # Proxy!
        include conf.d/include/proxy.conf;
    }

    # Custom
    include /data/nginx/custom/server_proxy.conf;
}

[Zoey2936]

can you disable modsec?

[Matthias-vdE]

Just disabled it as a test and that indeed seems to have resolved the issue.

So both appsec and modsec are causing issues with Immich. Good to know. And thanks again for looking into this!

[Zoey2936]

I think you can now also clear your advanced config

[Zoey2936]

The appsec issue should be fixed in the next release see: #873 (reply in thread)

[Zoey2936]

The Issue seems to still exist, please continue any discussions in #1241, if you have any further useful information you can also leave it here: crowdsecurity/lua-cs-bouncer#77