WebODM won't start - repeated webapp exception

[rahlhar]

How did you install WebODM (docker, installer, etc.)?

Installed docker v27.3.1. Used git clone followed by ./webodm.sh start.

What's your browser and operating system? (Copy/paste the output of https://www.whatismybrowser.com/)

Browser is irrelevant - the problem is a failure to start.

OS is Artix Linux running 6.12.9-hardened (confirmed also on 6.12.9).

What is the problem?

When running ./webodm.sh start, all seems to flow well until the webapp process keeps crashing, retrying, ad nauseum.

webapp      | celery beat v4.4.0 (cliffs) is starting.                                                                                         
webapp      | [2025-01-16 10:16:05 +0000] [240] [INFO] Starting gunicorn 19.7.1                                                                
webapp      | Traceback (most recent call last):                                                                                               
webapp      |   File "/usr/local/lib/python3.9/dist-packages/gunicorn/sock.py", line 44, in set_options                                        
webapp      |     sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)                                                                   
webapp      | OSError: [Errno 95] Operation not supported                                                                                      
webapp      |                                                                                                                                  
webapp      | During handling of the above exception, another exception occurred:                                                              
webapp      |                                                                                                                                  
webapp      | Traceback (most recent call last):                                                                                               
webapp      |   File "/usr/local/bin/gunicorn", line 8, in <module>                                                                            
webapp      |     sys.exit(run())                                      
webapp      |   File "/usr/local/lib/python3.9/dist-packages/gunicorn/app/wsgiapp.py", line 74, in run                                         
webapp      |     WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()                                                                     
webapp      |   File "/usr/local/lib/python3.9/dist-packages/gunicorn/app/base.py", line 203, in run                                           
webapp      |     super(Application, self).run()                                                                                               
webapp      |   File "/usr/local/lib/python3.9/dist-packages/gunicorn/app/base.py", line 72, in run                                            
webapp      |     Arbiter(self).run()                                                                                                          
webapp      |   File "/usr/local/lib/python3.9/dist-packages/gunicorn/arbiter.py", line 198, in run
webapp      |     self.start()                                         
webapp      |   File "/usr/local/lib/python3.9/dist-packages/gunicorn/arbiter.py", line 157, in start                                          
webapp      |     self.LISTENERS = sock.create_sockets(self.cfg, self.log, fds)                                                                
webapp      |   File "/usr/local/lib/python3.9/dist-packages/gunicorn/sock.py", line 180, in create_sockets                                    
webapp      |     sock = sock_type(addr, conf, log)                                                                                            
webapp      |   File "/usr/local/lib/python3.9/dist-packages/gunicorn/sock.py", line 117, in __init__                                          
webapp      |     super(UnixSocket, self).__init__(addr, conf, log, fd=fd)                                                                     
webapp      |   File "/usr/local/lib/python3.9/dist-packages/gunicorn/sock.py", line 32, in __init__                                           
webapp      |     self.sock = self.set_options(sock, bound=bound)                                                                              
webapp      |   File "/usr/local/lib/python3.9/dist-packages/gunicorn/sock.py", line 46, in set_options                                        
webapp      |     if err[0] not in (errno.ENOPROTOOPT, errno.EINVAL):                                                                          
webapp      | TypeError: 'OSError' object is not subscriptable                                                                                 
webapp exited with code 0 

How can we reproduce this? (What steps trigger the problem? What parameters are you using for processing? Include screenshots. If you are having issues processing a dataset, you must include a copy of your dataset uploaded on Dropbox, Google Drive or https://dronedb.app)

Not really reproducable without knowing what the problem is.

[hauke96]

I have the same problem using Arch Linux with latest kernel 6.12.9, latest docker, latest WebODM, etc.. I just went down the rabbit hole:

Root cause of the issue

• The error comes from a change in the Linux Kernel regarding SO_REUSEPORT fixing vulnerability CVE-2024-57903
• gunicorn still uses this property in their file sock.py (within the container at /usr/local/lib/python3.9/dist-packages/gunicorn/sock.py)
• Since gunicorn 19.8.0 this issue can be prevented with a new config entry reuse_port so that you can disable this reusing of ports: https://github.com/benoitc/gunicorn/blob/19.8.0/gunicorn/sock.py#L42
• WebODM uses the older version 19.7.1 which will always try to use this SO_REUSEPORT feature.

My temporary solution

I didn't want to change my system (i.e. downgrading the kernel) or compiling WebODM manually or something like this. So I came up with this:

1. Copy sock.py from the WebODM container or their GitHub repo (19.7.1 is currently used in WebODM).
2. Comment out or remove lines 42 to 47 so that the whole SO_REUSEPORT is not active.
3. Edit the docker-compose.yml of WebODM and add an entry - ./sock.py:/usr/local/lib/python3.9/dist-packages/gunicorn/sock.py into the volumes: section of the webapp service. Use spaces not tabs, it's important 😉

That's it, WebODM works now for me

My suggestion of a permanent solution

Use a version 19.8.0 of gunicorn and set reuse_port to false. As far as I understand their release page, it's even disabled by default, so nothing need to be done apart from updating.

[rahlhar]

@hauke96 Brilliant. Thanks for doing the deep dive.
I can confirm that the temporary fix works nicely.

And can second that WebODM should update gunicorn asap.