Segfaults every 10 minutes when built against miniupnpc 2.2.8

[devvick]

Happens on at least 2 different machines (one bare-metal, one VM).

OS & version: Zerotier 1.14.0 on Archlinux
Package recently rebuilt against miniupnpc 2.2.8
https://gitlab.archlinux.org/archlinux/packaging/packages/zerotier-one/-/commit/cb502ed73643bc9bbf26f9f08f92620508f492f4

Journal log with stacktrace

Jul 06 09:09:47 archlinux kernel: zerotier-one[35799]: segfault at fffffffb ip 000074d25dacec46 sp 000074d25dffbc18 error 6 in libc.so.6[74d25da38000+16c000] likely on CPU 0 (core 0, socket 0)
Jul 06 09:09:47 archlinux kernel: Code: c3 0f 1f 00 48 8b 0e 8b 56 07 48 89 0f 89 57 07 49 83 e8 0b 48 8d 7f 0b 0f 85 66 07 00 00 c3 0f 1f 44 00 00 48 8b 0e 8b 56 08 <48> 89 0f 89 57 08 49 83 e8 0c 48 8d 7f 0c 0f 85 46 07 00 00 c3 0f
Jul 06 09:09:47 archlinux systemd-coredump[36532]: Process 35797 (zerotier-one) of user 971 terminated abnormally with signal 11/SEGV, processing...
Jul 06 09:09:47 archlinux systemd[1]: Started Process Core Dump (PID 36532/UID 0).
Jul 06 09:09:47 archlinux systemd-coredump[36533]: [🡕] Process 35797 (zerotier-one) of user 971 dumped core.
                                                   
                                                   Stack trace of thread 35799:
                                                   #0  0x000074d25dacec46 n/a (libc.so.6 + 0xbac46)
                                                   #1  0x000074d25e7afc36 UPNP_GetValidIGD (libminiupnpc.so.18 + 0x9c36)
                                                   #2  0x0000598847e84a4e n/a (zerotier-one + 0x1daa4e)
                                                   #3  0x0000598847e848cd n/a (zerotier-one + 0x1da8cd)
                                                   #4  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #5  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   
                                                   Stack trace of thread 35797:
                                                   #0  0x000074d25db278f7 __select (libc.so.6 + 0x1138f7)
                                                   #1  0x0000598847e3c1c3 n/a (zerotier-one + 0x1921c3)
                                                   #2  0x0000598847e54bfd n/a (zerotier-one + 0x1aabfd)
                                                   #3  0x0000598847e94f0c n/a (zerotier-one + 0x1eaf0c)
                                                   #4  0x0000598847cf4861 n/a (zerotier-one + 0x4a861)
                                                   #5  0x000074d25da39c88 n/a (libc.so.6 + 0x25c88)
                                                   #6  0x000074d25da39d4c __libc_start_main (libc.so.6 + 0x25d4c)
                                                   #7  0x0000598847cf4d45 n/a (zerotier-one + 0x4ad45)
                                                   
                                                   Stack trace of thread 35820:
                                                   #0  0x000074d25db2c05a recv (libc.so.6 + 0x11805a)
                                                   #1  0x0000598847e81c1c n/a (zerotier-one + 0x1d7c1c)
                                                   #2  0x0000598847e83b44 n/a (zerotier-one + 0x1d9b44)
                                                   #3  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #4  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   
                                                   Stack trace of thread 35798:
                                                   #0  0x000074d25daf2f43 clock_nanosleep (libc.so.6 + 0xdef43)
                                                   #1  0x000074d25dafed77 __nanosleep (libc.so.6 + 0xead77)
                                                   #2  0x0000598847d297b8 n/a (zerotier-one + 0x7f7b8)
                                                   #3  0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
                                                   #4  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #5  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   
                                                   Stack trace of thread 35801:
                                                   #0  0x000074d25daf2f43 clock_nanosleep (libc.so.6 + 0xdef43)
                                                   #1  0x000074d25dafed77 __nanosleep (libc.so.6 + 0xead77)
                                                   #2  0x0000598847d99f1b n/a (zerotier-one + 0xeff1b)
                                                   #3  0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
                                                   #4  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #5  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   
                                                   Stack trace of thread 35822:
                                                   #0  0x000074d25db278f7 __select (libc.so.6 + 0x1138f7)
                                                   #1  0x0000598847e7c6aa n/a (zerotier-one + 0x1d26aa)
                                                   #2  0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
                                                   #3  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #4  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   
                                                   Stack trace of thread 35802:
                                                   #0  0x000074d25db2bc7d accept (libc.so.6 + 0x117c7d)
                                                   #1  0x0000598847e5caf1 n/a (zerotier-one + 0x1b2af1)
                                                   #2  0x0000598847e36624 n/a (zerotier-one + 0x18c624)
                                                   #3  0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
                                                   #4  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #5  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   
                                                   Stack trace of thread 35812:
                                                   #0  0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
                                                   #1  0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
                                                   #2  0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
                                                   #3  0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
                                                   #4  0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
                                                   #5  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #6  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   
                                                   Stack trace of thread 35800:
                                                   #0  0x000074d25daf2f43 clock_nanosleep (libc.so.6 + 0xdef43)
                                                   #1  0x000074d25dafed77 __nanosleep (libc.so.6 + 0xead77)
                                                   #2  0x0000598847df1dc3 n/a (zerotier-one + 0x147dc3)
                                                   #3  0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
                                                   #4  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #5  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   
                                                   Stack trace of thread 35804:
                                                   #0  0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
                                                   #1  0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
                                                   #2  0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
                                                   #3  0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
                                                   #4  0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
                                                   #5  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #6  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   
                                                   Stack trace of thread 35805:
                                                   #0  0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
                                                   #1  0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
                                                   #2  0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
                                                   #3  0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
                                                   #4  0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
                                                   #5  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #6  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   
                                                   Stack trace of thread 35803:
                                                   #0  0x000074d25db2bc7d accept (libc.so.6 + 0x117c7d)
                                                   #1  0x0000598847e5caf1 n/a (zerotier-one + 0x1b2af1)
                                                   #2  0x0000598847e364e4 n/a (zerotier-one + 0x18c4e4)
                                                   #3  0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
                                                   #4  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #5  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   
                                                   Stack trace of thread 35806:
                                                   #0  0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
                                                   #1  0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
                                                   #2  0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
                                                   #3  0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
                                                   #4  0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
                                                   #5  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #6  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   
                                                   Stack trace of thread 35807:
                                                   #0  0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
                                                   #1  0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
                                                   #2  0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
                                                   #3  0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
                                                   #4  0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
                                                   #5  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #6  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   
                                                   Stack trace of thread 35808:
                                                   #0  0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
                                                   #1  0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
                                                   #2  0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
                                                   #3  0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
                                                   #4  0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
                                                   #5  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #6  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   
                                                   Stack trace of thread 35813:
                                                   #0  0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
                                                   #1  0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
                                                   #2  0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
                                                   #3  0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
                                                   #4  0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
                                                   #5  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #6  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   
                                                   Stack trace of thread 35814:
                                                   #0  0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
                                                   #1  0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
                                                   #2  0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
                                                   #3  0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
                                                   #4  0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
                                                   #5  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #6  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   
                                                   Stack trace of thread 35809:
                                                   #0  0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
                                                   #1  0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
                                                   #2  0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
                                                   #3  0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
                                                   #4  0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
                                                   #5  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #6  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   
                                                   Stack trace of thread 35815:
                                                   #0  0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
                                                   #1  0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
                                                   #2  0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
                                                   #3  0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
                                                   #4  0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
                                                   #5  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #6  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   
                                                   Stack trace of thread 35810:
                                                   #0  0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
                                                   #1  0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
                                                   #2  0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
                                                   #3  0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
                                                   #4  0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
                                                   #5  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #6  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   
                                                   Stack trace of thread 35811:
                                                   #0  0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
                                                   #1  0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
                                                   #2  0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
                                                   #3  0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
                                                   #4  0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
                                                   #5  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #6  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   
                                                   Stack trace of thread 35816:
                                                   #0  0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
                                                   #1  0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
                                                   #2  0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
                                                   #3  0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
                                                   #4  0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
                                                   #5  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #6  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   
                                                   Stack trace of thread 35817:
                                                   #0  0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
                                                   #1  0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
                                                   #2  0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
                                                   #3  0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
                                                   #4  0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
                                                   #5  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #6  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   
                                                   Stack trace of thread 35818:
                                                   #0  0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
                                                   #1  0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
                                                   #2  0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
                                                   #3  0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
                                                   #4  0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
                                                   #5  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #6  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   
                                                   Stack trace of thread 35819:
                                                   #0  0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
                                                   #1  0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
                                                   #2  0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
                                                   #3  0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
                                                   #4  0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
                                                   #5  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
                                                   #6  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
                                                   ELF object binary architecture: AMD x86-64
Jul 06 09:09:47 archlinux systemd[1]: [email protected]: Deactivated successfully.
Jul 06 09:09:47 archlinux systemd-networkd[410]: zt*********: Link DOWN
Jul 06 09:09:47 archlinux systemd-networkd[410]: zt*********: Lost carrier
Jul 06 09:09:47 archlinux systemd[1]: zerotier-one.service: Main process exited, code=dumped, status=11/SEGV
Jul 06 09:09:47 archlinux systemd[1]: zerotier-one.service: Failed with result 'core-dump'.
Jul 06 09:09:47 archlinux systemd[1]: zerotier-one.service: Triggering OnFailure= dependencies.
Jul 06 09:09:47 archlinux systemd[1]: Started Send a notification about a failed systemd unit.
Jul 06 09:09:47 archlinux systemd[1]: zerotier-one.service: Scheduled restart job, restart counter is at 53.

[laduke]

#2325

[devvick]

I never had to do this workaround, and I never got the "error loading shared libraries" problem. I updated my systems with both the latest miniupnpc and zerotier-one packages, which had the fix already.
The problem is that somehow this version causes the zerotier-one process to crash with a segfault every 10 minutes or so.

[laduke]

Sorry about that. We include a specific version of miniupnpc in our builds. Arch must be doing something special.

You can disable upnp in local.conf portMappingEnabled

[devvick]

Doesn't look like it. The miniupnpc package doesn't have any patches, nor any special build commands:
https://gitlab.archlinux.org/archlinux/packaging/packages/miniupnpc/-/blob/main/PKGBUILD?ref_type=heads

zerotier-one package also looks bog-standard: https://gitlab.archlinux.org/archlinux/packaging/packages/zerotier-one/-/blob/main/PKGBUILD?ref_type=heads

Nevertheless, I will try to disable upnp in config (which I guess is probably broken anyway, since segfaults).

We include a specific version of miniupnpc in our builds

Which version are you including, specifically? Can you include the latest?

[glimberg]

It's part of the default build: https://github.com/zerotier/ZeroTierOne/tree/dev/ext/miniupnpc

[devvick]

I can see it's old (last commit a year ago, and VERSION file lists "2.0").

Looking at the latest changelog from the project, this seems significant:
http://miniupnp.free.fr/files/changelog.php?file=miniupnpc-2.2.8.tar.gz

2024/05/08:
  upnpc.c: Add -f option to upnpc program (delete multiple port redirections)
  UPNP_GetValidIGD(): distinguish between not connected and connected to a
    "private" network (with a reserved IP address).
  Increments API_VERSION to 18

My segfault stacktrace lists this exact function:

Stack trace of thread 35799:
#0  0x000074d25dacec46 n/a (libc.so.6 + 0xbac46)
#1  0x000074d25e7afc36 UPNP_GetValidIGD (libminiupnpc.so.18 + 0x9c36)
#2  0x0000598847e84a4e n/a (zerotier-one + 0x1daa4e)
#3  0x0000598847e848cd n/a (zerotier-one + 0x1da8cd)
#4  0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#5  0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)

[pejobo]

@devvick Were you able to check if disabling upnp fixes the issue?

[devvick]

Yes indeed, sorry for not responding sooner.
It helped with the crashes. However, I haven't tested the impact on connectivity between non-port-forwarded hosts (I use manual port forwarding on at least one of the endpoints in any of my configurations).