cyclonedds kills nodes with THREAD_STATE_ALIVE failed #486

yvzksgl · 2024-03-14T14:29:08Z

Bug report

Required Info:

Operating System:
- Ubuntu 22.04.4 LTS
Installation type:
- Binaries
Version or commit hash:
- Humble
DDS implementation:
- rmw_cyclonedds_cpp
Client library (if applicable):
- rclcpp

My related sysctl settings:
net.core.rmem_default = 212992
net.core.rmem_max = 2147483647
net.ipv4.ipfrag_time = 3

While I was running more than 200 nodes on my PC. One of my nodes dies with:

/opt/ros/humble/include/dds/ddsi/q_thread.h:228: thread_state_awake_fixed_domain: Assertion 'thrst->state == THREAD_STATE_ALIVE' failed.

In the readme it says to set net.core.rmem_max=8388608 net.core.rmem_default=8388608. Could my error related with these settings. Should I set them to exactly 8388608 or can I set both 2147483647. Do I have to set max and defaults to same value?

Thanks in advance

The text was updated successfully, but these errors were encountered:

armaganarsln · 2024-03-14T17:41:02Z

Could you add steps to replicate this issue?

fujitatomoya · 2024-03-21T19:26:31Z

@yvzksgl could you share reproducible self-contained sample as @armaganarsln asked? that would be helpful.

CC: @eboasson

eboasson · 2024-03-22T09:08:23Z

While I was running more than 200 nodes on my PC. One of my nodes dies with:
/opt/ros/humble/include/dds/ddsi/q_thread.h:228: thread_state_awake_fixed_domain:  Assertion 'thrst->state == THREAD_STATE_ALIVE' failed.
In the readme it says to set net.core.rmem_max=8388608 net.core.rmem_default=8388608. Could my error related with these settings. Should I set them to exactly 8388608 or can I set both 2147483647. Do I have to set max and defaults to same value?

Don't worry about the socket buffer configuration stuff in relation to this assertion failure. Cyclone internally tracks what threads are doing to defer releasing memory in some cases.

This is an internal thread of Cyclone trying to update some counter related to that mechanism. I know it is an internal thread because it uses the fixed_domain variant, which by definition cannot be the case for an application thread because the application is allowed to use multiple (DDS) domains at the same time.

It really is weird that it complains the thread is not registered as "alive". For internal threads, the control block is allocated and initialized before the thread starts doing things and cleaned up after it left the thread's main function. My first thought is that this is bug that we're lucky to have discovered the existence of.

I have a suspicion that a self-contained sample might be tricky ... but as a first step towards figuring out what is happening, perhaps you can reproduce it and get stack traces of all the threads? Like, let it core dump, load the core dump in gdb and do thread apply all bt. That should be a manageable amount of text and with a bit of luck it gives me hint.

yvzksgl · 2024-03-22T11:28:10Z

Thank you all for your comments. @eboasson I will apply steps you mentioned. However the bug rarely occurs. When I encountered the error again, i will try to provide all you need.

yvzksgl · 2024-03-22T14:57:19Z

Hello again. I managed to reproduce the issue. There are '??'s in binary files in backtrace. I tried to solve it, but the solutions i found didn't work. So I decided to share output as it is. I hope it works. Here is the output:

clap_b7_driver_node: ./src/security/api/../../core/ddsi/include/dds/ddsi/q_thread.h:228: thread_state_awake_fixed_domain: Assertion `thrst->state == THREAD_STATE_ALIVE' failed.
clap_b7_driver_node: ./src/security/api/../../core/ddsi/include/dds/ddsi/q_thread.h:228: thread_state_awake_fixed_domain: Assertion `thrst->state == THREAD_STATE_ALIVE' failed.
clap_b7_driver_node: ./src/security/api/../../core/ddsi/include/dds/ddsi/q_thread.h:228: thread_state_awake_fixed_domain: Assertion `thrst->state == THREAD_STATE_ALIVE' failed.

Thread 6 "tev" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff4b8e640 (LWP 2759910)]
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140737299146304) at ./nptl/pthread_kill.c:44
44      ./nptl/pthread_kill.c: No such file or directory.
warning: target file /proc/2753237/cmdline contained unexpected null characters
warning: Memory read failed for corefile section, 4096 bytes at 0xffffffffff600000.
Saved corefile core.2753237
(gdb) thread apply all bt

Thread 11 (Thread 0x7fffe5fe2640 (LWP 2764333) "clap_b7_driver_"):
#0  0x00007ffff7525fde in epoll_wait (epfd=12, events=0x7fffe5fe0df0, maxevents=128, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x00005555555b592d in boost::asio::detail::epoll_reactor::run (this=0x5555556ace60, usec=<optimized out>, ops=...) at /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:471
#2  0x00005555555b70aa in boost::asio::detail::scheduler::do_run_one (this=this@entry=0x5555556cc330, lock=..., this_thread=..., ec=...) at /usr/include/boost/asio/detail/impl/scheduler.ipp:465
#3  0x00005555555b7ad9 in boost::asio::detail::scheduler::run (this=0x5555556cc330, ec=...) at /usr/include/boost/asio/detail/impl/scheduler.ipp:204
#4  0x00005555555b9952 in boost::asio::io_context::run (this=<optimized out>) at /usr/include/boost/asio/impl/io_context.ipp:63
#5  0x00007ffff78dc253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007ffff7494b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#7  0x00007ffff7526a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 10 (Thread 0x7fffe67fc640 (LWP 2760146) "clap_b7_driver_"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7fffcc000ddc) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7fffcc000ddc) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fffcc000ddc, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff7493ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fffcc000d88, cond=0x7fffcc000db0) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7fffcc000db0, mutex=0x7fffcc000d88) at ./nptl/pthread_cond_wait.c:627
#5  0x00007ffff731686d in ddsrt_cond_wait () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#6  0x00007ffff7316935 in ddsrt_cond_waituntil () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#7  0x00007ffff72fd22d in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#8  0x00007ffff73a9d2e in ?? () from /opt/ros/humble/lib/librmw_cyclonedds_cpp.so
#9  0x00007ffff78dc253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007ffff7494b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007ffff7526a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 9 (Thread 0x7fffe6ffd640 (LWP 2760005) "recvUC"):
#0  __recvmsg_syscall (flags=0, msg=0x7fffe6ffc2a0, fd=5) at ../sysdeps/unix/sysv/linux/recvmsg.c:27
#1  __libc_recvmsg (fd=5, msg=0x7fffe6ffc2a0, flags=0) at ../sysdeps/unix/sysv/linux/recvmsg.c:41
#2  0x00007ffff7312c5d in ddsrt_recvmsg () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#3  0x00007ffff726a511 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#4  0x00007ffff72d0b8e in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#5  0x00007ffff72d0e0d in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#6  0x00007ffff72cb2c5 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#7  0x00007ffff7316bec in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#8  0x00007ffff7494b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#9  0x00007ffff7526a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 8 (Thread 0x7fffe77fe640 (LWP 2759959) "recvMC"):
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737077306944) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140737077306944) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140737077306944, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7442476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff74287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff742871b in __assert_fail_base (fmt=0x7ffff75dd150 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7ffff733d728 "thrst->state == THREAD_STATE_ALIVE", file=0x7ffff7328d48 "./src/security/api/../../core/ddsi/include/dds/ddsi/q_thread.h", line=228, function=<optimized out>) at ./assert/assert.c:92
#6  0x00007ffff7439e96 in __GI___assert_fail (assertion=0x7ffff733d728 "thrst->state == THREAD_STATE_ALIVE", file=0x7ffff7328d48 "./src/security/api/../../core/ddsi/include/dds/ddsi/q_thread.h", line=228, function=0x7ffff733d940 "thread_state_awake_fixed_domain") at ./assert/assert.c:101
#7  0x00007ffff72c7f39 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#8  0x00007ffff7273b90 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#9  0x00007ffff72cda19 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#10 0x00007ffff72d0cc4 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#11 0x00007ffff72d0e0d in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#12 0x00007ffff72cb2c5 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#13 0x00007ffff7316bec in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#14 0x00007ffff7494b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#15 0x00007ffff7526a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 7 (Thread 0x7fffe7fff640 (LWP 2759957) "recv"):
#0  0x00007ffff751b7ed in __GI___select (nfds=10, readfds=0x5555556672e0, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69
#1  0x00007ffff7312e64 in ddsrt_select () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#2  0x00007ffff72d101a in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#3  0x00007ffff72cb2c5 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#4  0x00007ffff7316bec in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#5  0x00007ffff7494b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007ffff7526a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 6 (Thread 0x7ffff4b8e640 (LWP 2759910) "tev"):
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737299146304) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140737299146304) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140737299146304, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7442476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff74287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff742871b in __assert_fail_base (fmt=0x7ffff75dd150 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7ffff733d728 "thrst->state == THREAD_STATE_ALIVE", file=0x7ffff7328d48 "./src/security/api/../../core/ddsi/include/dds/ddsi/q_thread.h", line=228, function=<optimized out>) at ./assert/assert.c:92

While trying to reproduce the issue, dds asserted one more exception:

clap_b7_driver_node: ./src/core/ddsi/src/q_receive.c:3204: handle_rtps_message: Assertion `thread_is_asleep ()' failed.

Thread 8 "recvMC" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffe77fe640 (LWP 3454302)]
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140737077306944) at ./nptl/pthread_kill.c:44
44      ./nptl/pthread_kill.c: No such file or directory.
warning: target file /proc/3446161/cmdline contained unexpected null characters
warning: Memory read failed for corefile section, 4096 bytes at 0xffffffffff600000.
Saved corefile core.3446161
(gdb) thread apply all
Please specify a command at the end of 'thread apply all'
(gdb) thread apply all bt

Thread 11 (Thread 0x7fffe5fe2640 (LWP 3455367) "clap_b7_driver_"):
#0  0x00007ffff7525fde in epoll_wait (epfd=12, events=0x7fffe5fe0df0, maxevents=128, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x00005555555b592d in boost::asio::detail::epoll_reactor::run (this=0x5555556ad790, usec=<optimized out>, ops=...) at /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:471
#2  0x00005555555b70aa in boost::asio::detail::scheduler::do_run_one (this=this@entry=0x5555556cd760, lock=..., this_thread=..., ec=...) at /usr/include/boost/asio/detail/impl/scheduler.ipp:465
#3  0x00005555555b7ad9 in boost::asio::detail::scheduler::run (this=0x5555556cd760, ec=...) at /usr/include/boost/asio/detail/impl/scheduler.ipp:204
#4  0x00005555555b9952 in boost::asio::io_context::run (this=<optimized out>) at /usr/include/boost/asio/impl/io_context.ipp:63
#5  0x00007ffff78dc253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007ffff7494b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#7  0x00007ffff7526a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 10 (Thread 0x7fffe67fc640 (LWP 3454387) "clap_b7_driver_"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7fffd4000dd8) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7fffd4000dd8) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fffd4000dd8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff7493ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fffd4000d88, cond=0x7fffd4000db0) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7fffd4000db0, mutex=0x7fffd4000d88) at ./nptl/pthread_cond_wait.c:627
#5  0x00007ffff731686d in ddsrt_cond_wait () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#6  0x00007ffff7316935 in ddsrt_cond_waituntil () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#7  0x00007ffff72fd22d in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#8  0x00007ffff73a9d2e in ?? () from /opt/ros/humble/lib/librmw_cyclonedds_cpp.so
#9  0x00007ffff78dc253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007ffff7494b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007ffff7526a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 9 (Thread 0x7fffe6ffd640 (LWP 3454305) "recvUC"):
#0  __recvmsg_syscall (flags=0, msg=0x7fffe6ffc2a0, fd=5) at ../sysdeps/unix/sysv/linux/recvmsg.c:27
#1  __libc_recvmsg (fd=5, msg=0x7fffe6ffc2a0, flags=0) at ../sysdeps/unix/sysv/linux/recvmsg.c:41
#2  0x00007ffff7312c5d in ddsrt_recvmsg () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#3  0x00007ffff726a511 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#4  0x00007ffff72d0b8e in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#5  0x00007ffff72d0e0d in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#6  0x00007ffff72cb2c5 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#7  0x00007ffff7316bec in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#8  0x00007ffff7494b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#9  0x00007ffff7526a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 8 (Thread 0x7fffe77fe640 (LWP 3454302) "recvMC"):
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737077306944) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140737077306944) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140737077306944, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7442476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff74287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff742871b in __assert_fail_base (fmt=0x7ffff75dd150 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7ffff7343723 "thread_is_asleep ()", file=0x7ffff7344628 "./src/core/ddsi/src/q_receive.c", line=3204, function=<optimized out>) at ./assert/assert.c:92
#6  0x00007ffff7439e96 in __GI___assert_fail (assertion=0x7ffff7343723 "thread_is_asleep ()", file=0x7ffff7344628 "./src/core/ddsi/src/q_receive.c", line=3204, function=0x7ffff7344290 "handle_rtps_message") at ./assert/assert.c:101
#7  0x00007ffff72cf867 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#8  0x00007ffff72d0cc4 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#9  0x00007ffff72d0e0d in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#10 0x00007ffff72cb2c5 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#11 0x00007ffff7316bec in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#12 0x00007ffff7494b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007ffff7526a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 7 (Thread 0x7fffe7fff640 (LWP 3454134) "recv"):
#0  0x00007ffff751b7ed in __GI___select (nfds=10, readfds=0x5555556672e0, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69
#1  0x00007ffff7312e64 in ddsrt_select () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#2  0x00007ffff72d101a in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#3  0x00007ffff72cb2c5 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#4  0x00007ffff7316bec in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#5  0x00007ffff7494b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007ffff7526a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 6 (Thread 0x7ffff4b8e640 (LWP 3454095) "tev"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7ffff4b8d410, op=393, expected=0, futex_word=0x555555657efc) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x7ffff4b8d410, clockid=0, expected=0, futex_word=0x555555657efc) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x555555657efc, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7ffff4b8d410, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff7493f1b in __pthread_cond_wait_common (abstime=0x7ffff4b8d410, clockid=0, mutex=0x555555657ea8, cond=0x555555657ed0) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_timedwait64 (cond=0x555555657ed0, mutex=0x555555657ea8, abstime=0x7ffff4b8d410) at ./nptl/pthread_cond_wait.c:652
#5  0x00007ffff73168f9 in ddsrt_cond_waituntil () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#6  0x00007ffff72d89e8 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0

I hope these will be helpful. I can try to provide whatever else you need. Thanks in advance. @eboasson

eboasson · 2024-03-22T15:32:00Z

The dreaded no-debug-symbols villain strikes again ...

The second one could plausibly be a consequence of the first one: thread_is_asleep doesn't check whether it "alive" or whatever, it only checks whether it is asleep. "Could plausibly be" doesn't mean "is", but every now and then one has to stick one's neck out 🙂

It does give something to ponder. There are actually multiple threads asserting the same thing at the same time: "tev" and "recvMC". That makes it less likely that it is some quirk in one specific thread, and makes it more likely that the array(s) of thread states have been corrupted.

It looks like you don't have many threads in this process, but that doesn't necessarily mean it can't have anything to do with a7dcf8691edf284b775c6fb9e926eeff4591e9c9 because you could be creating and deleting threads all the time. Are you by chance?

yvzksgl · 2024-03-26T10:48:02Z

Thak you for your support @eboasson. I have functions in node that use threads. I will try to perform operations using something else instead of using threads in node. I will then repeat the tests I did while trying to kill the node. I will share results here.

PS: a7dcf8691edf284b775c6fb9e926eeff4591e9c9 do you have any chance for repost the link? I cannot access.

eboasson · 2024-03-26T12:46:32Z

Usually GitHub auto creates a link for a commit number: eclipse-cyclonedds/cyclonedds@a7dcf86

Related to it is also eclipse-cyclonedds/cyclonedds@b0727e5.

Those two removed the old hard-coded limit on the thread count. I am not aware of any problems with that work, except that your issue made me think of it because your issue seems to have something to do with messing up these tables of thread states. I probably only confused you by mentioning it ... I am sorry about that!

yvzksgl · 2024-04-05T10:37:17Z

Hi, I eliminated thread implementation in my node. It seems to have fixed the problem. However, I'm not sure if the problem is caused by wheter my thread implementation or dds. On my previous comments I accidentally shared an incomplete output of gdb. Here is the full output of thread apply all bt after exception:

clap_b7_driver_node: ./src/core/ddsi/src/q_receive.c:3204: handle_rtps_message: Assertion `thread_is_asleep ()' failed.
clap_b7_driver_node: ./src/security/api/../../core/ddsi/include/dds/ddsi/q_thread.h:228: thread_state_awake_fixed_domain: Assertion `thrst->state == THREAD_STATE_ALIVE' failed.
clap_b7_driver_node: ./src/security/api/../../core/ddsi/include/dds/ddsi/q_thread.h:228: thread_state_awake_fixed_domain: Assertion `thrst->state == THREAD_STATE_ALIVE' failed.

Thread 8 "recvMC" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffeffff640 (LWP 1451600)]
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140737219917376) at ./nptl/pthread_kill.c:44
44      ./nptl/pthread_kill.c: No such file or directory.
warning: target file /proc/1445318/cmdline contained unexpected null characters
warning: Memory read failed for corefile section, 4096 bytes at 0xffffffffff600000.
Saved corefile core.1445318
(gdb) 
(gdb) thread apply all bt

Thread 11 (Thread 0x7fffee7e3640 (LWP 1453752) "clap_b7_driver_"):
#0  0x00007ffff752601e in epoll_wait (epfd=12, events=0x7fffee7e1df0, maxevents=128, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x00005555555a855d in boost::asio::detail::epoll_reactor::run (this=0x5555556523a0, usec=<optimized out>, ops=...) at /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:471
#2  0x00005555555a9cda in boost::asio::detail::scheduler::do_run_one (this=this@entry=0x5555556ba5f0, lock=..., this_thread=..., ec=...) at /usr/include/boost/asio/detail/impl/scheduler.ipp:465
#3  0x00005555555aa709 in boost::asio::detail::scheduler::run (this=0x5555556ba5f0, ec=...) at /usr/include/boost/asio/detail/impl/scheduler.ipp:204
#4  0x00005555555ac582 in boost::asio::io_context::run (this=<optimized out>) at /usr/include/boost/asio/impl/io_context.ipp:63
#5  0x00007ffff78dc253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007ffff7494ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#7  0x00007ffff7526a40 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 10 (Thread 0x7fffeeffd640 (LWP 1451806) "clap_b7_driver_"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7fffd4000ddc) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7fffd4000ddc) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fffd4000ddc, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff7493a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fffd4000d88, cond=0x7fffd4000db0) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7fffd4000db0, mutex=0x7fffd4000d88) at ./nptl/pthread_cond_wait.c:627
#5  0x00007ffff731686d in ddsrt_cond_wait () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#6  0x00007ffff7316935 in ddsrt_cond_waituntil () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#7  0x00007ffff72fd22d in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#8  0x00007ffff73a9d2e in ?? () from /opt/ros/humble/lib/librmw_cyclonedds_cpp.so
#9  0x00007ffff78dc253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007ffff7494ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007ffff7526a40 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 9 (Thread 0x7fffef7fe640 (LWP 1451603) "recvUC"):
#0  __recvmsg_syscall (flags=0, msg=0x7fffef7fd2a0, fd=5) at ../sysdeps/unix/sysv/linux/recvmsg.c:27
#1  __libc_recvmsg (fd=5, msg=0x7fffef7fd2a0, flags=0) at ../sysdeps/unix/sysv/linux/recvmsg.c:41
#2  0x00007ffff7312c5d in ddsrt_recvmsg () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#3  0x00007ffff726a511 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libd
dsc.so.0
#4  0x00007ffff72d0b8e in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#5  0x00007ffff72d0e0d in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#6  0x00007ffff72cb2c5 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#7  0x00007ffff7316bec in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#8  0x00007ffff7494ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#9  0x00007ffff7526a40 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 8 (Thread 0x7fffeffff640 (LWP 1451600) "recvMC"):
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737219917376) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140737219917376) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140737219917376, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7442476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff74287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff742871b in __assert_fail_base (fmt=0x7ffff75dd150 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7ffff7343723 "thread_is_asleep ()", file=0x7ffff7344628 "./src/core/ddsi/src/q_receive.c", line=3204, function=<optimized out>) at ./assert/assert.c:92
#6  0x00007ffff7439e96 in __GI___assert_fail (assertion=0x7ffff7343723 "thread_is_asleep ()", file=0x7ffff7344628 "./src/core/ddsi/src/q_receive.c", line=3204, function=0x7ffff7344290 "handle_rtps_message") at ./assert/assert.c:101
#7  0x00007ffff72cf867 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#8  0x00007ffff72d0cc4 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#9  0x00007ffff72d0e0d in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#10 0x00007ffff72cb2c5 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#11 0x00007ffff7316bec in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libd
dsc.so.0
#12 0x00007ffff7494ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007ffff7526a40 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 7 (Thread 0x7ffff4a8d640 (LWP 1451598) "recv"):
#0  0x00007ffff751b82d in __GI___select (nfds=10, readfds=0x555555656390, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69
#1  0x00007ffff7312e64 in ddsrt_select () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#2  0x00007ffff72d101a in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#3  0x00007ffff72cb2c5 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#4  0x00007ffff7316bec in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#5  0x00007ffff7494ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007ffff7526a40 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 6 (Thread 0x7ffff538f640 (LWP 1451595) "tev"):
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737307539008) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140737307539008) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140737307539008, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7442476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff74287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff742871b in __assert_fail_base (fmt=0x7ffff75dd150 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7ffff733d728 "thrst->state == THRE
AD_STATE_ALIVE", file=0x7ffff7328d48 "./src/security/api/../../core/ddsi/include/dds/ddsi/q_thread.h", line=228, function=<optimized out>) at ./assert/assert.c:92
#6  0x00007ffff7439e96 in __GI___assert_fail (assertion=0x7ffff733d728 "thrst->state == THREAD_STATE_ALIVE", file=0x7ffff7328d48 "./src/security/api/../../core/ddsi/include/dds/ddsi/q_thread.h", line=228, function=0x7ffff733d940 "thread_state_awake_fixed_domain") at ./assert/assert.c:101
#7  0x00007ffff72d8d78 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0                                                                 ri
#8  0x00007ffff72cb2c5 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#9  0x00007ffff7316bec in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#10 0x00007ffff7494ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007ffff7526a40 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 5 (Thread 0x7ffff5b90640 (LWP 1451592) "dq.user"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555564fa30) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55555564fa30) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55555564fa30, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff7493a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55555564f9e0, cond=0x55555564fa08) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x55555564fa08, mutex=0x55555564f9e0) at ./nptl/pthread_cond_wait.c:627
#5  0x00007ffff731686d in ddsrt_cond_wait () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#6  0x00007ffff72c76ac in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#7  0x00007ffff72cb2c5 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#8  0x00007ffff7316bec in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#9  0x00007ffff7494ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007ffff7526a40 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 4 (Thread 0x7fffedb90640 (LWP 1451589) "dq.builtins"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555564f990) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55555564f990) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55555564f990, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff7493a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55555564f940, cond=0x55555564f968) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x55555564f968, mutex=0x55555564f940) at ./nptl/pthread_cond_wait.c:627
#5  0x00007ffff731686d in ddsrt_cond_wait () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#6  0x00007ffff72c76ac in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#7  0x00007ffff72cb2c5 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#8  0x00007ffff7316bec in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#9  0x00007ffff7494ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007ffff7526a40 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x7ffff6391640 (LWP 1451580) "gc"):
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737324324416) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140737324324416) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140737324324416, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7442476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff74287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff742871b in __assert_fail_base (fmt=0x7ffff75dd150 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7ffff733d728 "thrst->state == THREAD_STATE_ALIVE", file=0x7ffff7328d48 "./src/security/api/../../core/ddsi/include/dds/ddsi/q_thread.h", line=228, function=<optimized out>) at ./assert/assert.c:92
#6  0x00007ffff7439e96 in __GI___assert_fail (assertion=0x7ffff733d728 "thrst->state == THREAD_STATE_ALIVE", file=0x7ffff7328d48 "./src/security/api/../../core/ddsi/include/dds/ddsi/q_thread.h", line=228, function=0x7ffff733d940 "thread_state_awake_fixed_domain") at ./assert/assert.c:101
#7  0x00007ffff72ba60f in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#8  0x00007ffff72cb2c5 in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#9  0x00007ffff7316bec in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#10 0x00007ffff7494ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007ffff7526a40 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7ffff6b92640 (LWP 1451552) "clap_b7_driver_"):
#0  __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffff7dcff08) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffff7dcff08) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffff7dcff08, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
#3  0x00007ffff749cbdf in do_futex_wait (sem=sem@entry=0x7ffff7dcff08, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
#4  0x00007ffff749cc78 in __new_sem_wait_slow64 (sem=0x7ffff7dcff08, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
#5  0x00007ffff7d37a72 in rclcpp::SignalHandler::wait_for_signal() () from /opt/ros/humble/lib/librclcpp.so
#6  0x00007ffff7d38a4e in rclcpp::SignalHandler::deferred_signal_handler() () from /opt/ros/humble/lib/librclcpp.so
#7  0x00007ffff78dc253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007ffff7494ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#9  0x00007ffff7526a40 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7ffff7681f80 (LWP 1445318) "clap_b7_driver_"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555570c06c) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55555570c06c) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55555570c06c, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff7493a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55555570c018, cond=0x55555570c040) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x55555570c040, mutex=0x55555570c018) at ./nptl/pthread_cond_wait.c:627
#5  0x00007ffff731686d in ddsrt_cond_wait () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#6  0x00007ffff7316935 in ddsrt_cond_waituntil () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#7  0x00007ffff72fd22d in ?? () from /opt/ros/humble/lib/x86_64-linux-gnu/libddsc.so.0
#8  0x00007ffff73b974f in rmw_wait () from /opt/ros/humble/lib/librmw_cyclonedds_cpp.so
#9  0x00007ffff7bb8768 in rcl_wait () from /opt/ros/humble/lib/librcl.so
#10 0x00007ffff7cbe69c in rclcpp::Executor::wait_for_work(std::chrono::duration<long, std::ratio<1l, 1000000000l> >) () from /opt/ros/humble/lib/librclcpp.so
#11 0x00007ffff7cc13b3 in rclcpp::Executor::get_next_executable(rclcpp::AnyExecutable&, std::chrono::duration<long, std::ratio<1l, 1000000000l> >) () from /opt/ros/humble/lib/librclcpp.so
#12 0x00007ffff7cc8891 in rclcpp::executors::SingleThreadedExecutor::spin() () from /opt/ros/humble/lib/librclcpp.so
#13 0x00007ffff7cc8ab5 in rclcpp::spin(std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface>) () from /opt/ros/humble/lib/librclcpp.so
#14 0x00007ffff7cc8baf in rclcpp::spin(std::shared_ptr<rclcpp::Node>) () from /opt/ros/humble/lib/librclcpp.so
#15 0x00005555555791a1 in main (argc=<optimized out>, argv=<optimized out>) at /home/robicarr/projects/autoware.robicarr/src/sensor_component/external/clap_b7_driver/clap_b7_driver/src/clap_b7_driver_node.cpp:9

I hope it will give some tips.

quarkytale assigned eboasson Apr 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cyclonedds kills nodes with THREAD_STATE_ALIVE failed #486

cyclonedds kills nodes with THREAD_STATE_ALIVE failed #486

yvzksgl commented Mar 14, 2024 •

edited

Loading

armaganarsln commented Mar 14, 2024

fujitatomoya commented Mar 21, 2024

eboasson commented Mar 22, 2024

yvzksgl commented Mar 22, 2024 •

edited

Loading

yvzksgl commented Mar 22, 2024 •

edited

Loading

eboasson commented Mar 22, 2024

yvzksgl commented Mar 26, 2024 •

edited

Loading

eboasson commented Mar 26, 2024

yvzksgl commented Apr 5, 2024

cyclonedds kills nodes with THREAD_STATE_ALIVE failed #486

cyclonedds kills nodes with THREAD_STATE_ALIVE failed #486

Comments

yvzksgl commented Mar 14, 2024 • edited Loading

Bug report

armaganarsln commented Mar 14, 2024

fujitatomoya commented Mar 21, 2024

eboasson commented Mar 22, 2024

yvzksgl commented Mar 22, 2024 • edited Loading

yvzksgl commented Mar 22, 2024 • edited Loading

eboasson commented Mar 22, 2024

yvzksgl commented Mar 26, 2024 • edited Loading

eboasson commented Mar 26, 2024

yvzksgl commented Apr 5, 2024

yvzksgl commented Mar 14, 2024 •

edited

Loading

yvzksgl commented Mar 22, 2024 •

edited

Loading

yvzksgl commented Mar 22, 2024 •

edited

Loading

yvzksgl commented Mar 26, 2024 •

edited

Loading