occasional segfaults during shutdown

[bochen87]

Hi @ipa-mdl , we're having occasional segfaults when shutting down which leads to our tests occasionally failing.
I was wondering, is 1acd493 trying to tackle that issue?

In our dtor, it's simply just this piece of code, where can_driver_ is an object of type can::ThreadedSocketCANInterface

try
  {
    can_driver_.shutdown();
  }
  catch (const std::exception& e)
  {
    ROS_ERROR_STREAM("An exception occured while trying to shutdown the CAN driver");
    ROS_ERROR_STREAM(e.what());
  }

[bochen87]

Here a stacktrace i was able to capture, we've since upgraded to the latest released debian package version, but it is still occasionally happening:

#0  0x0000000000000000 in ?? ()
#1  0x0000555555601813 in boost::asio::detail::scheduler_operation::destroy (this=0x5555556d2260) at /usr/include/boost/asio/detail/scheduler_operation.hpp:45
#2  0x000055555561fd37 in boost::asio::detail::op_queue_access::destroy<boost::asio::detail::scheduler_operation> (o=0x5555556d2260) at /usr/include/boost/asio/detail/op_queue.hpp:47
#3  0x0000555555616184 in boost::asio::detail::op_queue<boost::asio::detail::scheduler_operation>::~op_queue (this=0x5555556d2288, __in_chrg=<optimized out>) at /usr/include/boost/asio/detail/op_queue.hpp:81
#4  0x000055555560963d in boost::asio::detail::scheduler::~scheduler (this=0x5555556d21b0, __in_chrg=<optimized out>) at /usr/include/boost/asio/detail/impl/scheduler.ipp:135
#5  0x000055555560970c in boost::asio::detail::scheduler::~scheduler (this=0x5555556d21b0, __in_chrg=<optimized out>) at /usr/include/boost/asio/detail/impl/scheduler.ipp:142
#6  0x0000555555602ec6 in boost::asio::detail::service_registry::destroy (service=0x5555556d21b0) at /usr/include/boost/asio/detail/impl/service_registry.ipp:110
#7  0x0000555555602d35 in boost::asio::detail::service_registry::destroy_services (this=0x5555556d3360) at /usr/include/boost/asio/detail/impl/service_registry.ipp:54
#8  0x00005555556036c1 in boost::asio::execution_context::destroy (this=0x7fffffffccd8) at /usr/include/boost/asio/impl/execution_context.ipp:46
#9  0x00005555556035bb in boost::asio::execution_context::~execution_context (this=0x7fffffffccd8, __in_chrg=<optimized out>) at /usr/include/boost/asio/impl/execution_context.ipp:35
#10 0x000055555560b67a in boost::asio::io_context::~io_context (this=0x7fffffffccd8, __in_chrg=<optimized out>) at /usr/include/boost/asio/impl/io_context.ipp:56
#11 0x0000555555618bc7 in can::AsioDriver<boost::asio::posix::basic_stream_descriptor<boost::asio::executor> >::~AsioDriver (this=0x7fffffffcbb0, __in_chrg=<optimized out>) at ros_canopen/socketcan_interface/include/socketcan_interface/asio_base.h:81
#12 0x000055555561a1cf in can::SocketCANInterface::~SocketCANInterface (this=0x7fffffffcbb0, __in_chrg=<optimized out>) at ros_canopen/socketcan_interface/include/socketcan_interface/socketcan.h:20
#13 0x000055555561a3e5 in can::ThreadedInterface<can::SocketCANInterface>::~ThreadedInterface (this=0x7fffffffcbb0, __in_chrg=<optimized out>) at ros_canopen/socketcan_interface/include/socketcan_interface/threading.h:68
#14 0x0000555555613876 in CANSender::~CANSender (this=0x7fffffffcbb0, __in_chrg=<optimized out>) at test_receiver.cpp:39
#15 0x00005555555f6a8c in OpModeReceiver_invalidMode_Test::TestBody (this=0x5555556d2120) at test_receiver.cpp:77
#16 0x00007ffff7c59a99 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x5555556d2120, method=&virtual testing::Test::TestBody(), location=0x7ffff7c6edab "the test body") at /usr/src/googletest/googletest/src/gtest.cc:2433
#17 0x00007ffff7c521b1 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x5555556d2120, method=&virtual testing::Test::TestBody(), location=0x7ffff7c6edab "the test body") at /usr/src/googletest/googletest/src/gtest.cc:2469
#18 0x00007ffff7c2c556 in testing::Test::Run (this=0x5555556d2120) at /usr/src/googletest/googletest/src/gtest.cc:2508
#19 0x00007ffff7c2cf41 in testing::TestInfo::Run (this=0x5555556d1600) at /usr/src/googletest/googletest/src/gtest.cc:2684
#20 0x00007ffff7c2d699 in testing::TestSuite::Run (this=0x5555556d1450) at /usr/src/googletest/googletest/src/gtest.cc:2816
#21 0x00007ffff7c39843 in testing::internal::UnitTestImpl::RunAllTests (this=0x5555556d0a30) at /usr/src/googletest/googletest/src/gtest.cc:5338
#22 0x00007ffff7c5afc2 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x5555556d0a30, method=(bool (testing::internal::UnitTestImpl::*)(class testing::internal::UnitTestImpl * const)) 0x7ffff7c39426 <testing::internal::UnitTestImpl::RunAllTests()>,
    location=0x7ffff7c6f7e8 "auxiliary test code (environments or event listeners)") at /usr/src/googletest/googletest/src/gtest.cc:2433
#23 0x00007ffff7c533ef in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x5555556d0a30, method=(bool (testing::internal::UnitTestImpl::*)(class testing::internal::UnitTestImpl * const)) 0x7ffff7c39426 <testing::internal::UnitTestImpl::RunAllTests()>,
    location=0x7ffff7c6f7e8 "auxiliary test code (environments or event listeners)") at /usr/src/googletest/googletest/src/gtest.cc:2469
#24 0x00007ffff7c3804d in testing::UnitTest::Run (this=0x7ffff7c995e0 <testing::UnitTest::GetInstance()::instance>) at /usr/src/googletest/googletest/src/gtest.cc:4925
#25 0x00007ffff7f4d29a in RUN_ALL_TESTS () at /usr/src/googletest/googletest/include/gtest/gtest.h:2473
#26 0x00007ffff7f4d21c in main (argc=1, argv=0x7fffffffd508) at /usr/src/googletest/googletest/src/gtest_main.cc:45
#27 0x00007ffff76850b3 in __libc_start_main (main=0x7ffff7f4d1d9 <main(int, char**)>, argc=2, argv=0x7fffffffd508, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffd4f8) at ../csu/libc-start.c:308
#28 0x00005555555f571e in _start ()

[mathias-luedtke]

Thanks for reporting!

Are you calling shutdown explicitly in your destructor? (even without the try/catch block).

1acd493 should have improved the shutdown behavior, but it mostly was about not call virtual member in the destructor..

[bochen87]

Thanks for reporting!

Are you calling shutdown explicitly in your destructor? (even without the try/catch block).

1acd493 should have improved the shutdown behavior, but it mostly was about not call virtual member in the destructor..

We are calling it in the destructor exactly the way I have copied the code snippet above with the try / catch block.
But it segfaults every once in a while. My guess is there is some race in destructing threads and once in a while gtest is faster and that's why boost asio gets a segfault?