ThreadSanitizer: data race during ResponseWriter destructor

[TheQue42]

Hi,

When testing my own application using GCC's thread sanitizer, I am occasionally receiving complaints with a backtrace that traces all the way down into pistache (when pistache is build with -fsanitize=thread -fno-omit-frame-pointer.

I am starting pistache via serve(), from its own std::jthread (simply because I got more complaints when using serveThreaded()).

I am using pistache in an asynchronous way, so that when requests are received via my Rest::Router to my endpoints, a different thread will fetch the response contents via a proprietary API (aka IMMI), and while waiting for that response I will clone the Http::ResponseWriter such as:

ImmiHandler::ImmiHandler(EcmiApi& ecmiSrv, Http::ResponseWriter& rsp)
    : m_ecmiApi(ecmiSrv),
      m_rspWriter(std::move(rsp.clone()))    // A plain Pistache::Http::ResponseWriter member in ImmiHandler.
{
    NW_TRACE_OBNOXIOUS(NwTrace::Ecmi, "", this, &rspWriter);
    NW_TRACE_OBJECT(this);
};

Once the ImmiHandler receives its results, it uses the m_rspWriter to send the response, and eventually the ImmiHandler will die, taking the m_rspWriter with it.

But that's were I get (and where I need help analyzing the complaint from GCC 14)

WARNING: ThreadSanitizer: data race (pid=83694)
  Write of size 8 at 0x724400042688 by thread T10 (mutexes: write M0, write M1):
    #0 operator delete(void*, unsigned long) ../../../../libsanitizer/tsan/tsan_new_delete.cpp:150 (libtsan.so.2+0xa4a2b) (BuildId: f57d904f7f377f8ad981d755372777f1e90df49e)
    #1 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /usr/include/c++/14/bits/shared_ptr_base.h:347 (ccm+0xacc1c9) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #2 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /usr/include/c++/14/bits/shared_ptr_base.h:317 (ccm+0xacc1c9)
    #3 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/include/c++/14/bits/shared_ptr_base.h:1069 (ccm+0xacc1c9)
    #4 std::__shared_ptr<Pistache::Tcp::Peer, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/include/c++/14/bits/shared_ptr_base.h:1525 (ccm+0xacc1c9)
    #5 std::shared_ptr<Pistache::Tcp::Peer>::~shared_ptr() /usr/include/c++/14/bits/shared_ptr.h:175 (ccm+0xacc1c9)
    #6 Pistache::Tcp::Transport::onReady(Pistache::Aio::FdSet const&) /home/taisto/repos/rtt-focus/3pp/pistache/src/common/transport.cc:233 (ccm+0xacc1c9)

  Previous atomic write of size 4 at 0x72440004268c by thread T6 (mutexes: write M2, write M3):
    #0 __gnu_cxx::__exchange_and_add(int volatile*, int) /usr/include/c++/14/ext/atomicity.h:66 (ccm+0x81e35c) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #1 __gnu_cxx::__exchange_and_add_dispatch(int*, int) /usr/include/c++/14/ext/atomicity.h:101 (ccm+0x81e35c)
    #2 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_weak_release() /usr/include/c++/14/bits/shared_ptr_base.h:212 (ccm+0x81e35c)
    #3 std::__weak_count<(__gnu_cxx::_Lock_policy)2>::~__weak_count() /usr/include/c++/14/bits/shared_ptr_base.h:1168 (ccm+0x8c89d1) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #4 std::__weak_ptr<Pistache::Tcp::Peer, (__gnu_cxx::_Lock_policy)2>::~__weak_ptr() /usr/include/c++/14/bits/shared_ptr_base.h:1997 (ccm+0x8c89d1)
    #5 std::weak_ptr<Pistache::Tcp::Peer>::~weak_ptr() /usr/include/c++/14/bits/shared_ptr.h:810 (ccm+0x8c89d1)
    #6 Pistache::Http::ResponseWriter::~ResponseWriter() 3pp/pistache/include/pistache/http.h:433 (ccm+0x8c89d1)
    #7 ImmiHandler::~ImmiHandler() src/ecmi/ImmiHandler.cpp:51 (ccm+0x8e5e74) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #8 void std::destroy_at<ImmiHandler>(ImmiHandler*) /usr/include/c++/14/bits/stl_construct.h:88 (ccm+0x8bf07d) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #9 void std::_Destroy<ImmiHandler>(ImmiHandler*) /usr/include/c++/14/bits/stl_construct.h:149 (ccm+0x8bf07d)
    #10 void std::allocator_traits<std::allocator<void> >::destroy<ImmiHandler>(std::allocator<void>&, ImmiHandler*) /usr/include/c++/14/bits/alloc_traits.h:720 (ccm+0x8bf07d)
    #11 std::_Sp_counted_ptr_inplace<ImmiHandler, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() /usr/include/c++/14/bits/shared_ptr_base.h:616 (ccm+0x8bf07d)
    #12 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use() /usr/include/c++/14/bits/shared_ptr_base.h:175 (ccm+0x6d938f) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #13 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /usr/include/c++/14/bits/shared_ptr_base.h:361 (ccm+0x6d9495) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #14 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/include/c++/14/bits/shared_ptr_base.h:1069 (ccm+0x8bde93) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #15 std::__shared_ptr<ImmiHandler, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/include/c++/14/bits/shared_ptr_base.h:1525 (ccm+0x8bde93)
    #16 std::shared_ptr<ImmiHandler>::~shared_ptr() /usr/include/c++/14/bits/shared_ptr.h:175 (ccm+0x8bde93)
    #17 EcmiApi::immiHandlerFinished(ImmiHandler*) src/ecmi/EcmiApi.cpp:48 (ccm+0x8bde93)
    #18 ImmiHandler::gotImmiRsp(std::shared_ptr<ApiResponse>&) src/ecmi/ImmiHandler.cpp:332 (ccm+0x8e874c) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #19 ImmiApi::responseQueueLoop() src/immi/ImmiApi.cpp:737 (ccm+0x8f6a8d) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #20 void std::__invoke_impl<void, void (ImmiApi::*)(), ImmiApi*>(std::__invoke_memfun_deref, void (ImmiApi::*&&)(), ImmiApi*&&) /usr/include/c++/14/bits/invoke.h:74 (ccm+0x91c3b0) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #21 std::__invoke_result<void (ImmiApi::*)(), ImmiApi*>::type std::__invoke<void (ImmiApi::*)(), ImmiApi*>(void (ImmiApi::*&&)(), ImmiApi*&&) /usr/include/c++/14/bits/invoke.h:96 (ccm+0x91c3f0) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #22 void std::thread::_Invoker<std::tuple<void (ImmiApi::*)(), ImmiApi*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/include/c++/14/bits/std_thread.h:301 (ccm+0x91c3f0)
    #23 std::thread::_Invoker<std::tuple<void (ImmiApi::*)(), ImmiApi*> >::operator()() /usr/include/c++/14/bits/std_thread.h:308 (ccm+0x91c3f0)
    #24 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (ImmiApi::*)(), ImmiApi*> > >::_M_run() /usr/include/c++/14/bits/std_thread.h:253 (ccm+0x91c429) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #25 execute_native_thread_routine ../../../../../libstdc++-v3/src/c++11/thread.cc:104 (libstdc++.so.6+0xed3a3) (BuildId: 0b7ef69c31abcc1170435394579683adb8e54f79)

  Mutex M0 (0x7284000f1038) created at:
    #0 pthread_mutex_lock ../../../../libsanitizer/tsan/tsan_interceptors_posix.cpp:1341 (libtsan.so.2+0x5e4ce) (BuildId: f57d904f7f377f8ad981d755372777f1e90df49e)
    #1 __gthread_mutex_lock /usr/include/c++/14/x86_64-suse-linux/bits/gthr-default.h:762 (ccm+0xb11409) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #2 std::mutex::lock() /usr/include/c++/14/bits/std_mutex.h:113 (ccm+0xb11409)
    #3 std::lock_guard<std::mutex>::lock_guard(std::mutex&) /usr/include/c++/14/bits/std_mutex.h:250 (ccm+0xb11409)
    #4 Pistache::Aio::SyncImpl::addHandler(std::shared_ptr<Pistache::Aio::Handler> const&, bool) /home/taisto/repos/rtt-focus/3pp/pistache/src/common/reactor.cc:167 (ccm+0xb11409)
    #5 Pistache::Aio::AsyncImpl::addHandler(std::shared_ptr<Pistache::Aio::Handler> const&, bool) /home/taisto/repos/rtt-focus/3pp/pistache/src/common/reactor.cc:557 (ccm+0xb11409)
    #6 void std::__invoke_impl<void, void (EcmiServer::*)(), EcmiServer*>(std::__invoke_memfun_deref, void (EcmiServer::*&&)(), EcmiServer*&&) /usr/include/c++/14/bits/invoke.h:74 (ccm+0x8e3618) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #7 std::__invoke_result<void (EcmiServer::*)(), EcmiServer*>::type std::__invoke<void (EcmiServer::*)(), EcmiServer*>(void (EcmiServer::*&&)(), EcmiServer*&&) /usr/include/c++/14/bits/invoke.h:96 (ccm+0x8e3658) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #8 void std::thread::_Invoker<std::tuple<void (EcmiServer::*)(), EcmiServer*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/include/c++/14/bits/std_thread.h:301 (ccm+0x8e3658)
    #9 std::thread::_Invoker<std::tuple<void (EcmiServer::*)(), EcmiServer*> >::operator()() /usr/include/c++/14/bits/std_thread.h:308 (ccm+0x8e3658)
    #10 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (EcmiServer::*)(), EcmiServer*> > >::_M_run() /usr/include/c++/14/bits/std_thread.h:253 (ccm+0x8e3691) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #11 execute_native_thread_routine ../../../../../libstdc++-v3/src/c++11/thread.cc:104 (libstdc++.so.6+0xed3a3) (BuildId: 0b7ef69c31abcc1170435394579683adb8e54f79)

  Mutex M1 (0x7284000f1008) created at:
    #0 pthread_mutex_lock ../../../../libsanitizer/tsan/tsan_interceptors_posix.cpp:1341 (libtsan.so.2+0x5e4ce) (BuildId: f57d904f7f377f8ad981d755372777f1e90df49e)
    #1 __gthread_mutex_lock /usr/include/c++/14/x86_64-suse-linux/bits/gthr-default.h:762 (ccm+0xb1311e) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #2 std::mutex::lock() /usr/include/c++/14/bits/std_mutex.h:113 (ccm+0xb1311e)
    #3 std::lock_guard<std::mutex>::lock_guard(std::mutex&) /usr/include/c++/14/bits/std_mutex.h:250 (ccm+0xb1311e)
    #4 Pistache::Aio::SyncImpl::runOnce() /home/taisto/repos/rtt-focus/3pp/pistache/src/common/reactor.cc:307 (ccm+0xb1311e)

  Mutex M2 (0x000000dad680) created at:
    #0 pthread_mutex_lock ../../../../libsanitizer/tsan/tsan_interceptors_posix.cpp:1341 (libtsan.so.2+0x5e4ce) (BuildId: f57d904f7f377f8ad981d755372777f1e90df49e)
    #1 __gthread_mutex_lock /usr/include/c++/14/x86_64-suse-linux/bits/gthr-default.h:762 (ccm+0x7a07b0) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #2 std::mutex::lock() /usr/include/c++/14/bits/std_mutex.h:113 (ccm+0x7a07b0)
    #3 std::unique_lock<std::mutex>::lock() /usr/include/c++/14/bits/unique_lock.h:147 (ccm+0x7a0f33) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #4 std::unique_lock<std::mutex>::unique_lock(std::mutex&) /usr/include/c++/14/bits/unique_lock.h:73 (ccm+0x8f6347) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #5 ImmiApi::responseQueueLoop() src/immi/ImmiApi.cpp:675 (ccm+0x8f6347)
    #6 void std::__invoke_impl<void, void (ImmiApi::*)(), ImmiApi*>(std::__invoke_memfun_deref, void (ImmiApi::*&&)(), ImmiApi*&&) /usr/include/c++/14/bits/invoke.h:74 (ccm+0x91c3b0) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #7 std::__invoke_result<void (ImmiApi::*)(), ImmiApi*>::type std::__invoke<void (ImmiApi::*)(), ImmiApi*>(void (ImmiApi::*&&)(), ImmiApi*&&) /usr/include/c++/14/bits/invoke.h:96 (ccm+0x91c3f0) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #8 void std::thread::_Invoker<std::tuple<void (ImmiApi::*)(), ImmiApi*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/include/c++/14/bits/std_thread.h:301 (ccm+0x91c3f0)
    #9 std::thread::_Invoker<std::tuple<void (ImmiApi::*)(), ImmiApi*> >::operator()() /usr/include/c++/14/bits/std_thread.h:308 (ccm+0x91c3f0)
    #10 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (ImmiApi::*)(), ImmiApi*> > >::_M_run() /usr/include/c++/14/bits/std_thread.h:253 (ccm+0x91c429) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #11 execute_native_thread_routine ../../../../../libstdc++-v3/src/c++11/thread.cc:104 (libstdc++.so.6+0xed3a3) (BuildId: 0b7ef69c31abcc1170435394579683adb8e54f79)

  Mutex M3 (0x724c00000018) created at:    ==> My own mutex for the storage container keeping track of ImmiHandlers.
    #0 pthread_mutex_lock ../../../../libsanitizer/tsan/tsan_interceptors_posix.cpp:1341 (libtsan.so.2+0x5e4ce) (BuildId: f57d904f7f377f8ad981d755372777f1e90df49e)
    #1 __gthread_mutex_lock /usr/include/c++/14/x86_64-suse-linux/bits/gthr-default.h:762 (ccm+0x7a07b0) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #2 std::mutex::lock() /usr/include/c++/14/bits/std_mutex.h:113 (ccm+0x7a07b0)
    #3 std::lock_guard<std::mutex>::lock_guard(std::mutex&) /usr/include/c++/14/bits/std_mutex.h:250 (ccm+0x8be0ff) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #4 EcmiApi::createImmiHandler(Pistache::Http::ResponseWriter&) src/ecmi/EcmiApi.cpp:32 (ccm+0x8be0ff)
    #5 EcmiConferenceApi::createConference(Pistache::Rest::Request const&, Pistache::Http::ResponseWriter) src/ecmi/EcmiConferenceApi.cpp:75 (ccm+0x8c2fe8) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #6 Pistache::Rest::Routes::bind<void, EcmiConferenceApi, Pistache::Rest::Request const&, Pistache::Http::ResponseWriter, EcmiConferenceApi*>(void (EcmiConferenceApi::*)(Pistache::Rest::Request const&, Pistache::Http::ResponseWriter), EcmiConferenceApi*)::{lambda(Pistache::Rest::Request const&, Pistache::Http::ResponseWriter)#1}::operator()(Pistache::Rest::Request const&, Pistache::Http::ResponseWriter) const 3pp/pistache/include/pistache/router.h:413 (ccm+0x8c8a7d) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #7 Pistache::Rest::Route::Result std::__invoke_impl<Pistache::Rest::Route::Result, Pistache::Rest::Routes::bind<void, EcmiConferenceApi, Pistache::Rest::Request const&, Pistache::Http::ResponseWriter, EcmiConferenceApi*>(void (EcmiConferenceApi::*)(Pistache::Rest::Request const&, Pistache::Http::ResponseWriter), EcmiConferenceApi*)::{lambda(Pistache::Rest::Request const&, Pistache::Http::ResponseWriter)#1}&, Pistache::Rest::Request, Pistache::Http::ResponseWriter>(std::__invoke_other, Pistache::Rest::Routes::bind<void, EcmiConferenceApi, Pistache::Rest::Request const&, Pistache::Http::ResponseWriter, EcmiConferenceApi*>(void (EcmiConferenceApi::*)(Pistache::Rest::Request const&, Pistache::Http::ResponseWriter), EcmiConferenceApi*)::{lambda(Pistache::Rest::Request const&, Pistache::Http::ResponseWriter)#1}&, Pistache::Rest::Request&&, Pistache::Http::ResponseWriter&&) /usr/include/c++/14/bits/invoke.h:61 (ccm+0x8c8b0b) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #8 std::enable_if<is_invocable_r_v<Pistache::Rest::Route::Result, Pistache::Rest::Routes::bind<void, EcmiConferenceApi, Pistache::Rest::Request const&, Pistache::Http::ResponseWriter, EcmiConferenceApi*>(void (EcmiConferenceApi::*)(Pistache::Rest::Request const&, Pistache::Http::ResponseWriter), EcmiConferenceApi*)::{lambda(Pistache::Rest::Request const&, Pistache::Http::ResponseWriter)#1}&, Pistache::Rest::Request, Pistache::Http::ResponseWriter>, Pistache::Rest::Route::Result>::type std::__invoke_r<Pistache::Rest::Route::Result, Pistache::Rest::Routes::bind<void, EcmiConferenceApi, Pistache::Rest::Request const&, Pistache::Http::ResponseWriter, EcmiConferenceApi*>(void (EcmiConferenceApi::*)(Pistache::Rest::Request const&, Pistache::Http::ResponseWriter), EcmiConferenceApi*)::{lambda(Pistache::Rest::Request const&, Pistache::Http::ResponseWriter)#1}&, Pistache::Rest::Request, Pistache::Http::ResponseWriter>(Pistache::Rest::Routes::bind<void, EcmiConferenceApi, Pistache::Rest::Request const&, Pistache::Http::ResponseWriter, EcmiConferenceApi*>(void (EcmiConferenceApi::*)(Pistache::Rest::Request const&, Pistache::Http::ResponseWriter), EcmiConferenceApi*)::{lambda(Pistache::Rest::Request const&, Pistache::Http::ResponseWriter)#1}&, Pistache::Rest::Request&&, Pistache::Http::ResponseWriter&&) /usr/include/c++/14/bits/invoke.h:114 (ccm+0x8c8b86) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #9 std::_Function_handler<Pistache::Rest::Route::Result (Pistache::Rest::Request, Pistache::Http::ResponseWriter), Pistache::Rest::Routes::bind<void, EcmiConferenceApi, Pistache::Rest::Request const&, Pistache::Http::ResponseWriter, EcmiConferenceApi*>(void (EcmiConferenceApi::*)(Pistache::Rest::Request const&, Pistache::Http::ResponseWriter), EcmiConferenceApi*)::{lambda(Pistache::Rest::Request const&, Pistache::Http::ResponseWriter)#1}>::_M_invoke(std::_Any_data const&, Pistache::Rest::Request&&, Pistache::Http::ResponseWriter&&) /usr/include/c++/14/bits/std_function.h:290 (ccm+0x8c8b86)
    #10 std::function<Pistache::Rest::Route::Result (Pistache::Rest::Request, Pistache::Http::ResponseWriter)>::operator()(Pistache::Rest::Request, Pistache::Http::ResponseWriter) const /usr/include/c++/14/bits/std_function.h:591 (ccm+0xae6f34) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #11 void Pistache::Rest::Route::invokeHandler<Pistache::Rest::Request, Pistache::Http::ResponseWriter>(Pistache::Rest::Request&&, Pistache::Http::ResponseWriter&&) const /home/taisto/repos/rtt-focus/3pp/pistache/src/../include/pistache/router.h:99 (ccm+0xae6f34)
    #12 Pistache::Rest::Router::route(Pistache::Http::Request const&, Pistache::Http::ResponseWriter) const /home/taisto/repos/rtt-focus/3pp/pistache/src/server/router.cc:538 (ccm+0xae6f34)

  Thread T10 (tid=83707, running) created by thread T9 at:
    #0 pthread_create ../../../../libsanitizer/tsan/tsan_interceptors_posix.cpp:1022 (libtsan.so.2+0x5f4a6) (BuildId: f57d904f7f377f8ad981d755372777f1e90df49e)
    #1 __gthread_create /usr/src/debug/gcc-14.2.1+git11321/obj-x86_64-suse-linux/x86_64-suse-linux/libstdc++-v3/include/x86_64-suse-linux/bits/gthr-default.h:676 (libstdc++.so.6+0xed478) (BuildId: 0b7ef69c31abcc1170435394579683adb8e54f79)
    #2 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) ../../../../../libstdc++-v3/src/c++11/thread.cc:172 (libstdc++.so.6+0xed478)
    #3 void std::__invoke_impl<void, void (EcmiServer::*)(), EcmiServer*>(std::__invoke_memfun_deref, void (EcmiServer::*&&)(), EcmiServer*&&) /usr/include/c++/14/bits/invoke.h:74 (ccm+0x8e3618) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #4 std::__invoke_result<void (EcmiServer::*)(), EcmiServer*>::type std::__invoke<void (EcmiServer::*)(), EcmiServer*>(void (EcmiServer::*&&)(), EcmiServer*&&) /usr/include/c++/14/bits/invoke.h:96 (ccm+0x8e3658) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #5 void std::thread::_Invoker<std::tuple<void (EcmiServer::*)(), EcmiServer*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/include/c++/14/bits/std_thread.h:301 (ccm+0x8e3658)
    #6 std::thread::_Invoker<std::tuple<void (EcmiServer::*)(), EcmiServer*> >::operator()() /usr/include/c++/14/bits/std_thread.h:308 (ccm+0x8e3658)
    #7 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (EcmiServer::*)(), EcmiServer*> > >::_M_run() /usr/include/c++/14/bits/std_thread.h:253 (ccm+0x8e3691) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #8 execute_native_thread_routine ../../../../../libstdc++-v3/src/c++11/thread.cc:104 (libstdc++.so.6+0xed3a3) (BuildId: 0b7ef69c31abcc1170435394579683adb8e54f79)

  Thread T6 (tid=83703, running) created by main thread at:
    #0 pthread_create ../../../../libsanitizer/tsan/tsan_interceptors_posix.cpp:1022 (libtsan.so.2+0x5f4a6) (BuildId: f57d904f7f377f8ad981d755372777f1e90df49e)
    #1 __gthread_create /usr/src/debug/gcc-14.2.1+git11321/obj-x86_64-suse-linux/x86_64-suse-linux/libstdc++-v3/include/x86_64-suse-linux/bits/gthr-default.h:676 (libstdc++.so.6+0xed478) (BuildId: 0b7ef69c31abcc1170435394579683adb8e54f79)
    #2 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) ../../../../../libstdc++-v3/src/c++11/thread.cc:172 (libstdc++.so.6+0xed478)
    #3 std::thread std::jthread::_S_create<void (ImmiApi::*)(), ImmiApi*>(std::stop_source&, void (ImmiApi::*&&)(), ImmiApi*&&) /usr/include/c++/14/thread:275 (ccm+0x908d08) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #4 std::jthread::jthread<void (ImmiApi::*)(), ImmiApi*, void>(void (ImmiApi::*&&)(), ImmiApi*&&) /usr/include/c++/14/thread:163 (ccm+0x908d08)
    #5 ImmiApi::init(AddressInfo const&, AddressInfo const&, AddressInfo const&) src/immi/ImmiApi.cpp:165 (ccm+0x8f40de) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)
    #6 main src/main.cpp:68 (ccm+0x6d553b) (BuildId: 4d3d1451aaca0879b8bc7470ebf6ce10db906d3c)

SUMMARY: ThreadSanitizer: data race ../../../../libsanitizer/tsan/tsan_new_delete.cpp:150 in operator delete(void*, unsigned long)

[dgreatwood]

Hi @TheQue42,

Thanks for reporting this issue. I don't have an immediate answer, but let me note a couple of thoughts.

First glance, it looks quite odd.

For thread T10, the place where the problem is identified in transport.cc:233 looks like this:

                if (isPeerFd(tag))
                {
                    auto peer = getPeer(tag);
                    PS_LOG_DEBUG("handleIncoming");
                    handleIncoming(peer);
                }

The problem happens in the destructor of the shared pointer peer, a std::shared_ptr<Peer> (the problem is in the destructor of the pointer, not in the destruction of the object to which peer points, though it's possible that object is also being destroyed just as the pointer peer goes out of scope).

Per ThreadSanitizer, the second 32-bit-word of the same location being written to by the shared pointer destructor was also written to in thread T6 by the weak pointer destructor for std::weak_ptr<Peer>, invoked when the ResponseWriter destructor (http.h:433) of the object containing that weak pointer is called.

We can perhaps guess that the Peer pointed to is the same for the shared pointer and the weak pointer.

But the strangeness here is that I would expect it to be perfectly OK for a weak pointer to an object to go out of scope asynchronously with a shared pointer. Surely that's one design criteria for weak pointers, unless I'm missing something.

The fact that thread T10 is doing a 64-bit write, while T6 is doing a 32-bit atomic write, might be indicative of a larger memory corruption / heap corruption issue, since the writes seem mismatched.

transport.cc/.h does pass references to std::shared_ptr<Peer> (i.e. std::shared_ptr<Peer>&) in a few places, which could be a source of bugs if the referenced shared pointer were to go out of scope before it is finished with or before an additional shared reference is created. We could try replacing the std::shared_ptr<Peer>& with std::shared_ptr<Peer> throughout transport.cc and .h. But looking through it, I don't see a place where that would be an issue in practice.

Experiments to try:

1. Add an explicit destructor to ResponseWriter in http.h, like ~ResponseWriter() { peer_.reset(); } and see if that makes any difference (it shouldn't, but it would be interesting if it did).
2. Replace all the std::shared_ptr<Peer>& parameters in transport.h and .cc with std::shared_ptr<Peer>.

If it practical for you to make each of those changes, recompile Pistache, and then retry your scenario?

[TheQue42]

I just did a quick try of your suggestions, and doesnt look like it made any difference!

[TheQue42]

If there is any other simple modification you want me to try, let me know. I am not very experienced when it comes to interpreting these backtraces, so I wont get far on my own I think.

Do we have tests in the repo, that does anything similar as my use-case? That is, cloning the response-writers and answering in a different thread?

If I can, I'll try to see if there is any special use case where this happens.

[dgreatwood]

@TheQue42 -

Thanks for trying those experiments.

I don't see a Pistache test that uses ResponseWriter::clone, though ResponseWriter::clone is used internally by Pistache in Router::route (router.cc), which is likely exercised by router_test.

If you could create a test case (e.g. add a case to http_server_test.cc or router_test.cc) that demonstrated the issue, that would be super helpful and I'd be happy to dig into it further.