Skip to content

Segfault when restarting RMQ broker with MPI #125

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
lpottier wants to merge 14 commits into from
Closed

Segfault when restarting RMQ broker with MPI #125

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
ba7ed87
Trying get some more accurate debug log from the segfault
lpottier Mar 31, 2025
7b9fce6
Add better debugging to the AMS DB benchmark
lpottier Mar 31, 2025
6c03a05
Removed Umpire from the DB benchmark
lpottier Mar 31, 2025
b7963a7
Temporary debug code that crashes the broker after 3 messages
lpottier Mar 31, 2025
e379f10
Added various print to debug the segfault (WIP)
lpottier Apr 1, 2025
050b749
A concept of redesigning the AMS RMQ protocol
koparasy Apr 1, 2025
e9f8a52
Add TSAN flags and comment them out, can be useful
koparasy Apr 1, 2025
0b2e810
Fixed incorrect type of lock + add more time to connect to the broker
lpottier Apr 1, 2025
69d0e2e
Added the singleton hashmap we used in our current implementation
lpottier Apr 2, 2025
ed9525f
Removed some artifacts from debug/test
lpottier Apr 2, 2025
57d3166
Pushed an MPI version (tested with 112 ranks, no segfault). I had to …
lpottier Apr 2, 2025
787500f
Fixed comments
lpottier Apr 2, 2025
bf3d780
use locks instead of shared_locks, add _flushEvent, simplified the lo…
koparasy Apr 2, 2025
c87505e
Added queue creation and channel error management
lpottier Apr 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 18 additions & 2 deletions src/AMSlib/wf/basedb.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -959,10 +959,10 @@ class AMSMessage
*/
class AMSMessageRecords
{
private:
using record_t = std::pair<std::shared_ptr<uint8_t>, size_t>;
using iterator_t = std::unordered_map<int, record_t>::iterator;

private:
/** @brief Internal data structure that keeps messages nack */
std::unordered_map<int, record_t> _msgs;
/** @brief Shared mutex to ensure thread-safe access */
Expand All @@ -977,6 +977,22 @@ class AMSMessageRecords
AMSMessageRecords(AMSMessageRecords&&) = delete;
AMSMessageRecords& operator=(AMSMessageRecords&&) = delete;

/**
* @brief Custom destructor for our shared_ptr
*/
struct AMSMessageDeleter
{
void operator()(void* x) {
DBG(AMSMessageDeleter, "Freeing %p", x)
free(x);
x = nullptr;
}
};

static AMSMessageDeleter getDeleter() {
return AMSMessageDeleter();
}

/**
* @brief Return an iterator at the beggining of the records
*/
Expand Down Expand Up @@ -1715,7 +1731,7 @@ class RMQInterface

if (!_publisher->connectionValid()) restartPublisher();

std::shared_ptr<uint8_t> ptr(msg.data());
std::shared_ptr<uint8_t> ptr(msg.data(), AMSMessageRecords::getDeleter());
auto record = std::make_pair(std::move(ptr), msg.size());

// if we have some messages to send first (from a potential restart)
Expand Down