YMQ -> python exception handling

[sharpener6]

@gxuu, @magniloquency, please put your ideas here and pros and cons here

[magniloquency]

From my discussion with @gxuu regarding how to pass back asynchronous unrecoverable errors from the core.

The idea is to have a global callback. This is pseudocode.

I will post the definition of Exception shortly.

// in c++ core
static void (*function_hook)(Exception ex) = nullptr;

extern void global_error_hook(Exception ex) {
    if (function_hook)
        function_hook();
    else {
        // If no hook is set, we can log the error or handle it in a default way
        std::cerr << "No error hook set. Exception: " << ex.what() << std::endl;
        std::abort(); // Terminate the program
    }
}

// python interface
#include "core.h" // import c++ core

void global_error_hook(Exception ex) {
    acquire_gil();

    python_fatal_exception("An error occurred in the C++ core: " + ex.what());

    release_gil();
}

void module_init() {
    // Set the global error hook to the Python interface
    function_hook = global_error_hook;
}

// c++ interface
void global_error_hook(Exception ex) {
    std::cerr << "Fatal error in C++ core: " << ex.what() << std::endl;

    std::abort(); // This will terminate the program
}

void main() {
    function_hook = global_error_hook; // Set the C++ core error hook
}

[gxuu]

I think there are some repeats. Please remove repeated code, and use different name for function that you will assign to function_hook (at the top of your file).

But if I understand it correctly, this is what I would want.

[magniloquency]

Errors and Exceptions in YMQ

Here I will try to summarize my thoughts and ideas that have arisen from discussions with @gxuu regarding error and exception handling in YMQ.

Two Kinds of Errors

I feel that it's important to distinguish between two kinds of errors: Errors arising from the system (e.g. errno), and those coming from our own invariants which I will call "logical" errors.

The first class of errors should be quite clear, you can imagine for example that the user (Python code) is sending a message, but write() fails due to one of a myriad of reasons, some of these errors we can handle in-place, things like EAGAIN, but others need to be captured by us and communicated back to the user.

The logical class of errors are those that we create ourselves, this could include for example things like trying to create two sockets with the same identity, trying to create a socket with an empty identity (debatable on whether or not we should allow this), trying to bind a sub socket, etc.

Thus, we will need a way to represent both kinds of errors in our Exception data type.

Asynchronous Errors

Many operations in the core are the result of a direct request from the user, for example Python code trying to send a message directly causes write() to be called, which can throw an error. In this case, the user provides a callback with the operation and we can use that to send the exception back to them. No problems here.

However errors can arise in other contexts. The core does some work behind the scenes that are not at the direct request of a user, for example what should the core do if a socket's connection suddenly drops? Well, we know this, the core should try to reconnect it 'transparently', i.e. without any intervention from the user. In this case we would 'swallow' the error, we handle it internally and the user need not do anything.

However, not all errors can be handled within the core, for example EIO. This error code can mean many things, but ultimately it is not something we can handle in the core and should be considered fatal.

We need to communicate this back to the user somehow, but because this operation was not directly requested by the user, we have no callback. I thus propose a "global error hook", a global function that can be called with the exception and is defined by the user of the library. In the case of Python, we can acquire the GIL, report the error, then gracefully shut down the interpreter.

For pseudocode, see the previous comment.

The Exception Type

I propose the following exception type. Please note that this is subject to change, names, etc. are NOT final.

enum Syscall {
    Read,
    Write,
    Accept,
    Bind,
    // etc.
};

enum Logical: int {
    DuplicateIdentity = -1,
    // etc.
};

struct Exception {
    int errno; // > 0 means errno, < 0 means `Logical`
    Syscall syscall;
};

void demo() {
    auto exc1 = Exception { .errno = (int)Logical::DuplicateIdentity };
    auto exc2 = Exception { .errno = EADDRINUSE, .syscall = Syscall::Bind };
}

Without Type Punning

This makes use of type punning to store the errno and logical error cases within the same variable. The Logical enum has negative integer values, and so that can be used to determine the case. An alternative is to use a tagged union:

// alternative

struct Exception {
    enum { System, Logical } tag;
    union {
        Logical logical;
        struct { int errno; Syscall syscall };
    };
};

This can be used to achieve the same thing, but is more verbose.

I am equally ok with both of these options, please let me know if you have a preference: @sharpener6, @gxuu.

Representation in Python

I will have to create an exception class in Python to match what we do in C++. The translation here should be quite straightforward, but I'll show it to you before I implement it.

[gxuu]

Let us store a string to the error type. This string will contain some logging information. Rationale is that this might be helpful to the user. For example when we have duplicate identity (not allowed), we would want to point out what the user passed in; when we have address in use, we want to point out what address the user is trying to bind to.

This error string should be something like std::string. Please tell me if there's any difficulty to handle that in the Python interface.

UPD: Let us not store a string to the error type. Let's pass this string directly to user callback.

[gxuu]

I'm with the type punning solution.

[sharpener6]

@magniloquency,

this is way way too complicated for just a connection layer, you are trying to build rocket can never launched again. please just implement following my design, I am not opened to talk any designs that trying to structure error types into C++ class structures, you need flatten all error types as one plain enum.

@magniloquency, @gxuu
This is design decision for the first implementation of logging module:

• Anything happened in ymq, should only have one exception type YMQError, EAGIN should be one of the customized error type, see below how it get showed
• each exception should contain a string message, which should include enough message for user to debug, no matter it's user error or your bug (it should not have subtle bug in ymq)
• the string message format can be composed as
  "<iso timestamp>: <identity>|<socket_id>|<customized error type>: <detail message>"
  if need more detail, add to this format, but I don't think it need more
• there is not log level, no info, anything, exception meant to abort and give correct message to user why it's aborting

[gxuu]

Just to clarify: You want errno to be translated to custom error types that contain the generator? So we would have something like

READ_EBADF,
WRITE_EBADF,
READ_EFAULT,
WRITE_EFAULT,

etc.

[rafa-be]

I think we are trying to solve 2 problems with 1 solution:

1. unexpected errors that the library cannot recover from;
2. regular exceptions that should be propagated to the user.

For 1), I'd suggest we log the error (see @sharpener6 's format) and call std::terminate():

void unrecoverrableError(Args&&... args) {
    static constexpr auto formatStr = getFormatString<sizeof...(Args)>();

    std::cerr << std::format(std::string_view {formatStr}, std::forward<Args>(args)...)
              << std::endl;

    std::terminate(1);
}


int main() {
    if (error) {
        unrecoverrableError("Error while during syscall (errno=%d)", errno);
    }
}

For 2), I'd make a specific YMError class that uses the same interface as std::exception, for example:

class YMQError : public std::exception {
public:
    enum class ErrorType {
        ConnectionReset,
        WriteError,
        ...
    };

    YMQError(ErrorType errorType, Args&&... args) {
        this->errorType = errorType;
        
        static constexpr auto formatStr = getFormatString<sizeof...(Args)>();
    
        message = std::move(std::format(std::string_view {formatStr}, std::forward<Args>(args)...));
    }

    const char* what() const noexcept override {
        return message.c_str();
    }

    ErrorType getErrorType() const {
        return errorType;
    }

private:
    ErrorType errorType;
    std::string message;
};


int main() {
    YMQError(ErrorType::ConnectionReset, "Pair reset from %s", "127.0.0.1");
}

This error could be propagated back to users through the callbacks, and combined with std::expected.

[gxuu]

Reference implementation of error and its utility can be found here.

All std::cerr or fprintf or printf will be changed to calls to unrecoverableError. See above link.

@sharpener6

[gxuu]

1. You are essentially saying all inputs are generated by the user and therefore they know whether their input arguments are correct. This is false. For example the sockaddr remoteAddr is not generated by the user, and they might care; for example the actual int fd is generated by the library and the user might care (typical scenario is that if you run strace you only see fd as a constant).
2. See 1.
3. Why should we always try to send the error back to the user if possible? Imagine that you are an user and you see EIO, what can you do? If the underlying library is reporting you an underlying fd encounters EBADF, what can you do? In fact, the extreme example EIO is exactly why we should abort ASAP. It means file system corruption and keep the program running might corrupt the system even more. Library handle-able errors should be handled in the library. Library not handle-able but user handle-able errors should be give back to the user (for example ADDRINUSE on bind), library not handle-able AND user not-handle-able error should just abort (for example EIO).

[magniloquency]

You are essentially saying all inputs are generated by the user and therefore they know whether their input arguments are correct.

This is not what I am saying. I am saying that the caller knows the values they are passing into the library, and therefore do not need to receive them back in the exception. However you make a good point about data that originates within the library, like FDs and remote addresses. For debugability it would be nice to have those logged somewhere, but I'm also thinking that it's probably leaking the abstraction a lot to send them into user code? 🤔 Maybe @sharpener6 has a thought on it

2. .

Imagine that you are an user and you see EIO, what can you do

Most likely just exit the program, the benefit of returning the exception through the callback is that the user will get a sensible location in the program to start debugging or looking into things. The global error hook should imo be our last resort, when we can't do anything else, because then the user has no way to handle it, and it has no traceability in their code.

[gxuu]

1. Leaking abstraction is not a problem in this case. I honestly have never understood what that combination of words means. But, when unrecoverable errors happens all abstraction should be break for investigation. It's like doing autopsy where you need to open someone up, even though it "leaks abstractions" (and of course many things like intestine and liver etc.)
2. That's the exact point! We need the source location ideally to be carried out. That's exactly why we need a string that's created and passed to user callback. You would have something like `UnrecoverableError({.errno = SomeFancyError}, PRETTY_FUNCTION, LINE);

the benefit of returning the exception through the callback

You still get an Exception. You can have one Exception and one string. As a matter of fact, if you write the Python hook, you can choose to ignore the string passed in completely, and only care about the exception struct that's passed in.

Please think about how you would pass the line number to the global function, if you don't actively passed in this __prettyfunction__ thing.

[magniloquency]

1. For maintainability, it's common to modularize code and keep concerns separate. I have been thinking of YMQ as a standalone networking library within our repository. To that end, it provides an abstraction over networking and I believe should keep its details internal to itself. The library's user knows nothing about line numbers, and nothing about fds, therefore providing these things in our exceptions leaks details and presents information that the user cannot act on.
   However, if we choose to see YMQ as a part of scaler, and the abstraction is not so important, then I guess we can include things like fds and line numbers.. This is definitely possible, though I prefer the former approach.

2. The problem is that the global hook is completely detached from any source location within the user's code. When a function returns an error (propagated via a callback) the user knows where that error came from. The global error hook is detached from any callsite and is therefore I believe more difficult to figure out.

[gxuu]

1. I have actually been thinking about YMQ in the former way. Imagine what you would do with this line number if YMQ is some external library pulled from Github. What I would do is to paste the error to the owner of the project, and tell them to fix it. You would do exactly the same thing to me when weird error occurs from YMQ: Give me the line number and tell me to fix it. Now on the other hand, if YMQ hides everything, and most importantly hides what happens and where it happens, then what can I do when user reports error like "something went wrong in your lib, no useful information I can provide"?
2. As I have said many times, when you have a callback that is reasonable to use on, the YMQ core will always use it. If you really think that setNoDelay should use callbacks then I am fine with it. It's just an opinion instead of a fact that whether setNoDelay errors are considered reasonable to be fed into the user callback.

[rafa-be]

We just had a call with @gxuu, here is what we came up with for error handling.

A base YMQError exception class:

class YMQError : public std::exception {
public:
    enum class ErrorType {
        ConnectionReset,
        WriteError,
        ...
    };

    YMQError(ErrorType errorType, Args&&... args);

    const char* what() const noexcept override;

    ErrorType getErrorType() const;
    
private:
    ErrorType errorType;
    std::string message;
};

The YMQError object provides a string formatting constructor:

YMQError(ErrorType.WriteError, "Unable to write to socket, errno=%d", errno)

Most of the time, YMQ will raise this error object back to the user through std::expected callbacks:

class IOSocket {
    ...
    void recvMessage(std::function<std::expected<Message, YMQError>> onRecvMessage) noexcept;
    ...
}

In addition, there are cases where YMQ cannot recover. There are unexpected errors that the library cannot recover from and should terminate.

In these cases, the library will call a unrecoverableError(YMQError error) function:

Message* msg = new Message();
 
if (msg == nullptr) {
    unrecoverableError({ErrorType.MemoryError, "Unable to allocate memory"});
}

We will allow the user to set a specific handler for these errors. The reason is that the user might want to do some cleanup operations:

void scalerUnrecoverableErrorHandler(YMQError error) {
    // Do some Python related cleanup
    ...
    Py_Finalize();
}

setUnrecoverableErrorHandler(scalerUnrecoverableErrorHandler);

The default error handler would just print the error message and exit the process.

Please note that this does not address logging yet.

[gxuu]

@rafa-be, @magniloquency,

Reference implementation to the aforementioned call can be found at scaler/io/ymq/error.h.

Reference usage to the implementation can be found at scaler/io/ymq/utils.h.
Reference output

2025-07-11 06:58:32+0800: An error generated by system call that's likely due to mis-configuration: Originated from: std::expected<sockaddr, int> stringToSockaddr(const std::string&): Errno is: Success: out_addr.sin_family: 2: out_addr.sin_port: 36895

Note that,

• the Error type is now inherited from std::exception and we provide what() for easier use.
• an ErrorCode and a string is stored so the hook user can choose to give his own explanation if he wants.
• We get rid of errno handling and store it inside the string, as user don't care about that that much.

To address logging, we might want to change Error to Status and replace Uninit with Success where its explanation is simply empty string, and also provide log(level, status). This way, we support logging without another structure.

[rafa-be]

Nice work

However, I'm wondering if we need these _errorToExplanation messages, as these seem redundant with _logMsg.

Personally, I'd just print the ErrorType value, e.g:

2025-07-11 06:58:32+0800: ConfigurationError originated from: std::expected<sockaddr, int> stringToSockaddr(const std::string&): Errno is: Success: out_addr.sin_family: 2: out_addr.sin_port: 36895

[gxuu]

Implementation to the Logging can be found in logging.h. Reference output is [DEBUG]: 2025-07-15 05:07:54+0800: Client retrying times: 1