aio: message not associated on FreeBSD

[interkosmos]

Describe the bug
The setup consists of a server program and multiple clients that connect through TCP transport using the request/reply protocol (reqrep0). Each client (synchronously) sends a single request message to the server. The server spins up a number of asynchronous workers to respond to the requests. The program in which the bug is observed is written in Fortran and uses interface bindings to libnng.

On FreeBSD 15, the function nng_aio_get_msg() sometimes returns a NULL pointer if multiple messages are received in parallel. On Linux (Debian 12), the message pointer is always associated instead (as expected). Additionally, it is required to call nng_sleep_aio() with an arbitrary delay after nng_aio_result() on FreeBSD or no message will be received by the server and the clients run into a timeout.

Expected behavior
The function nng_aio_get_msg() should return the received message.

Actual Behavior
The function nng_aio_get_msg() may return NULL on FreeBSD.

To Reproduce
The Fortran code in which the bug occurs is based on the structure of the server_cb callback procedure from example demo/async/server.c:

void server_cb(void *arg)
{
    struct work *work = arg;
    nng_msg *    msg;
    int          rv;
    uint32_t     when;

    switch (work->state) {
    case INIT:
        work->state = RECV;
        nng_ctx_recv(work->ctx, work->aio);
        break;
    case RECV:
        if ((rv = nng_aio_result(work->aio)) != 0) {
            fatal("nng_ctx_recv", rv);
        }
        nng_sleep_aio(0, work->aio);      /* <-- somehow necessary on FreeBSD */
        msg = nng_aio_get_msg(work->aio); /* <-- the returned pointer is NULL */
        if ((rv = nng_msg_trim_u32(msg, &when)) != 0) {
            nng_msg_free(msg);
            nng_ctx_recv(work->ctx, work->aio);
            return;
        }
        work->msg   = msg;
        work->state = WAIT;
        nng_sleep_aio(when, work->aio);
        break;
    case WAIT:
        nng_aio_set_msg(work->aio, work->msg);
        work->msg   = NULL;
        work->state = SEND;
        nng_ctx_send(work->ctx, work->aio);
        break;
    case SEND:
        if ((rv = nng_aio_result(work->aio)) != 0) {
            nng_msg_free(work->msg);
            fatal("nng_ctx_send", rv);
        }
        work->state = RECV;
        nng_ctx_recv(work->ctx, work->aio);
        break;
    default:
        fatal("bad state!", NNG_ESTATE);
        break;
    }
} 

On FreeBSD, the message pointer returned by nng_aio_get_msg() may be NULL if multiple clients send messages to the server in parallel. When tested with 10 clients, either all messages are received successfully or approx. 1/3 to 1/2 are lost. The bug occurs independently of the number of parallel aio workers started by the server (tested with 10 and 100 workers). The bug is present both with GCC (gcc, gfortran) and LLVM (clang, flang).

Environment Details

• NNG v1.11.0
• FreeBSD 15 (amd64)
• GCC 15, LLVM 21
• libnng.so

[interkosmos]

I’ve tested it again with NNG v2.0.0 alpha 7, and the bug can be observed as well. Futhermore, it is still necessary to call nng_sleep_aio() after nng_aio_result() on FreeBSD.

Edit: With NNG v2, stopping the server also causes a panic:

panic: /tmp/nng-2.0.0-alpha.7/src/core/aio.c: 383: assert err: !aio->a_stopped
This message is indicative of a BUG.

[gdamore]

I would like to see the rest of the program that reproduces this. The callbacks alone look sane to me, but I suspect that we have dueling callbacks somehow.

[gdamore]

Its really really strange that this is FreeBSD specific. The code where this stuff gets used should not be platform specific.

One thing you could try to is to use the poll() based implmentation instead of kqueue.

[gdamore]

One though ... could the CPU have an incorrectly set of cached content for the work? With I'm wondering if this is a problem where we just need to use some locks to protect the work structure -- while it shouldn't be getting accessed precisely at the same time (no true concurrency), it does seem possible for values to be cached by a processor core.

Can you try adding locks on your checks for the work structures? (To be clear I'd consider this a bug in the example, and in yur code, rather than a bug in NNG, although it seems like maybe NNG could do something here to help, but I'm not 100% sure how. Lets find out if adding locks resolves.

[interkosmos]

I’ve added a mutex to lock the client data of the thread callback but the described behaviour persists. I made sure to use compiler flag -frecursive so that each thread has its own function stack.

After further investigation it seems that the requests fail on client-side because the pointer returned by nng_msg_body() is NULL after nng_recvmsg() returned without errors. On server-side, there is no condition in which an empty message could be returned. It is also weird that sometimes the clients fail to receive a response recurrently, as the log messages of the test program show:

2026-02-09T19:51:07.596370+01:00 [DEBUG] received response after 1 attempts in 2.351309000000E-3 sec
2026-02-09T19:51:07.597450+01:00 [DEBUG] received response after 1 attempts in 3.091868000000E-3 sec
2026-02-09T19:51:07.599308+01:00 [DEBUG] received response after 1 attempts in 2.565180000000E-3 sec
2026-02-09T19:51:07.603371+01:00 [DEBUG] received response after 1 attempts in 5.380549000000E-3 sec
2026-02-09T19:51:07.604276+01:00 [DEBUG] received response after 8 attempts in 1.310239500000E-2 sec
2026-02-09T19:51:07.633276+01:00 [ERROR] request failed after 10 attempts
2026-02-09T19:51:07.640240+01:00 [ERROR] request failed after 10 attempts
2026-02-09T19:51:07.642233+01:00 [ERROR] request failed after 10 attempts
2026-02-09T19:51:07.643259+01:00 [ERROR] request failed after 10 attempts
2026-02-09T19:51:07.646228+01:00 [ERROR] request failed after 10 attempts

Here, 5 out of 10 parallel clients receive a NULL message even after 10 attempts, one client needed 8 attempts until the message body was present. When running only one (non-parallel) client, no error occurs and the transmission always succeeds.

I’ll try to write an MRE in C in the coming days.

Edit: Maybe, the behaviour of the whole setup changed after updating from nng v1 to v2 because it looks like nng_aio_get_msg() now returns an associated message pointer on server-side.