[DO NOT MERGE] single threaded nats client POC by levb · Pull Request #769 · nats-io/nats.c

levb · 2024-06-24T15:47:24Z

PR for discussion only, not to be merged.

uses libevent for all read/write, still uses natsSock_ConnectTcp to connect
heap implementation abstracted, and the use consolidated
can connect, initial ping, responds to server pings, sends messages
no reconnect, TLS, auth, subs, requests, etc.
writes are not optimized at all, yet. Turn on DEV_MODE_CONN_TRACE to see the trace.

Example: https://github.com/nats-io/nats.c/blob/mininats/examples/libevent-pub.c
It connects and sends 5 messages to a server. The entire heap usage is logged (there are still leaks, please ignore, cleanup is not fully implemented).

To see more details, turn on more logging here

➜  build git:(mininats) ./bin/nats-libevent-pub
TRACE:   HEAP: allocated bytes:4096, ptr:0x153008800 (nats_log_createPool:mem_pool.c:600)
Hello, mini-nats! Will try to send a message to 'foo'
TRACE:   HEAP: allocated bytes:4096, ptr:0x15300b800 (nats_log_createPool:mem_pool.c:600)
TRACE:   HEAP: allocated bytes:4096, ptr:0x15300d800 (nats_log_createPool:mem_pool.c:600)
TRACE:   HEAP: allocated bytes:4096, ptr:0x154008200 (nats_log_createPool:mem_pool.c:600)
TRACE:   HEAP: allocated bytes:65536, ptr:0x158008000 (natsPool_getReadBuffer:mem_pool.c:216)
TRACE:   HEAP: allocated bytes:4096, ptr:0x154009200 (_allocSmall:mem_pool.c:135)
TRACE:   HEAP: allocated bytes:4096, ptr:0x154808200 (nats_log_createPool:mem_pool.c:600)
Message enqueued, text: Hello, NATS!
TRACE:   HEAP: allocated bytes:4096, ptr:0x154809200 (nats_log_createPool:mem_pool.c:600)
Message enqueued as a copy, freeing the data now: Hello, NATS! I was allocated on the heap, my bytes were copied into another buffer, and I was freed immediately by the app.
TRACE:   HEAP: allocated bytes:4096, ptr:0x15480a200 (nats_log_createPool:mem_pool.c:600)
Message enqueued, text: Hello, NATS! I was allocated on the heap, my bytes were NOT copied, and I was freed by a cusom 'message published' callback
TRACE:   HEAP: allocated bytes:4096, ptr:0x15480b200 (nats_log_createPool:mem_pool.c:600)
Message enqueued, text: Hello, NATS! I was allocated on the heap, my bytes were NOT copied, and I was 'free'-ed by the cleanup, no custom callback needed.
TRACE:   HEAP: allocated bytes:4096, ptr:0x15480c200 (nats_log_createPool:mem_pool.c:600)
FINAL message enqueued, text: Hello, NATS! I was allocated on the heap, my bytes were NOT copied, and I was 'free'-ed by the cleanup, no custom callback needed.
TRACE:   HEAP: freeing ptr:0x154808200 (_free:mem_pool.c:84)
TRACE:   HEAP: freeing ptr:0x154809200 (_free:mem_pool.c:84)
Message published, freeing the data: Hello, NATS! I was allocated on the heap, my bytes were NOT copied, and I was freed by a cusom 'message published' callback
TRACE:   HEAP: freeing ptr:0x15480a200 (_free:mem_pool.c:84)
TRACE:   HEAP: freeing ptr:0x15480b200 (_free:mem_pool.c:84)
Last message published, now disconnecting:
Connection closed, last error: ''
TRACE:   HEAP: freeing ptr:0x154008200 (_free:mem_pool.c:84)
TRACE:   HEAP: freeing ptr:0x154009200 (_free:mem_pool.c:84)
TRACE:   HEAP: freeing ptr:0x15300d800 (_free:mem_pool.c:84)
TRACE:   HEAP: freeing ptr:0x15300b800 (_free:mem_pool.c:84)
TRACE:   HEAP: freeing ptr:0x153008800 (_free:mem_pool.c:84)

server log:

[70659] 2024/06/24 08:52:14.781129 [TRC] [::1]:59103 - cid:85 - <<- [CONNECT {"verbose":false,"pedantic":false,"tls_required":false,"name":"","lang":"C","version":"4.0.0-mininats","protocol":1,"echo":true,"headers":true,"no_responders":true}]
[70659] 2024/06/24 08:52:14.781167 [TRC] [::1]:59103 - cid:85 - <<- [PING]
[70659] 2024/06/24 08:52:14.781169 [TRC] [::1]:59103 - cid:85 - ->> [PONG]
[70659] 2024/06/24 08:52:14.781292 [TRC] [::1]:59103 - cid:85 - <<- [PUB foo 12]
[70659] 2024/06/24 08:52:14.781296 [TRC] [::1]:59103 - cid:85 - <<- MSG_PAYLOAD: ["Hello, NATS!"]
[70659] 2024/06/24 08:52:14.781301 [TRC] [::1]:59103 - cid:85 - <<- [PUB foo 123]
[70659] 2024/06/24 08:52:14.781351 [TRC] [::1]:59103 - cid:85 - <<- MSG_PAYLOAD: ["Hello, NATS! I was allocated on the heap, my bytes were NOT copied, and I was freed by a cusom 'message published' callback"]
[70659] 2024/06/24 08:52:14.781368 [TRC] [::1]:59103 - cid:85 - <<- [PUB foo 123]
[70659] 2024/06/24 08:52:14.781382 [TRC] [::1]:59103 - cid:85 - <<- MSG_PAYLOAD: ["Hello, NATS! I was allocated on the heap, my bytes were NOT copied, and I was freed by a cusom 'message published' callback"]
[70659] 2024/06/24 08:52:14.781387 [TRC] [::1]:59103 - cid:85 - <<- [PUB foo 130]
[70659] 2024/06/24 08:52:14.781391 [TRC] [::1]:59103 - cid:85 - <<- MSG_PAYLOAD: ["Hello, NATS! I was allocated on the heap, my bytes were NOT copied, and I was 'free'-ed by the cleanup, no custom callback needed."]
[70659] 2024/06/24 08:52:14.781398 [TRC] [::1]:59103 - cid:85 - <<- [PUB foo 0]
[70659] 2024/06/24 08:52:14.781402 [TRC] [::1]:59103 - cid:85 - <<- MSG_PAYLOAD: [""]
[70659] 2024/06/24 08:52:14.781409 [DBG] [::1]:59103 - cid:85 - Client connection closed: Read Error

kozlovic

Too much to review. This is obviously very early and more of a POC, but you should still try to run a little benchmark for the publish to see how it will perform. I know that early in the C client development the publish perf was really bad compared to the Go client because each publish ended-up result in the tcp write (because notifying the flusher would do the flush right away). He we copy/move the date to a chain but how often the chain will be flushed/processed will dictate the publish performance.

kozlovic · 2024-06-28T19:51:29Z

src/conn.c


+    // Initialize the connect-time memory pool.
+    s = nats_createPool(&nc->connectPool, &nc->opts->mem, "conn-connect");
+    if (s != NATS_OK)
        return NATS_UPDATE_ERR_STACK(s);


Should the state set above remain the same or be updated on failure? Or maybe set the state after this possible men failure.

kozlovic · 2024-06-28T19:52:14Z

src/conn.c

+    s = natsSock_ConnectTcp(&(nc->sockCtx), nc->connectPool, nc->cur->url->host, nc->cur->url->port);
+    if (STILL_OK(s))
+        nc->sockCtx.fdActive = true;
+    if (STILL_OK(s))


Not needed, falls under the above STILL_OK.

kozlovic · 2024-06-28T19:54:01Z

src/conn.c

-
-        return NATS_UPDATE_ERR_STACK(s);
-    }
+    natsConn_retain(nc);


Comment says that locking is done inside the function itself, but I don't see it happening (setting "nc->state", etc..).

levb · 2024-06-28T20:46:33Z

@kozlovic ACK that it is way too early to review, and I am focused on other things for the time being anyway. I marked you to get your "FYI"-level attention, feel free to un-assign if this is too noisy.

...each publish ended-up result in the tcp write

the plan was to optimize it to use writev, like in https://github.com/nginx/nginx/blob/e734df6664e70f118ca3140bcef6d4f1750fa8fa/src/os/unix/ngx_writev_chain.c#L14 but I didn't get to prototype it. So everything would still be buffered (into a chain), and then segments written out of the chain, 1 iovec at a time per write event. I just didn't get to the sockets at all, figured I'd wrap it on top of the existing code for now. Ditto establishing connections - that too would need to be event driven for re-connections.

POC

ef85e92

levb added the proposal Enhancement idea or proposal label Jun 24, 2024

levb requested a review from kozlovic June 24, 2024 15:47

kozlovic reviewed Jun 28, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DO NOT MERGE] single threaded nats client POC#769

[DO NOT MERGE] single threaded nats client POC#769
levb wants to merge 1 commit intomainfrom
mininats

levb commented Jun 24, 2024 •

edited

Loading

Uh oh!

kozlovic left a comment

Uh oh!

kozlovic Jun 28, 2024

Uh oh!

kozlovic Jun 28, 2024

Uh oh!

kozlovic Jun 28, 2024

Uh oh!

levb commented Jun 28, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

levb commented Jun 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kozlovic left a comment

Choose a reason for hiding this comment

Uh oh!

kozlovic Jun 28, 2024

Choose a reason for hiding this comment

Uh oh!

kozlovic Jun 28, 2024

Choose a reason for hiding this comment

Uh oh!

kozlovic Jun 28, 2024

Choose a reason for hiding this comment

Uh oh!

levb commented Jun 28, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

levb commented Jun 24, 2024 •

edited

Loading