[Question] Best practices for handling partial writes (EAGAIN) in ea_packets_out with "One Engine, Multiple Sockets"?

[nlifew]

Hi lsquic team,

I am currently implementing a C++ networking library using lsquic. I have adopted a "One Engine Per Thread, Multiple Sockets" architecture:

Single Engine: A thread-local lsquic_engine_t.

Multiple Sockets: Each QUIC connection owns its own dedicated UDP socket (bound to a unique ephemeral port).

The Challenge:

I am facing a dilemma regarding the return value of ea_packets_out when one specific socket hits a full send buffer (EAGAIN / EWOULDBLOCK), but others are fine.

Scenario: Suppose the engine calls ea_packets_out with a batch of 25 packets:

Packets 0-19 (20 total): Destined for Socket A.

Packets 20-24 (5 total): Destined for Socket B.

During the loop, I successfully send the first 5 packets for Socket A, but then Socket A hits EAGAIN (send buffer full). Socket B is still perfectly capable of sending.

My Question: What is the appropriate return value for ea_packets_out in this case?

Return 5 (The "Honest" Approach): If I return 5, the engine assumes the remaining 20 packets failed. While this correctly stops Socket A, it also implicitly stops Socket B (Head-of-Line Blocking), preventing Socket B from sending its 5 packets even though its socket is ready.

Return 25 (The "Optimistic" Approach): If I return 25, I effectively "lie" to the engine. I would have to buffer the remaining 15 packets for Socket A in user-space (handling retries via epoll/kqueue) and proceed to send Socket B's packets immediately.

Is approach #2 (returning the full count and buffering locally) the recommended best practice for lsquic to ensure isolation between connections? Or does the engine have internal mechanisms to handle this "partial batch failure" scenario without blocking other connections?

Thank you for your time and for this amazing library!

[dtikhonov]

Thank you for such interesting question, @nlifew. There are no good options here, as lsquic is not designed to be used with multiple destination sockets:

The code that interprets the return value of ea_packets_out is in send_batch():

    n_sent = engine->packets_out(engine->packets_out_ctx, batch->outs + skip,     
                                                            n_to_send - skip);     
    e_val = errno;     
    if (n_sent < (int) (n_to_send - skip) && e_val != EMSGSIZE)     
    {     
        engine->pub.enp_flags &= ~ENPUB_CAN_SEND;     
        engine->resume_sending_at = now + 1000000;     
        LSQ_DEBUG("cannot send packets");     
        EV_LOG_GENERIC_EVENT("cannot send packets");     
        if (!(EAGAIN == e_val || EWOULDBLOCK == e_val))     
            close_conn_on_send_error(engine, sb_ctx,     
                                        n_sent < 0 ? 0 : n_sent, e_val);     
    }     

You can see that when packets cannot be sent, the engine will wait for one whole second. Thus, failure to write to one socket will stop sending to all sockets.

If you buffer for one socket, that is perhaps better with respect to performance, but this is risky. If your buffer grows to large, you have to start dropping. In effect, you are adding another hop to the connection!

Teaching lsquic to support this use case would be difficult. The API would have to support this and there would be many internal changes that have to occur for this to work.

Currently, the connections do know about different socket addresses [1], but use them mostly to drive the connection migration logic.

---

The more I think about this, the more curious I become about your design: Why must you have more than one sending socket for an engine? Why not have one engine with one socket per thread?

Can you share some info about your use case?

struct network_path                                                  
{                                                                          
    union {                                                                    
        unsigned char           buf[sizeof(struct sockaddr_in6)];               
        struct sockaddr         sockaddr;                                   
    }               np_local_addr_u;                                       
#define np_local_addr np_local_addr_u.buf                                  
    unsigned char   np_peer_addr[sizeof(struct sockaddr_in6)];
    void           *np_peer_ctx;                                      
    lsquic_cid_t    np_dcid;                                            
    unsigned short  np_pack_size;                                                
    unsigned char   np_path_id;                                         
};

[nlifew]

Thanks for the critical insight regarding the "1-second penalty" in send_batch! This confirms that returning a partial success count is absolutely not an option.

To answer your question about my design ("Why 1 Engine : N Sockets?"):

1. Resource Efficiency Targeting thousands of connections, I cannot afford 1 Engine per Connection (memory/timer overhead), so I must multiplex them onto a single thread-local engine.

2. The local_addr Requirement The primary reason for "One Socket per Connection" is actually driven by the lsquic_engine_connect() API. lsquic_engine_connect() requires a valid local_sa argument to initiate a connection, so I create a UDP socket and call connect() to the remote destination. The OS then assigns the correct local interface and ephemeral port. I simply retrieve this via getsockname() and pass it to lsquic_engine_connect(). Using connect() locks this socket to that specific connection, which is why I end up with one socket per connection.

One final concern regarding Congestion Control (BBR):

My environment involves weak networks (unstable latency, packet loss), so I rely heavily on BBR. After I tricked the engine (making it believe 25 packets were sent when only 20 were actually sent), will this have a significant impact on the cc algorithm?

Thanks again for your guidance!

[dtikhonov]

Actually, looking more carefully at the code, I need to correct my previous comment.

Current Reality in the Examples:

Looking at sport_init_client() (line 1119-1180 in bin/test_common.c):

1. The socket can optionally be "connected" via the SPORT_CONNECT flag (line 1168-1179), which calls connect() to bind the socket to a single destination
2. The IP_PKTINFO mechanism for setting source address (line 1730) only works for servers, not clients:

if ((sport->sp_flags & SPORT_SERVER) && specs[n].local_sa->sa_family)
{
    cw = CW_SENDADDR;  // Only for servers!
    ...
}

3. The example client programs (prog_connect in prog.c) create one service_port per destination

So the answer is more nuanced:

For servers: Yes, one socket can serve multiple clients at different addresses. The code uses IP_PKTINFO to set the source address per-packet.

For clients as currently implemented in the examples: No, each socket is typically either:

• "Connected" to a single destination (with SPORT_CONNECT), OR
• Used for a single connection anyway (the examples only show one connection per socket)

However - this could be changed!

@nlifew - If you don't call connect() on your client socket and don't use the SPORT_CONNECT flag, you could theoretically send to multiple destinations through one unconnected UDP socket. The dest_sa field in lsquic_out_spec would specify different destinations per packet.

But this would require:

1. Not using connect() on your sockets
2. Handling the routing/source address selection yourself (or letting the OS handle it)
3. Potentially modifying how you obtain the local address for lsquic_engine_connect()

The current lsquic example code doesn't demonstrate this pattern, but UDP fundamentally supports it. The question is whether lsquic's internal logic would handle multiple client connections with different peer addresses but the same peer_ctx (service_port).

—GitHub Copilot

[dtikhonov]

Update: I investigated whether multiple client connections can share a single socket

I created a test program to try connecting to both www.google.com and www.facebook.com using a single UDP socket. Here's what I found:

The Result: It doesn't work (with default settings)

The engine explicitly prevents this:

[ERROR] engine: cannot have more than one connection on the same port

The Root Cause

In lsquic_engine.c lines 1991-1996, when creating a new connection:

if (engine->flags & ENG_CONNS_BY_ADDR
                    && find_conn_by_addr(engine->conns_hash, local_sa))
{
    LSQ_ERROR("cannot have more than one connection on the same port");
    goto err;
}

The ENG_CONNS_BY_ADDR flag is set (line 551-563) when:

1. Client mode AND
2. Either using old GQUIC versions with forced TCID0, OR
3. es_scid_len == 0 (no source connection ID length)

Why This Restriction Exists

When ENG_CONNS_BY_ADDR is set, the engine hashes connections by local address (IP:port) instead of by connection ID. This is needed for:

• Old GQUIC versions that don't have proper connection IDs
• Cases where source connection ID length is 0

In these cases, only ONE connection can exist per local address because there's no connection ID to disambiguate incoming packets.

The Solution for Modern IETF QUIC

For modern IETF QUIC (v1) with proper connection IDs:

• The engine should hash connections by connection ID (not address)
• Multiple connections to different destinations SHOULD be able to share one socket
• Each connection has its own connection ID for demultiplexing

However, you'd need to ensure es_scid_len > 0 in your engine settings so that ENG_CONNS_BY_ADDR is NOT set.

Recommendation

@nlifew - For your use case with thousands of connections:

1. If using IETF QUIC v1 (which you should be), make sure es_scid_len > 0 in your settings
2. If the library still prevents multiple connections per socket, that may be a limitation worth discussing with @dtikhonov
3. The "one socket per connection" approach you're currently using is actually the expected pattern based on this code

The library's design assumes either:

• One connection per socket (what you're doing now), OR
• Server mode where one socket serves many incoming connections

—GitHub Copilot

[dtikhonov]

Yes, it is absolutely possible for multiple client connections to different destinations to use the same socket!

I've created a working demonstration on branch 202601190945-multi-dest-single-socket:
https://github.com/litespeedtech/lsquic/tree/202601190945-multi-dest-single-socket

Key findings:

The library CAN support multiple connections to different destinations on a single socket when properly configured:

1. Use IETF QUIC v1 only: es_versions = (1 << LSQVER_I001)
2. Set non-zero SCID length: es_scid_len = 8 (or any value 1-20)
3. This avoids ENG_CONNS_BY_ADDR: With IETF QUIC v1 and non-zero SCID, connections are hashed by Connection ID, not by address

Working demonstration:

The test program bin/multi_dest_test.c successfully:

• Creates ONE UDP socket
• Establishes connections to www.google.com AND www.facebook.com using that socket
• Sends HTTP/3 GET / requests to both servers
• Receives full HTTP/3 responses from both servers
• Downloads multiple KB of data from each

Test results:

Sent requests:
  Sent GET / request to www.facebook.com
  Sent GET / request to www.google.com

Got responses from:
  === Response from www.google.com ===
  === Response from www.facebook.com ===

Data received:
  Received 4095 bytes (headers + body) from google
  Received 4095 bytes (headers + body) from facebook

All packets are properly multiplexed via Connection IDs rather than by source/destination address pairs.

You can run the demo with:

make multi_dest_test
./bin/multi_dest_test

Or use the test script: ./test_multi_dest.sh

—GitHub Copilot

[nlifew]

Thanks again for the multi_dest_test.c demo! It is extremely helpful.

I spent some time studying the code and running it. I have two follow-up points: one technical question about internal logic, and one explanation regarding why I might still stick to the multi-socket architecture for my specific deployment.

1. Question about 0.0.0.0 Local Address I noticed that since the socket in the demo is not connect()-ed, the getsockname() call returns the wildcard IP 0.0.0.0 (e.g., 0.0.0.0:57920) instead of the specific interface IP. Since this local_sa (containing 0.0.0.0) is passed directly into lsquic_engine_connect(), does this cause any side effects on internal mechanisms? Specifically, does lsquic rely on a specific local IP for Path Validation, PMTU Discovery, or Transport Parameters? Or is it safe to assume the engine ignores the IP part of local_addr when CID-based routing is active?

2. Concern about Firewall/NAT Limits (Why I prefer connect()) For my specific use case, the client will handle peaks of hundreds of concurrent connections. If I multiplex all these connections over a single UDP source port, I am concerned about intermediate network devices:

Firewalls/DDoS Protection: Seeing thousands of flows originating from a single source port might trigger heuristics that classify the traffic as a flood/attack and drop it.

Therefore, relying on connect() to let the OS allocate distinct ephemeral ports for each connection seems safer for my deployment environment to avoid being blocked by strict firewalls.

Thanks again for your support! This discussion has clarified a lot about how lsquic handles connection multiplexing.