Posix connection load balancer selects shard 0 too much #3514

A ScyllaDB Alternator workload, using Seastar's HTTP server and the workload was doing many (tens of thousands per second) short-lived connections, exhibited bad load balancing among the shard: Shard 0 was asked to handle as many as three times the number of connections as other shards did.

We always knew that there are kinks on how Seastar load balances incoming TCP connections among shards. For example, if all the shards have no connections, shard 0 is always picked (see #290). But it always sounded like on a real, busy, system, this should be negligible. However, in this issue I outline a real scenario where given a high load of very short lived connections, shard 0 will get assigned more than its fair share of connections. The root cause is that shard 0, who does the connection assignment, has a good estimate on how busy it is, but an inflated estimate on how busy other shards are, so it assigns work to itself more often then it should:

Background: how Seastar distributes connections

Seastar runs one reactor ("shard") per CPU core. When the HTTP server calls listen() on each shard, the POSIX network stack behaves asymmetrically:

• Shard 0 (the "main thread") gets a posix_network_stack and calls the OS bind()/listen()/accept() — it owns the one real OS listening socket.
• All other shards get a posix_ap_network_stack ("accept-push") and receive a posix_ap_server_socket_impl — a virtual socket with no OS file descriptor that simply waits for connections to be forwarded to it.

This is because posix_reuseport_available() is hardcoded to return false in reactor.hh (a separate issue, not the one described here).

All incoming TCP connections are accepted exclusively by shard 0. Shard 0 then uses a conntrack load balancer to pick a target shard for each connection. The load balancer implements a least-connections algorithm: it tracks _cpu_load[i] for each shard and always picks the shard with the smallest count. When a connection is assigned to shard N, shard 0 moves the connected socket to shard N via a cross-core message (smp::submit_to), and shard N takes over processing.

Each connection carries a conntrack::handle. When the connection closes, the handle's destructor sends a decrement message back to shard 0 (smp::submit_to(_host_cpu, [cpu=N]{ lb->closed_cpu(N); })), so the load balancer can reduce _cpu_load[N].

The bug

smp::submit_to has an important optimization: when the target shard is the same as the calling shard, the function is invoked immediately and synchronously (inline), bypassing the cross-core message queue entirely:

// smp.hh
if (t == this_shard_id()) {
    return futurize<ret_type>::invoke(std::forward<Func>(func));  // inline, instant
} else {
    return _qs[t][this_shard_id()].submit(...);  // cross-core message queue
}

Now consider what happens when a connection closes:

• Connection was assigned to shard 0: the handle destructor runs on shard 0 and calls smp::submit_to(0, ...). Target == caller, so _cpu_load[0]-- happens instantly, inline.
• Connection was assigned to shard N (N ≠ 0): the handle destructor runs on shard N and calls smp::submit_to(0, ...). Target (0) ≠ caller (N), so the decrement lambda is enqueued in shard 0's cross-core message queue and only processed when shard 0 next polls that queue.

Under a high connection rate, shard 0 spends most of its time in its accept loop draining the OS accept queue. During this time it does not poll the cross-core message queues. (UPDATE co_await in posix_server_socket_impl::accept() does preempt after a quota and poll the message queues, but until this happens many accepts can run with the inflated numbers). Pending closed_cpu(N) decrements for shards 1..N-1 accumulate. At any given moment:

• _cpu_load[0] is accurate — shard 0's decrements are processed instantly.
• _cpu_load[1], _cpu_load[2], ... are inflated — their increments are instant but their decrements are sitting unprocessed in the message queue.

The least-connections algorithm therefore consistently sees shard 0 as having the lowest apparent load, and routes the next new connection there. This creates a feedback loop:

• More connections are routed to shard 0.
• Shard 0 becomes more CPU-busy.
• Shard 0 polls its message queue less frequently.
• The backlog of unprocessed decrements for other shards grows larger.
• Other shards appear even more loaded relative to shard 0.
• Even more connections are routed to shard 0.

Root cause in one sentence
The conntrack load balancer measures connection load using a counter that is decremented synchronously (inline) when a shard-0 connection closes but asynchronously (via cross-core queue) when any other shard's connection closes, causing shard 0's load to appear persistently lower and attracting a disproportionate share of new connections.

Possible fixes

• Use SO_REUSEPORT: Remove the hardcoded return false in reactor::posix_reuseport_available(). Each shard would then have its own OS socket and call accept() independently; the kernel distributes connections using its own hash, and conntrack is bypassed entirely.
• Switch load-balancing algorithm to port: Set listen_options::lba = server_socket::load_balancing_algorithm::port. This distributes by client_tcp_port % num_shards with no connection counting at all, so there is no decrement-lag issue.
• Fix the conntrack decrement path: Use a counter per shard (e.g. a std::atomic) that the closing shard updates directly, rather than always posting back to shard 0. Alternatively, ensure shard 0 processes its cross-core message queue between every accept-loop iteration so the lag stays bounded.