Detect stale server connections during client idle in transaction

[vadv]

Problem

A client (TLS/SSL) was holding a transaction through pg_doorman. The client connection dropped (cause unknown — crash, network, freeze), but pg_doorman did not detect it. The server connection to PostgreSQL remained in "active/idle" state. PostgreSQL eventually killed the backend via idle_in_transaction_session_timeout (~6 min), but pg_doorman still considered the slot occupied. New clients received QueryWaitTimeout.

Root cause: In the transaction loop (src/client/transaction.rs), pg_doorman only reads from the client socket. When PostgreSQL killed the backend (sent FATAL + TCP FIN), pg_doorman didn't notice because the server socket was not monitored between queries.

Solution

1. Server socket monitoring via select!

In the transaction loop, where pg_doorman waits for the next client message, we now also monitor the server socket. This is encapsulated in Client::wait_for_next_message():

async fn wait_for_next_message(&mut self, server: &Server) -> Result<NextClientMessage, Error> {
    loop {
        tokio::select! {
            biased;
            result = read_message(&mut self.read, self.max_memory_usage) => {
                return result.map(NextClientMessage::Message);
            }
            _ = server.server_readable() => {
                if server.check_server_alive() {
                    continue; // spurious readiness
                }
                return Ok(NextClientMessage::ServerDead);
            }
        }
    }
}

When PostgreSQL kills a backend (idle_in_transaction_session_timeout, pg_terminate_backend), pg_doorman immediately detects it and releases the pool slot.

2. Server liveness check encapsulated in Server

New methods on Server:

• server_readable() — async future for select!, waits until socket becomes readable
• check_server_alive() — synchronous probe: try_read returning WouldBlock means alive, anything else (EOF/error) means dead

New methods on StreamInner:

• readable() — cancel-safe readiness notification on the underlying socket
• try_read() — non-blocking read for spurious readiness verification

No more server.stream.get_ref().try_read() leaking from transaction.rs — the liveness logic is fully encapsulated.

3. RAII guard for CLIENTS_IN_TRANSACTIONS counter

The old code managed CLIENTS_IN_TRANSACTIONS with manual fetch_add/fetch_sub calls separated by ~200 lines and multiple early return paths. This leaked the counter on several error paths.

Fix: TransactionGuard — increments on creation, decrements on drop:

struct TransactionGuard;

impl TransactionGuard {
    fn new() -> Self {
        CLIENTS_IN_TRANSACTIONS.fetch_add(1, Ordering::Relaxed);
        Self
    }
}

impl Drop for TransactionGuard {
    fn drop(&mut self) {
        CLIENTS_IN_TRANSACTIONS.fetch_sub(1, Ordering::Relaxed);
    }
}

Counter leaks fixed for free on these paths:

• sync_parameters().await? error
• Deferred BEGIN send/recv errors
• write_all_flush error after transaction
• Terminate (X) handler (existing bug — was missing decrement)

4. Spurious readiness handling

server_readable() can fire spuriously due to stale epoll readiness flags from BufStream operations. check_server_alive() handles this: try_read returning WouldBlock clears the stale flag and the next readable() poll blocks correctly.

Performance Analysis

The tokio::select! is on the idle path, not the hot path.

Aspect	Impact
Hot path (query execution)	Zero — select! not involved
Idle path (waiting for client)	~nanoseconds — poll overhead on an I/O-blocked path
Spurious readiness verification	One non-blocking read() syscall per query roundtrip
Memory	Zero additional allocations

Cancel safety

All futures in the select! are cancel-safe:

• read_message() — blocked on read_u8() (first byte), no bytes consumed until that returns
• server_readable() — readiness notification only, no data consumed

BDD Test

tests/bdd/features/stale-server-detection.feature:

@stale-server-pg-terminate-backend — backend killed via pg_terminate_backend() while client holds transaction with pool_size = 1. pg_doorman detects dead server, releases pool slot, new client successfully gets connection.

Test plan

• cargo build compiles
• cargo clippy — no warnings
• cargo test --lib — 259 passed
• BDD @stale-server-detection — 16 steps passed
• grep CLIENTS_IN_TRANSACTIONS.fetch_sub — only in TransactionGuard::drop
• grep server.stream.get_ref() in transaction.rs — zero occurrences

🤖 Generated with Claude Code