Race condition: notifications processed out of order due to goroutine-per-message #20
Description
Summary
The ACP SDK processes incoming notifications in parallel goroutines, which causes message ordering to be lost when multiple notifications arrive in quick succession.
Root Cause
In connection.go line 86-87, each incoming method/notification spawns a new goroutine:
case msg.Method != "":
go c.handleInbound(&msg) // Each notification in separate goroutine!While this is fine for request/response patterns, it breaks ordering guarantees for notifications like session/update with agent_message_chunk data.
Impact
When an agent streams text rapidly (e.g., multiple agent_message_chunk notifications), the client receives them out of order. This causes:
- Garbled/scrambled text in the UI
- Corrupted data written to databases
- Poor user experience
Reproduction
We created a test that demonstrates this issue. It sends 100 session/update notifications with agent_message_chunk data in sequence and checks if they arrive in order.
Test Results
Running the test multiple times shows consistent out-of-order delivery:
- Run 1: 80/100 chunks out of order
- Run 2: 5/100 chunks out of order
- Run 3: 71/100 chunks out of order
- Run 4: 40/100 chunks out of order
- Run 5: 78/100 chunks out of order
Proposed Fix
For notifications (messages without an id field), process them sequentially instead of in parallel goroutines. One approach:
case msg.Method != "":
if msg.ID == nil {
// Notification - process sequentially to preserve order
c.handleInbound(&msg)
} else {
// Request - can process in parallel
go c.handleInbound(&msg)
}Alternatively, use a dedicated channel/queue for notifications to maintain FIFO ordering while still allowing concurrent processing of requests.
Environment
- acp-go-sdk version: v0.6.3
- Go version: 1.23