unknown command go-stream-locator

[Rozztig]

Describe the bug

Our service using rabbitmq-stream-go-client gets stuck at random times resulting in downtime.
The issue starts when the RabbitMQ server logs "unknown command".

¤ unknown command {request,11,{close,1,<<79,75>>}}, closing connection.
¤ unknown command {response,18,{close,1}}, closing connection.

After seeing the unknown command our service fails to create new consumers. Instead we can see an increasing number of go-steam-locator in the RabbitMQ UI. This will go on until our service runs out of memory and restarts. So the issue is that the go-stream-locators get stuck forever. (Usually around 30-40k connections with only go-stream-locators before our service OOM)

Reproduction steps

1.Throttle RabbitMQ. (resource wise in Kubernetes) NOTE: This is not necessary but increases the frequency of the problem
2.Having a churn of 200/s opening and closing. NOTE: In production we have a churn around 3/s. Still happens once/twice a week
3.Look for unknown command in RabbitMQ logs.
4.Look in the RabbitMQ UI for stuck go-stream-locators
...

Expected behavior

For the go-stream-locators to vanish and messages going out to the consumers. Not hanging and taking about resources until crashing.

Additional context

We had this issue for 6 months now. We have decreased the frequency by lowering the opening and closing of connections. But it still happens randomly even with a lot of resources available. Using RabbitMQ 3.11.14 and rabbitmq-stream-go-client 1.1.2

[michaelklishin]

We cannot suggest much with this amount of information. You have to be more specific as to how exactly you “throttle RabbitMQ” in step 1.

No protocol RabbitMQ supports, and streams in particular, were designed for connection churn and short lived connections. A closing connection can end up with an incomplete frame and things like that.

A traffic capture or an executable way to reproduce are essential when troubleshooting such workloads.

[michaelklishin]

Are you absolutely sure that your clients do not leak connections? This is a very common scenario, and there are dedicated metrics to connection churn and counts for that reason.

The maximum number of connections limit can be set up to a value of a few hundred to avoid the unbounded growth.

There is also a separate set of recommendations for environments where connection churn is unavoidable. It helps RabbitMQ nodes "recycle" TCP connections quicker, up to a point.

[Gsantomaggio]

@michaelklishin we can close this discussion in favour of rabbitmq/rabbitmq-stream-go-client#210