Single active consumers locked in "waiting" state

[jonnepmyra]

Hi!

We’ve recently adopted RabbitMQ Streams (currently ~800 in production, some with up to 20 partitions) and they’ve been an excellent fit for multiple use cases.

However, we’re struggling with a scenario where all consumers in a consumer group end up “locked” in waiting state.

We don’t have a structured way to reproduce this yet, but it typically occurs during RabbitMQ node restarts.

Cluster setup

• 3-node RabbitMQ cluster
• RabbitMQ version: 4.1.5 on all nodes

---

What we observe

Given a SuperStream with 4 partitions (1, 2, 3, 4), it’s almost always Partition 3 where consumers get stuck.

Partition 3 has many consumer groups (in this example 11 groups).
Each group has 3 consumers (so 33 consumers total for that partition).

For each group, we therefore expect:

• 1 active consumer
• 2 waiting consumers

But from time to time, all consumers within a consumer group end up in waiting.
Once this happens, the client app never receives any Activation event in the .NET client’s ConsumerUpdateListener callback.

This remains true even if we try:

1. Kill only the stream consumers (not the whole client app) and let them reconnect
   • After reconnecting, they end up in waiting again
2. Restart the client application instances one by one (in case there is a race condition / lock in our application)

Only workaround we’ve found

The only thing that recovers the system is to:

• Stop all application instances at once, and then start them again

In other words:

As long as we have any existing connections to the stream, the “locks” don’t go away.

---

Additional details

This screenshot shows the output of: rabbitmq-streams list_stream_group_consumers for the affected stream + consumer group:

[acogoluegnes]

Thanks for the report. We identified a bug that could be the cause of this deadlock, see rabbitmq/rabbitmq-server#15353.