Consumer cannot consume messages

[dormanze]

Exclusion and autodelete queue have one consumer.
The client are normal, and no exception information is displayed. It is unknown why the value of Consumer capacity is 0% and no message is being consumed.
other channle consume message is ok.

channel list:

connection config:

I'm guessing this is a concurrency issue because our client has the ability to automatically recreate queues due to channel exceptions as well as consumer capabilities.
The faulty queues have experienced channel exceptions and then the queues are restored.
At the same time, I run the rabbitmqctl list_consumers command, and no consumer in the abnormal queue is found.

[michaelklishin]

@dormanze see server logs and your own application's logs. We do not guess in this community.

Two most common reasons for such behaviors are

1. A consumer has reached its prefetch limit and does not acknowledge deliveries (e.g. it is stuck)
2. A delivery handler had an exception it did not handle (how exactly these are handled depends on the client library, and the Java one has two different exception handler implementations)

If this is a condition you can reproduce, take a traffic capture and see exactly what your client receives and sends.

Please take it from here, we will not be troubleshooting your apps.

[dormanze]

I have checked the two causes. If the problem is caused by the card owner on the client and the client uses the manual ACK mode, the unack message should exist on the monitoring page. Actually, there is no unack message.
An exception on the client causes the channel to be deleted and re-created. The client code contains the manual recover action. Therefore, after the consumption times out, the channel is closed abnormally. I restore the channel and create a new queue and consumer.
This is not an inevitable problem, I have 50,000 temporary + exclusive queues, and only 20 queues have this phenomenon.
Consumer consumption timeout logs exist on the server.


After the exception occurs, the queue and consumer disappear and rebuild.
For the same client code, I found no problem in version 3.11.9. Compared with the difference, I guess it may be caused by this problem.
https://github.com/rabbitmq/rabbitmq-server/pull/6821/files#diff-18ee4d03f8a7877fb0efb3befbdce6b22e2726458dffed65453dbab0a6c96aeeL283

I don't know why transactional operations are removed here.
I'm not good at erlang. Please forgive me if you have ridiculous remarks.

[michaelklishin]

Your consumer does not acknowledge the delivery in time and hits this delivery timeout. After that the channel is closed and therefore there can be no more deliveries on it.

You can either acknowledge earlier or increase the timeout be it exists for a reason: to prevent stuck consumers from preventing log compaction of quorum queues. So running without it is highly recommended against.

[dormanze]

Yes, I know my consumers have a problem with spending timeout. I'll optimize this, but my client has a mechanism to recover after a channel exception. The problem I have is that after the consumption times out, I rebuild my channel, create a new queue, and create a new consumer, but the new consumer cannot continue to consume messages.

[michaelklishin]

@dormanze this channel closure is no different from any other channel closure. We won't debug your apps (and you haven't provided any details for anyone to be able to), so please take it from here.

[michaelklishin]

You haven't provided any evidence that snippets in rabbit_andqueue_process are relevant. When a queue process stops, it updates the schema data store. The only way a client can observe this is if it tries to re-declare the queue as it is being deleted concurrently. Then the behavior can indeed be confusing

In fact, this is a fairly well known scenario:

1. A client with an exclusive queue with a well-known name (that is, not server-generated) loses connection with (while using an exclusive queue EQ)
2. RabbitMQ starts deleting EQ
3. The client concurrently reconnects and tries to redeclare it

So the client tries to consume from a queue it thought it's just declared but the queue was concurrently deleted and the client now tries to consume from a queue that no longer exists, which is a channel exception that your code does not handle or handles in a sub-optimal way.

This is a fundamental race condition between two separate processes, and it fundamentally cannot be resolved without client-side retries or waiting before recovery.

@dormanze this scenario has been around for a long time. The solution is to not use exclusive queues and/or auto-delete queues with well-known names (let the server generate them), and inject a recovery delay into clients, then they will try to declare and consume after the exclusive queue has been deleted.

And, of course, avoid those consumer timeouts that close your channel in the first place.

[michaelklishin]

I have documented this behavior in the Queues guide. It will be live in a few minutes.