Consider unacknowledged messages for delete queue if-empty

[zvirja]

Is your feature request related to a problem? Please describe.

In our product we have jobs subsystem where we create a queue per job. The idea is that queue contains job items and queue consumer handles those job items. As a result of single item handling we could enqueue more job items to the end of the queue. So queue could dynamically grow and shrink. But eventually queue becomes empty when all the items are handled.

The problem is to detect a moment when the queue is empty and it's safe to finalize the job and clean up the queue. We want to do that as quick as possible, so job is marked as done fast. It would be very convenient to use if-empty parameter for the queue delete action (so if we managed to successfully delete the queue we know that we are done), but unfortunately if-empty does not consider non-acked messages. Apparently it was intentional: see this mail discussion.

Describe the solution you'd like

Either would work for my problem:

• Have something like if-empty for Delete Queue, but also consider the non-ACKed messages
• Allow to get up-to-date queue stats to detect if queue is empty including non-acknowledged. Currently Declare Queue returns only Ready count, so it's not suitable.

Describe alternatives you've considered

For now we are using Management API, but that is non-reliable, slow and extremely inaccurate. Sometimes it shows zero count, while in reality the queue is not empty. Often it out of date. So the only desperate thing we are doing is querying stats from Management API for a few times with a delay. If every time queue is empty - we believe it's empty. But that approach is slow, clumsy and still does not guarantee the correctness. We also delay job completion for ~30 secs which degrades user experience.

Additional context

I would say that the existing if-empty behavior for delete queue is not very practical. When you have messages in flight (non-acked yet), it is still possible to that they are NACK-ed (e.g. due to connection issues) and they are returned back to queue. So you actually never want to delete a queue if you have those, as that's a possible to get them lost. That's exactly what happened when I tested them locally and app crashed after deleting the queue but before ACK-ing the messages in flight - messages simply got lost.

The main argument for the current behavior given the mail was alignment and consistency across various API. I would though argue that the statement from the mail is true:

Also, conceptually, messages that are pending acknowledgement are not really in the queue anymore; or, rather, there is nothing in AMQP that suggests, let alone mandates, such a view.

Given that they are returned to the queue in case of lost connection makes them still belonging to the queue.

I would suggest to just change the existing behavior of if-empty to consider non-ACKed or add a configuration flag for that.

[michaelklishin]

This was discussed several times and was discussed against.

This violates two fairly common assumptions:

• the idea that a delivered and unacknowledged messages are "owned" by the application (an assumption that exists in all messaging protocols RabbitMQ supports)
• the idea that an unacknowledged message can be acknowledged at any moment after it's been delivered (and before consumer delivery timeout hits, or connection closes, various failure-like scenarios like that)

If you really want to delete a queue, just delete it without any conditions. Otherwise, a queue with oustanding unacknowledged messages is not really semantically empty.

[lukebakken]

Is that something which is possible raise as a feature request to the core team? So that maintainers could spec the design and implement the change?

Both @michaelklishin and I are on the core team.

I believe it's a feature many will benefit from.

It would be great if other RabbitMQ users could request this feature, especially if they pay for RabbitMQ (via a support contract). They should follow up here to express interest.

Do you think it's realistic something like this will be implemented?

If a customer requests it, it is much more likely. Otherwise, Team RabbitMQ has a long, long list of higher priority work to do.

[michaelklishin]

Alternatively you can consider setting queue TTL on your queues, not any additional features or behaviour changes. And maybe message TTL but that's a much more consequential option to consider.

As long as the consumers are there, the queue won't be deleted. When they go away, eventually they will be.

A queue with delivered messages pending acknowledgement are not empty. We are not interested in expanding the matrix of features related to queue deletion, there are more than enough.

[zvirja]

Okay, so it seems that the plan is to keep it open and then see if there is more interesting. Gotcha!

If a customer requests it, it is much more likely.

@lukebakken What is that? Like I am also a user of the product 😄 Do you mean those paid requests/support contracts?

@michaelklishin Thanks for the advice. I considered using that as well, but I don't see how to make it work. Queue TTL assumes that there shouldn't be any consumers. But specifically in my case I have chicken-and-egg problem. I need to somehow detect that queue became totally empty, so I could unsubscribe from it, so it could be removed. But I don't have any API to check if the queue is actually empty to accomplish that...

We are not interested in expanding the matrix of features related to queue deletion, there are more than enough.

That is fair enough. More I am thinking about it, more it makes sense to have feature about total queue size (Ready + non-ACK), as then it could be used here.

---

@michaelklishin @lukebakken I have another question though, out of curiosity. With the current implementation and from what we have discussed above, if you delete queue with pending non-ACKed messages, you effectively have data loss (e.g. if consumer crashes). With that it means that QueueDelete(if-empty: true) with active consumers is ALWAYS going to have problems of data loss. Like ALWAYS. And those issues are quite subtle to catch.

If QueueDelete checked for total count, then it would be another story. If combined with Publish(mandatory: true), you would technically prevent any possible data loss. Like if deletion of the queue fails - it means that it's not safe yet to remove it. If it managed to remove it, then publishing would fail. It would be caught at both sides. The key here would be atomicity of "transactions", so usage of QueueDelete(if-empty: true) would be a must.

What is the envisioned use case of QueueDelete(if-empty: true) in its current behavior? Like why people would ever use it and what for? Which practical problem does it solve which cannot be solved with e.g. QueueDeclare(passive: true)->Count + QueueDelete())?

Would be happy if you elaborate on that in more detail, so I and hopefully others will have answers 🙏

[lukebakken]

Do you mean those paid requests/support contracts?

Yes. Feature requests from people who use RabbitMQ for free are rarely taken into consideration, unless the request is accompanied by a PR that implements it.

if you delete queue with pending non-ACKed messages, you effectively have data loss (e.g. if consumer crashes). With that it means that QueueDelete(if-empty: true) with active consumers is ALWAYS going to have problems of data loss. Like ALWAYS. And those issues are quite subtle to catch.

Yep. And yet, we don't receive reports about this issue 🤷‍♂️

QueueDeclare(passive: true)->Count + QueueDelete())?

Those operations would not be atomic.

You make a good case for changing the QueueDelete / if-empty behavior, or adding a different API that takes outstanding confirms into account. Again, if this were to be requested from a large number of RabbitMQ users, or from a paying customer, we would raise its priority.

As it stands, Matthias' response here still applies today:

Alas I don't know how we'd go about meeting such a requirement; I'd be reluctant to change the semantics of something that has been around for five years, particularly since we've had zero complaints to date about the current semantics.

[michaelklishin]

Consumers are usually responsible for declaring and deleting their own queues. Deleting someone else's queues can result in data loss in many different situations. In modern messaging protocols, applications are responsible for the topology. RabbitMQ cannot know if what an application attempts to do is safe or not. Saving applications from themselves has been an ongoing topic over the last ≈ 15 years or so that I've been a contributor, and it's a battle you cannot win.

Queue and message TTL give you more options. So do auto-delete and exclusive queues. For example, another application simply cannot delete a queue declared on a different connections (another app, a different app instance). If your apps cannot agree on when and how to delete queues, use options that do not require explicit deletion.

If you need a shared data structure where consumers do not or cannot coordinate, consider using a stream. Streams are rarely deleted and their only way of deleting data is TTL.

There are many drastically more important problems to solve in RabbitMQ, its client libraries, its operational characteristics, and so on.

Introducing a protocol extension (a new protocol method to obtain more queue metrics) would mean updating at least several client libraries to start with. That can roughly triple or quadruple the amount of work.