Node queue limit?

[SimonUnge]

Hi,

One of the tools we have to make sure users do not overload the broker or cluster, which would lead to operational pain, is limits.
One way to overload a broker is by creating too many queues, so I have been thinking about the possibility to introduce a 'queue limit' that is 'per node', or maybe 'per cluster' - there is already a queue limit per vhost. I would like to pick the community's brain a bit, as a per node limit for a node is not out of the box obvious - a queue is not really tied to a node, such as a connection or channel is, but instead with replication can live on more nodes, and even be moved around on nodes.

But, it would be useful to limit the queue numbers on a more global way than just per vhost - you can always bypass a limit by adding more vhosts etc.

My 'simple' suggestion would be a per node queue limit - set in the config file - that is only enforces when a queue is create 'externally', i.e via a client or via HTTP, but not 'internally', e.g the limit would not be taken into account during replication of queues from another node, if that node for some reason has a higher limit. (Preferably it would not interfere say the shovel plugin to declare queues either, even if limit is reached, but I am not sure it can be distinguished?)

The limit would count all queues on the local node, regardless if they are replicated, a leader or follower etc, as the would regardless take up resources and if the limit is reached, we want to stop the creation of new queues.

Technically it is fairly straight forward to add such a limit, but I want to hear concerns and input before any potential PR.

[SimonUnge]

Any thoughts @michaelklishin @lukebakken et al?

[lukebakken]

It's only been four days since you asked!

[SimonUnge]

haha true I am sorry for seeming pushy!

[mkuratczyk]

I agree that having such a limit would be very useful in some scenarios. Some questions that we'd need to answer:

1. How could we handle exclusive queues? They are always connection-local so you'd be able to declare them or not, based on where your connection happened to established.

2. Placement of classic non-mirrored queues would probably need to consider this limit. We need to reconsider this area anyway (Deprecate queue_master_locator #10702) and perhaps the new approach would need to consider node limits to avoid declaration failures when there's capacity on other nodes.

3. We would need to expose Prometheus metric(s) to allow calculating how much of the limit is already used. Existing rabbitmq_queues metric doesn't account for followers I believe so we would need either:

• the limit as a metric and a new counter for queues including followers (or for followers only)
• or a single new metric that exposes the percentage of node limit used

4. In a multi-tenant / multi-vhost environment, it could be too easy to exceed the node limit on some vhosts so that another vhost can't potentially declare a single queue. Probably worth considering a bunch of precautions / warnings. Otherwise we can expect a lot of "I can't declare a queue" issue reports. Perhaps a soft limit is also necessary to trigger a warning?

[mkuratczyk]

Having played with the PR and discussing this with other folks on the team, we have serious concerns about the feasibility of this feature. There are still many additional nueanced but critical "details" to work out - stream and QQ replica placement, queues explicitly declared with the local placement, AMQP 1.0 / MQTT / STOMP (what the behaviour should be for each) and on top of that, performance (at the very least - this should be zero-overhead if the limit is not set).

Can you please provide more detailed justification for this feature? Given that even a single queue can consume much more resources than a large number of idle / mostly idle queues, we feel like it's a lot of complexity for relatively little gain.

As you said, there's currently a per-vhost limit. We could additionally introduce a per-cluster limit for users with multiple vhosts. That would be a much simpler feature and perhaps sufficient for many of the same situations?

[SimonUnge]

Hey @mkuratczyk Yeah I agree the complexity of this feature grew exponentially, and there are many queue types and scenarios to take into account, and getting that right, will be a challenge.

A much simpler 'per cluster' setting would absolutely be sufficient. Just want to make sure we mean the same thing. For me, a cluster wide limit would basically mean, as soon as the queue limit is hit, by looking at say rabbit_db_queue:count/0, no more queues can be created, even though in theory some node in the cluster have 'room'. Basically what the vhost limit does today, but looking at the entire cluster.

As a justification, it would be the same as for the vhost limit - avoid a broker becoming unresponsive (and in some cases crash), when the operator cannot control the actions of the client(s)/users. So, the same reasoning behind vhost limit, but acknowledging that having more vhost doesn't mean the cluster can handle more queues.

If I go ahead and make a much simpler cluster limit, where would such a limit live? I would suggest in the rabbit.conf file still, and you as an operator just have to make sure you configure your cluster correctly, with the same limits. Could of course do something more advanced and store the config in the database, or consolidate the config between the nodes (take the min value or something) but personally I would think its fine with setting it per node?

For now, I will close my current PR, and work on a simple version!

[michaelklishin]

Note that we do not have a perfect example of how such global limits would be stored.

For virtual hosts, they are a part of the virtual host metadata. For operator policies, they are, well, runtime parameters of a special kind, like all policies.

For per-node limits they are configuration settings.

I don't know if this would fit the global parameters store well or we'd need a yet another table/Khepri sub-tree.

[SimonUnge]

Right.. I'll get the logic for adhering to the limit in, and tinker with 'where' the limit would live and we can see what feels right!

[michaelklishin]

Thank you, @SimonUnge!