Max-length policy is not respected

[jaroslaw-bochniak]

Using 3.9.13 version of RabbitMQ in kubernetes. When quorum queue has no consumers it shows in the UI the applied policy, but the settings are not respected. Queue has more messages than it should.

Worth to note that on the other environment where we have the same setup (same RabbitMQ, same configuration of the policies and the same queue) it does work as expected. I was unable to pinpoint the reason for such miss behaviour, but I have observed this situation in the past in similar cases.

[kjnilsson]

Did you apply the policy after the messages arrived at the queue?

…

On Wed, 23 Feb 2022 at 10:49, Jarosław Bochniak ***@***.***> wrote: Using 3.9.13 version of RabbitMQ in kubernetes. When quorum queue has no consumers it shows in the UI the applied policy, but the settings are not respected. Queue has more messages than it should. [image: Opera Snapshot_2022-02-23_114135_mq int-fran srinternal net] <https://user-images.githubusercontent.com/5468094/155304619-599cd2c4-77f3-4b75-8c00-c6c9cdae990c.png> — Reply to this email directly, view it on GitHub <#4174>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJAHFDQ33SQEQZ35BXMLGLU4S3S3ANCNFSM5PD7EZWQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- *Karl Nilsson*

[jaroslaw-bochniak]

Hard to say that, but definitely I am still observing new messages coming in to the queue after the policy has been applied.

I have also made a test and moved all messages to different queue, and later moved them back to this queue. This also had no effect on number of messages kept in queue after the second move.

[jaroslaw-bochniak]

I believe that this problem may occur in situations when:

• policy definition exists in the cluster
• queue definitions are imported via management plugin

Deletion and subsequent recreation of policy seems to be fixing an issue.

[michaelklishin]

We do not guess on this team. Please provide a set of steps we can use to reproduce or we won't be able to help.

99% of policy effects issues come down to policy priorities and users
having multiple policies that match a queue with equal priorities. In that case, consider that a random policy (of the matching ones) will be applied, and it can change after you recreate a policy.

[michaelklishin]

OK, according to the screenshot, that's an effective policy definition on the queue page. Then it may be important when it was applied
(at queue declaration time or some time later).

[jaroslaw-bochniak]

@michaelklishin I have managed to replicate it in two scenarios:

Case 1 steps

• only single policy is defined ha-all (no max-len limitation)
  
• queue is created test-q2 and ha-all policy is applied
  
• perf test is executed to publish 2000 messages to that queue

perf-test --producers 10 -C 200 --predeclared --queue test-q2 --confirm 1 --consumers 0

• as expected there is 2000 messages in the queue
  
• queue is purged from messages
• new max-len policy is created with higher priority (2) and with max-length limitation set to 1000
  
• it is visible that test-q2 has the max-len policy applied
• another batch of 2000 messages is published via perf-test
• queue contains 2000 of messages despite the fact that max-length is set to 1000
  

[jaroslaw-bochniak]

The second case also evolves around HA settings for classic queues, the sequence is as follows:

• 2 policies are available
  
• queue is created and receives policy with higher priority (ec2-migration)
• ec2-migration policy is removed, which leaves only max-len policy applied to the queues
• 2000 messages are being published and like in previous scenario 2000 messages lands in the queue instead of 1000
• deleting max-len policy and recreating it again works as expected

[dcorbacho]

I cannot reproduce your issue with 3.9.13 or 3.9.14, neither with pre-existing policies nor applying policies once the queue is created. What really surprises me, is that in one of your screenshots the ha-all policy has been applied to quorum queues. That is not possible in recent versions of Rabbitmq.

As Michael said, only one policy is applied at a time, so take that into account when applying several policies. But your ec2-migration will never be applied to quorum queues as ha-mode is not supported. Thus, that policy is irrelevant for this example and we're dealing with a single policy here.

I'd be glad to have another look at this if you can provide me with a script that automatically reproduces this issue.

[jaroslaw-bochniak]

I have uploaded replication video, maybe it will help. Not sure if it matters but for setting ec2-migration policy

I have used rabbitmqctl set_policy -p "/" --apply-to queues ec2-migration ".*" '{"federation-upstream":"ec2-upstream","ha-mode":"all","ha-sync-mode":"automatic"}' --priority 11

I am aware about policy priorities and matching.
https://www.youtube.com/watch?v=zRvg6OvpJ-8

[dcorbacho]

Thanks @jaroslaw-bochniak, this video has been really useful and it is indeed a bug. I haven't looked at the internals yet, but I can reproduce it now if the ha policy is pre-existing - which I had only tried with supported policies.

[dcorbacho]

PR available in #4463