Queue redeclaration behavior when x-consumer-timeout does not match the original declaration

[qraynaud]

Describe the bug

Trying to assert a queue that already exists but giving it different x-arguments does neither change those nor fails. I think it should do either one of them.

Reproduction steps

Example using amqplib from node.js:

1. assertQueue("some-queue", { arguments: { "x-consumer-timeout": 2_000 } }) => 🟢
2. assertQueue("some-queue", { arguments: { "x-consumer-timeout": 5_000 } }) => 🟢 (but does nothing)

That is the same using any custom argument:

1. assertQueue("some-queue", { arguments: { "x-custom-company-arg": 0 } }) => 🟢
2. assertQueue("some-queue", { arguments: { "x-custom-company-arg": 1 } }) => 🟢 (but does nothing)

Expected behavior

Either:

1. assertQueue("some-queue", { arguments: { "x-consumer-timeout": 2_000 } }) => 🟢
2. assertQueue("some-queue", { arguments: { "x-consumer-timeout": 5_000 } }) => 🟥 (PRECONDITION_FAILED? I think it should do the same thing as for x-dead-letter-exchange for example)

Or:

1. assertQueue("some-queue", { arguments: { "x-consumer-timeout": 2_000 } }) => 🟢
2. assertQueue("some-queue", { arguments: { "x-consumer-timeout": 5_000 } }) => 🟢 (argument edited on the queue)

Additional context

Right now, I don’t think there is a single way using amqplib to find out a queue is not properly configured on an environnement and getting those error messages early is very important to my company work.

Maybe I'm missing something but I looked for this everywhere and could not find any ad-hoc solution.

Also, reading the official documentation, one might expect those arguments to be edited since it clearly reads: Most optional arguments can be dynamically changed after queue declaration but there are exceptions. But right now, the documentation fails to say how those can be edited if not using this method.

Note: last tested on [email protected].

[michaelklishin]

@qraynaud an attempt to redeclare a queue with a different set of arguments results in a channel exception with a few specific exceptions, namely there are settings that avoid an exception where the difference is in the queue type field and it is not set by the client.

The docs do not mention that these optional parameters are changed via a policy. You should not expect to be able to change them via redeclarations.

Finally, our team does not maintain amqplib.

[qraynaud]

Yeah. I know about amqplib. It was just for the example. It does the same thing using RabbitMQ Management. And it does not result in a channel exception with at least the 2 examples I provided.

If I try to change durable in the UI it fails (as expected):

If I try to change x-consumer-timeout, it does not (not as expected then):

Clicking on Add Queue with those refresh the page and everything goes as if it worked but it did nothing.

[qraynaud]

I really think this is truely a bug and not expected behavior and deserving of an issue…

[michaelklishin]

We will see after #12339 is addressed.

[michaelklishin]

I have clarified the docs on the subject of optional argument immutability.

So far the re-declaration behavior is consistent between 4.0.2 and 3.13.7. I will compare it to 3.13.0 and 3.12.14 tomorrow then share it with my findings.

[qraynaud]

I expect it to be consistent too. I'm not saying this changed. I'm more arguing this is not normal that trying to declare a queue with a different argument value would sometimes do nothing and sometimes fail. It is totally unexpected behaviour. And my main point is there is absolutely no way to ensure a queue is properly configured on the consumer side because of this. We have to trust it is okay. And if a developer wants to change an argument, he has no way of knowing if his change was applied or not when the queue declaration happens. For things like durable the queue assertion fails and then you can either raise a ticket with ops or if the queue is not very crucial, just migrate it automatically (using a temporary queue as a buffer for example). For things like x-consumer-timeout or custom internal args, you are just out of luck.

[lukebakken]

And my main point is there is absolutely no way to ensure a queue is properly configured on the consumer side because of this.

Sure there is, use the HTTP API. It's not convenient, but it's possible.

[michaelklishin]

If there absolutely no way, how does the entire community do it then?

The answer is very likely: they use policies for the optional arguments that can change.

[michaelklishin]

Exaggerations will not convince me this is something that has to change. If this behaviour was present in 3.11, 3.12, 3.13 and now 4.0, this is not obvious whether changing it would be a net gain.

If no one else has noticed in years, it's very difficult to argue that "there is absolutely no way to configure a consumer correctly." There is: don't set optional arguments except for queue type in the apps, use policies.

[michaelklishin]

The plan so far is:

1. Move my irb snippets to a script that can be turned into a test
2. Confirm whether the channel exception is raised in 4.0.2, 3.13.7, 3.13.0, 3.12.14 and 3.12.0 to compare
3. See what the rest of the team thinks

This may be a side effect of a DQT-related set of changes in 4.0.x and 3.13.x. But if 3.12.x behaves the same way, this would be harder to justify as a change.

[michaelklishin]

@qraynaud the following script uses Bunny and an exception is thrown upon redeclaration:

require 'bunny'

bunny = Bunny.new
bunny.start
ch = bunny.create_channel

qn = "cq.server.12349"

ch.queue_declare(qn, type: "classic", durable: true, arguments: {"x-max-length": 10})
puts "Declared a queue #{qn}"

puts "Redeclaring..."
begin
  ch.queue_declare(qn, type: "classic", durable: true, arguments: {"x-max-length": 20})
rescue Exception => e
  puts "Caught an exception: #{e.message}"
end

sleep 1
puts "Exiting"

Here is the output:

Declared a queue cq.server.12349
Redeclaring...
Caught an exception: PRECONDITION_FAILED - inequivalent arg 'x-max-length' for queue 'cq.server.12349' in vhost '/': received '20' but current is '10'
Exiting

Here is what the script defines:

{
  "consumer_details": [],
  "arguments": {
    "x-max-length": 10,
    "x-queue-type": "classic"
  },
  "auto_delete": false,
  "consumer_capacity": 0,
  "consumer_utilisation": 0,
  "consumers": 0,
  "deliveries": [],
  "durable": true,
  "effective_policy_definition": {},
  "exclusive": false,
  "exclusive_consumer_tag": null,
  "garbage_collection": {
    "fullsweep_after": 65535,
    "max_heap_size": 0,
    "min_bin_vheap_size": 46422,
    "min_heap_size": 233,
    "minor_gcs": 12
  },
  "head_message_timestamp": null,
  "idle_since": "2024-09-23T19:47:34.222-04:00",
  "incoming": [],
  "memory": 4616,
  "message_bytes": 0,
  "message_bytes_paged_out": 0,
  "message_bytes_persistent": 0,
  "message_bytes_ram": 0,
  "message_bytes_ready": 0,
  "message_bytes_unacknowledged": 0,
  "messages": 0,
  "messages_details": {
    "rate": 0.0
  },
  "messages_paged_out": 0,
  "messages_persistent": 0,
  "messages_ram": 0,
  "messages_ready": 0,
  "messages_ready_details": {
    "rate": 0.0
  },
  "messages_ready_ram": 0,
  "messages_unacknowledged": 0,
  "messages_unacknowledged_details": {
    "rate": 0.0
  },
  "messages_unacknowledged_ram": 0,
  "name": "cq.server.12349",
  "node": "rabbit@sunnyside",
  "operator_policy": null,
  "policy": null,
  "reductions": 11491,
  "reductions_details": {
    "rate": 0.0
  },
  "single_active_consumer_tag": null,
  "state": "running",
  "storage_version": 2,
  "type": "classic",
  "vhost": "/"
}

In other words, with some optional arguments, the exception is thrown. This argument does not have
a default and can be configured via a policy, while the one used in your example does have a default.

[michaelklishin]

The following three tests demonstrate a difference in behavior for x-consumer-timeout:

  context "when queue is declared with a mismatching x-max-length" do
    it "raises an exception" do
      ch   = connection.create_channel
      cleanup_ch = connection.create_channel

      name = "bunny.tests.low-level.queues.proprty-equivalence.x-args.x-max-length"
      cleanup_ch.queue_delete(name)

      q = ch.queue_declare(name, type: "classic", durable: true, arguments: {"x-max-length": 10})
      expect do
        ch.queue_declare(name, type: "classic", durable: true, arguments: {"x-max-length": 20})
      end.to raise_error(Bunny::PreconditionFailed)

      expect(ch).to be_closed

      cleanup_ch = connection.create_channel
      cleanup_ch.queue_delete(name)
      cleanup_ch.close
    end
  end

  context "when queue is declared with a mismatching x-max-bytes" do
    it "raises an exception" do
      ch   = connection.create_channel
      cleanup_ch = connection.create_channel

      name = "bunny.tests.low-level.queues.proprty-equivalence.x-args.x-max-length-bytes"
      cleanup_ch.queue_delete(name)

      q = ch.queue_declare(name, type: "classic", durable: true, arguments: {"x-max-length-bytes": 1000000})
      expect do
        ch.queue_declare(name, type: "classic", durable: true, arguments: {"x-max-length-bytes": 99000000})
      end.to raise_error(Bunny::PreconditionFailed)

      expect(ch).to be_closed

      cleanup_ch = connection.create_channel
      cleanup_ch.queue_delete(name)
      cleanup_ch.close
    end
  end

  context "when queue is declared with a mismatching x-consumer-timeout" do
    it "raises an exception" do
      ch   = connection.create_channel
      name = "bunny.tests.low-level.queues.proprty-equivalence.x-args.x-consumer-timeout"

      q = ch.queue_declare(name, type: "classic", durable: true, arguments: {"x-consumer-timeout": 50000})
      # expect do
      #   ch.queue_declare(name, type: "classic", durable: true, arguments: {"x-consumer-timeout": 987000})
      # end.to raise_error(Bunny::PreconditionFailed)
      ch.queue_declare(name, type: "classic", durable: true, arguments: {"x-consumer-timeout": 987000})

      # expect(ch).to be_closed

      cleanup_ch = connection.create_channel
      cleanup_ch.queue_delete(name)
      cleanup_ch.close
    end
  end

In other words, x-max-length-bytes, x-max-length are verified differently from x-consumer-timeout.

This behavior is consistent using following RabbitMQ versions:

• 4.0.2
• 3.13.6
• 3.13.0
• 3.12.14
• 3.12.0

Given that consumer-timeout is best set via a policy, I suspect that virtually no one has run into this. Using a policy will side step the problem entirely.

I'll ask the rest of the team whether this is something worth spending time on.

[michaelklishin]

With a bit of refactoring to make it easier to add declarative examples:

RSpec.shared_examples "enforces optional x-argument equivalence" do |arg, val1, val2|
  it "raises an exception when optional argument #{arg} values do not match that of the original declaration" do
    queue_name = "bunny.tests.low-level.queues.proprty-equivalence.x-args.#{arg}"

    ch   = connection.create_channel
    cleanup_ch = connection.create_channel
    cleanup_ch.queue_delete(queue_name)

    q = ch.queue_declare(queue_name, type: "classic", durable: true, arguments: {arg => val1})
    expect do
      ch.queue_declare(queue_name, type: "classic", durable: true, arguments: {arg => val2})
    end.to raise_error(Bunny::PreconditionFailed)

    expect(ch).to be_closed

    cleanup_ch.queue_delete(queue_name)
    cleanup_ch.close
  end
end

include_examples "enforces optional x-argument equivalence", "x-max-length", 100, 200
include_examples "enforces optional x-argument equivalence", "x-max-length-bytes", 1000000, 99900000
include_examples "enforces optional x-argument equivalence", "x-expires", 2200000, 5500000
include_examples "enforces optional x-argument equivalence", "x-message-ttl", 3000, 5000


RSpec.shared_examples "leniently verifies optional x-argument equivalence" do |arg, val1, val2|
  it "DOES NOT raise an exception when optional argument #{arg} values do not match that of the original declaration" do
    queue_name = "bunny.tests.low-level.queues.proprty-equivalence.x-args.#{arg}"

    ch   = connection.create_channel
    cleanup_ch = connection.create_channel
    cleanup_ch.queue_delete(queue_name)

    q = ch.queue_declare(queue_name, type: "classic", durable: true, arguments: {arg => val1})
    # no exception raised
    ch.queue_declare(queue_name, type: "classic", durable: true, arguments: {arg => val2})

    cleanup_ch.queue_delete(queue_name)
    cleanup_ch.close
  end
end

include_examples "leniently verifies optional x-argument equivalence", "x-consumer-timeout", 10_000, 20_000
include_examples "leniently verifies optional x-argument equivalence", "x-alternate-exchange", "amq.fanout", "amq.topic"

You can see that x-consumer-timeout and Alternate Exchange-related keys are simply ignored upon redeclaration without an exception. This is not just one field.

Four common x-arguments that do not have defaults are verified for equivalence.

[qraynaud]

Exaggerations will not convince me this is something that has to change. If this behaviour was present in 3.11, 3.12, 3.13 and now 4.0, this is not obvious whether changing it would be a net gain.

I'm sorry if I came across this way. I do not know every aspect of RabbitMQ to affirm it would be. I was just saying that the way it works now is counterintuitive and not consistent.

If no one else has noticed in years, it's very difficult to argue that "there is absolutely no way to configure a consumer correctly." There is: don't set optional arguments except for queue type in the apps, use policies.

I feel like my point is missed here. I was clearly only talking about the AMQP 0-9-1 implementation side of things. Right now we do not handle optional arguments using policies. We do know it is the recommended way of doing it though. But we never found it a convenient way of handling those. Too prone to misconfiguration. And too hard to sync between all our environments (we literally have hundreds since each dev has at least one, sometimes multiple, docker envs with its own little RabbitMQ).

What we want to target as much a possible are consumers that fails at startup if a queue is misconfigured for any reason and I don’t think it is achievable using policies. Sadly policies cannot be checked using AMQP. Or if they can I never found how.

In other words, with some optional arguments, the exception is thrown. This argument does not have a default and can be configured via a policy, while the one used in your example does have a default.

Yeah. I should have been more accurate saying it was doing what I think is the proper things on some args but not all of them. I want to add that custom args that have no meaning to RabbitMQ (eg "x-mycompany-arg") do not throw either though they also do not have defaults (obviously).

To rephrase more accurately what I think RabbitMQ should do is: be consistent and predictible. Like I said from the beginning, I don't think silently refusing to do a change that has been requested is predictible behavior. I think failing explicitely in each and every instance would be much better.

I would even argue that if a policy sets a specific argument on a queue and someone ask for this same argument on it, it would be even more awesome to check that the argument value match the policy and to not add it. It would provide a way for consumers to check policies at startup and it would allow us to really consider using them. Which we would love to, it's always hard for us to ignore an official recommendation of a team that knows its product, when we do it, it's only because we find it very impractical and/or problematic to follow. And for us right now we have 2 main issues with policies:

• we can’t check they where properly applied at runtime using AMQP 0-9-1
• we did not find a convenient way of maintaining them across hundreds of envs (some docker and some on CloudAMQP).

I hope I clarified things. Sorry for coming across as exagerrating or maybe even unpleasant. I'm just passionate about things and sometimes I might be going a little overboard. But nothing personal here. I'm not even angry at all. Just trying to get my point across. And it's hard to do sometimes in writing. Most of all since English is not my native language.

And I'd like to thank you for the time you are giving this matter anyway. That's awesome of you!

[michaelklishin]

@qraynaud what would be even more awesome than this assorted set of ideas? If users actually read the docs and followed the recommendations.

Then instead of bastardizing RabbitMQ here and there because someone is N years late we would not spend a single minute on this discussion, or deciding what pixie dust magic may or may not cause more confusion for others than it solves for one specific user who does not use policies while a large majority of users do.

[michaelklishin]

we did not find a convenient way of maintaining them across hundreds of envs

Yes, you do. It is called policies. This is what the docs recommend. They have been around since 2011 or so.

They are trivial to transfer between clusters. They work the same way for every client. They work even when N different protocols are used.

Previously I have had some interest in covering these two groups of keys and investigating roughly what would it take but now that it's turned out to be a request to bastardize RabbitMQ some more because you refuse to use policies, all that interest is gone.