Potential race condition with 'x-consistent-hash' exchanges

[a1ezzz]

Hello, I haven't received any feedback on my post in google groups (https://groups.google.com/g/rabbitmq-users/c/sBVo61hHlWg). I hope someone will help me with my issue here =)

I've found error that looks like a race condition error. I don't know how to reproduce this bug, but still. First of all I use RabbitMQ 3.8.9 (Erlang 23.1.5) as cluster of five docker containers. I have several 'x-consistent-hash' exchanges and some time some of them begin to drop messages. In time of error following messages appear in logs:

2021-03-22 18:30:51.778 [warning] <0.25225.261> Bucket 7 not found

I've done some research and executed the command:

 rabbitmqctl eval 'rabbit_exchange_type_consistent_hash:ring_state(<<"/">>, <<"Exch1ConsistentHash">>).'

(Exch1ConsistentHash is a durable 'x-consistent-hash' exchange that has a single consumer with routing key 10)

it shows me this:

{ok,{chx_hash_ring,{resource,<<"/">>,exchange,<<"Exch1ConsistentHash">>},
                  #{10 =>
                        {resource,<<"/">>,queue,
                                  <<"aioamqp.gen-9183a64b-85fd-49e7-b648-f759e9ec2eed">>},
                    11 =>
                        {resource,<<"/">>,queue,
                                  <<"aioamqp.gen-9183a64b-85fd-49e7-b648-f759e9ec2eed">>},
                    12 =>
                        {resource,<<"/">>,queue,
                                  <<"aioamqp.gen-9183a64b-85fd-49e7-b648-f759e9ec2eed">>},
                    13 =>
                        {resource,<<"/">>,queue,
                                  <<"aioamqp.gen-9183a64b-85fd-49e7-b648-f759e9ec2eed">>},
                    14 =>
                        {resource,<<"/">>,queue,
                                  <<"aioamqp.gen-9183a64b-85fd-49e7-b648-f759e9ec2eed">>},
                    15 =>
                        {resource,<<"/">>,queue,
                                  <<"aioamqp.gen-9183a64b-85fd-49e7-b648-f759e9ec2eed">>},
                    16 =>
                        {resource,<<"/">>,queue,
                                  <<"aioamqp.gen-9183a64b-85fd-49e7-b648-f759e9ec2eed">>},
                    17 =>
                        {resource,<<"/">>,queue,
                                  <<"aioamqp.gen-9183a64b-85fd-49e7-b648-f759e9ec2eed">>},
                    18 =>
                        {resource,<<"/">>,queue,
                                  <<"aioamqp.gen-9183a64b-85fd-49e7-b648-f759e9ec2eed">>},
                    19 =>
                        {resource,<<"/">>,queue,
                                  <<"aioamqp.gen-9183a64b-85fd-49e7-b648-f759e9ec2eed">>}},
                  20}}

I don't know Erlang, but tell me please whether I'm wrong or not:

1. According to this https://github.com/rabbitmq/rabbitmq-server/blob/master/deps/rabbitmq_consistent_hash_exchange/src/rabbit_exchange_type_consistent_hash.erl
   the "SelectedBucket" value should always be between 0 and a size of a map (which in my case is 10)
2. The last integer value for chx_hash_ring is a next number that a next queue will use (in my case is 20):
   ...

   <<"aioamqp.gen-9183a64b-85fd-49e7-b648-f759e9ec2eed">>}},
   20}}

If this true, then there is a race condition somewhere.

Thank you

[michaelklishin]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(

[michaelklishin]

The ring is updated when bindings change. If this happens on a separate connection or channel right before or in the middle of a routing operation on another, this can affect publishing. This is generally true for all exchange types but this one Is stateful.

There were no major changes to this plugin since rabbitmq/rabbitmq-consistent-hash-exchange#37 except for one thing: rabbitmq/rabbitmq-consistent-hash-exchange#46, which adapts to binding recovery changes in RabbitMQ core.

Please upgrade to 3.8.14 or 3.8.15 which is expected to come out next week.

[michaelklishin]

I've noticed that one link ended up being broken. It should be correct now.

[michaelklishin]

We are not aware of any single channel cases where this plugin can end up with an inconsistent binding state. If you have a way to reproduce against 3.8.14, please share a repo on GitHub.

The plugin could be more forgiving and simply ignore the cases when the ring concurrently changes but not everyone would agree this is a good idea.

[hanej]

We just hit this issue today. The only way to fix it was to bring down the entire cluster and start them back again, one node at a time. We are on version 3.8.14.

[hanej]

We hit this issue again in 2 different clusters in different data centers. Anything we can do to help identify the cause, @michaelklishin?

[hanej]

We've been hitting this on a very consistent basis and have since switched away from x-consistent-hash to x-random. We couldn't afford losing messages or the corruption of the vhost when this problem occurred.

[Greatsamps]

We are on 3.8.16 and have been having issue with this as well. As others have said, the only solution is to down the entire cluster and start back up 1 by 1.

We can reproduce this quite easily in our own environment, but it would not be possible to share a repo that does this.

We have a 3-node cluster with about 5,000 publishers sending message to a consistent-hash exchange. On there we have 24 consumers processing these messages.

We can reproduce this by taking one of the nodes offline. even though the consumers who were on that node reconnect to other nodes, the hashing gets messed up and we start dropping messages. Even if the missing node comes back, its still not working.

Only solution is to down all nodes, then start back up 1 by 1.

For now we are running on a single node which we have put on a clustered VM, this is working fine and whilst we would have a couple of mins of outage if there is a problem with it, its better than having to manually reset the cluster.

As we can reproduce it quite easily, i am happy to do this next weekend and collect any data you may want, or even provide a screen share access if helpful.

[Greatsamps]

Just to add here. We tried a test setup with 3 node cluster, hashed-exchange and 24 consumers. We could not reproduce the setup, so it leads me to think its something to do with the scale of the messages being published to it.

[Alan-R]

We are currently running 3.9.10, and are seeing an issue which seems to be identical to this issue. We publish a message without error, and it never shows up. The logs show "bucket 1 not found". Rabbit had been up and stable quite a while before clients connected to it.

[Alan-R]

And clients had been connected a while (a few minutes) before we started publishing messages.

[Alan-R]

This appears to only be associated with exchanges which have persistent volume claims and durable messages. Our other exchanges seem to be working well.

[Alan-R]

For us, it's very low volume work, a 3-node cluster. Probably no more than a few hundred messages.

[michaelklishin]

There were changes to the consistent hashing exchange after 3.9.10, for example, #5124 (#3386) in 3.9.21.

3.9 is out of support so unless the issue can be reliably reproduced against 3.11.13, we will leave it to 3.9 users to investigate. Note that there almost certainly will be no more 3.9 releases.

[Alan-R]

OK. I'll see if I can get us upgraded and see if this fixes the problem. Thanks @michaelklishin!

[a1ezzz]

Hello. May be just for a history, or may future release will test it. The time we spot this issue we found the following code, that always fails

`
import pika
import random

host = ""
port = 5672
username = ""
password = ""

connection = pika.BlockingConnection(pika.ConnectionParameters(
host=host,
port=port,
credentials=pika.credentials.PlainCredentials(username=username, password=password)
))
channel = connection.channel()

exchange_name = 'TestExchange'
channel.exchange_declare(exchange=exchange_name,
exchange_type='x-consistent-hash')

queue_name = 'TestQueue'
channel.queue_declare(queue=queue_name)

channel.queue_bind(exchange=exchange_name, queue=queue_name, routing_key='10')

channel.queue_bind(exchange=exchange_name, queue=queue_name, routing_key='20')

channel.queue_delete(queue=queue_name)
`

After execution of this code a ring became corrupted:
rabbitmq@71e91092c35b:/$ rabbitmqctl eval 'rabbit_exchange_type_consistent_hash:ring_state(<<"/">>, <<"TestExchange">>).' RABBITMQ_ERLANG_COOKIE env variable support is deprecated and will be REMOVED in a future version. Use the $HOME/.erlang.cookie file or the --erlang-cookie switch instead. {ok,{chx_hash_ring,{resource,<<"/">>,exchange,<<"TestExchange">>},#{},10}}

[mkuratczyk]

Thanks a lot for the reproduction steps. I've created an issue for this: #8537

[mkuratczyk]

@a1ezzz what version do you see this problem on?

[a1ezzz]

We use outdated and unsupported version -- 'RabbitMQ 3.8.9'; the code is at least for a history, or may be someone with this code will check a newer version

[mkuratczyk]

Ok, then I must have got confused when I thought I reproduced the problem. As far as I can tell, this issue has been fixed a year ago: 878f369. I've been playing with the latest version and can't reproduce.

[a1ezzz]

Thanks! Then we will plan for update =)