Converting between Cqv2 and Cqv1 consumes a lot of memory when alarm is triggered.

[johanrhodin]

I noticed that if an memory or disk alarm triggers while converting queues the memory usage for the process grows a lot.

Steps to reproduce (RabbitMQ 3.10.1, Erlang 24.1.7)

Create a queue policy to use cqv2:

#!/bin/bash

VHOST=rwgvqgbl
HTTPS_URL="https://rwgvqgbl:[email protected]"

curl -i -X PUT -H 'Content-Type: application/json' $HTTPS_URL/api/policies/$VHOST/qnew2/ -d '{"pattern":".", "definition": {"queue-version":2}, "priority":1, "apply-to": "queues"}'

Run perf-test and load in 1M messages:
bin/runjava com.rabbitmq.perf.PerfTest -y0 -p -u "smallish" -s 0 --id "smallish" -ad false -f persistent -h $TARGET -pmessages 1000000

Once done remove the queue policy and when it starts logging:
2022-05-11 20:18:40.422355+00:00 [info] <0.2090.0> Converting running queue smallish in vhost rwgvqgbl from v2 to v1
set the disk alarm:
sudo rabbitmqctl set_disk_free_limit 1111G

Now the converter will use much more memory than without the disk alarm:

For information this is what it looks like for the process:

[lhoguin]

I suspect the file_handle_cache module to be responsible since it's the only part of the code worrying about alarms. But I am not sure why. I will look into it.

[lhoguin]

I do not suspect file_handle_cache anymore. I do not see a difference between with or without the disk alarm.

The queue is a default classic queue and has a million messages in memory. Those messages are held in its state as well as on disk since they are persistent. What the conversion does currently is go over the messages in memory first and then process on-disk messages. Because of how Erlang works, the original messages in memory are kept in memory until the conversion is complete (they are in the State of the caller and eventually that State gets dropped) and so they get duplicated for a while. For me the memory of the process goes from around 600MB to around 1200MB as a result. The message content does not get duplicated for >= 64 bytes messages, but the message metadata x 1 million can still be a fair amount of memory usage.

The same does not happen to lazy queues since they are all on disk and we have finer grained control over what gets into memory (we GC every few messages).

I suppose it would be possible to skip messages that are already on disk but that can get complicated when you have a mix of persistent and non-persistent messages in the queue.

A likely workaround would be to first convert the queue to lazy (which flushes all messages to disk), then do the v2->v1 conversion, then remove the lazy mode (which does not pull messages back to memory immediately).