Vhost corruption when recreated immediately after deletion

[inikulshin]

RabbitMQ version 3.10.8, Erlang 25.0.4

For historical reasons in our tests we reuse vhosts. The simplest way to prepare empty vhost (just like after creation) for the next test is to delete it and create again (+create user permissions again).

However in this RabbitMQ version (previous one was 3.8.27) this approach leads to occasional vhost corruptions (after some vhost creation that returns success, vhost is irresponsive and even can't be deleted).

In our case during vhost recreation, there can be still client connections to vhost, and clients will also retry to reconnect after the connections are closed during vhost deletion. Maybe this is a root cause, despite my expectation from delete/create vhost operations to support it.

Is there a safe (and also faster than stop_app+start_app, which I'm sure will resolve) workaround for this? Or completely different approach for resetting single vhost?
Was it fixed in later RabbitMQ versions?

Maybe rabbitmqctl set_vhost_limits -p vhost_name '{"max-connections": 0}' before deletion?

P.S. Maybe #6676 is related?

Thanks.

[lukebakken]

Try putting the node into maintenance mode before reusing a vhost - https://www.rabbitmq.com/rabbitmq-upgrade.8.html

rabbitmq-upgrade drain

Let us know if that works.

[inikulshin]

@lukebakken thanks, but when exactly should we run rabbitmq-upgrade drain?

I mean, what should be the order of

• rabbitmq-upgrade drain
• DELETE /api/vhosts/name
• PUT /api/vhosts/name
• rabbitmq-upgrade revive

?
Can I use management HTTP API in maintenance mode?

[michaelklishin]

There is a doc section on Maintenance mode.

1. Drain the node, it will drop all connections and refuse to accept new ones
2. Do any maintenance you need to the node
3. Revive it

The case where clients try to use a virtual host before it is created is not really maintenance
but putting all nodes under maintenance before virtual hosts are seeded would help.

Virtual hosts are not meant to be deleted and instantly re-created and potentially used at any point in between. Creating a virtual host and starting it requires a certain amount of setup on all nodes.

Nodes cannot do a cluster-wide check for virtual host startup state for every connection as that
would be really expensive in environments with high connection churn. So you have to accept this as
a limitation of the current design that we do not have plans to replace.

Nothing in this area has changed since 3.8.

[inikulshin]

@michaelklishin thanks.
We are not using any clustering. Do you think that set_vhost_limits -p vhost_name '{"max-connections": 0}' is enough in this case?

When it's set by PUT /api/vhost-limits/my-vhost/max-connections -d '{"value": 0}', followed by DELETE /api/connections/name it is

1. Much faster.
2. Doesn't depend on RabbitMQ installation (which can be inside docker container in our case).
3. Doesn't affect other vhosts.

Or vhost-limits are deleted before vhost deletion, similar to user permissions at vhost? Maybe, user-limits?

[michaelklishin]

Setting a virtual host limit to no more than 0 connections would block all new connections but won't drop any existing ones.

[michaelklishin]

So if you use the limit and close all conections via the HTTP API or CLI tools, that would give you what you want, as far as I can tell.