RabbitMQ upgrade from 3.9.29 to 3.10.24 failed and got queue index directory not found error

[CassieZAHNG]

Hey,

I am trying to upgrade RabbitMQ in AKS clusters (three pods per cluster) with rolling upgrade strategy. The old container image is rabbitmq:3.9-management, now try to upgrade to rabbitmq:3.10-management.

Have checked all required feature flags are enabled as required from RabbitMQ 3.10.0. All clusters using the same rabbit configurations, but 2 clusters successfully upgraded while 2 got failed, not sure what could make it different.

Attached logs for failed cluster, it failed at step of moving messages to per-vhost message store:

For the successfully upgrade one, it says no durable queue is found so it skips message store migration:

Have done additional testing, when I tried to downgrade back to 3.9.29, all pods running again, but if re-upgrade pods then it will fail with the same error.

From my understanding, RabbitMQ will take care of migrating durable queue when upgrading, does not require additional steps from users?

Please let me know if require further information, super appreciate your help!

[michaelklishin]

Somehow this node ended up without an index directory for one of its queues and a (3.7 era) migration function assumes that cannot be the case. We cannot tell why that happened.

You can delete the directory in question to let the node proceed.

RabbitMQ does not support downgrades.

[mkuratczyk]

I don't recall any massive changes, but you'd need to go through the changelog/commits yourself. It's been more than a year since 3.10 was released.

[CassieZAHNG]

yep, I have checked the release notes, but have not found index issue same as the one we met. Will continue doing some trials and errors, thanks for the help!

[CassieZAHNG]

Hey @mkuratczyk, got a follow up question regarding previous issue:

We are trying to upgrade from 3.9 to 3.10. And our msg stores are already in "/var/lib/rabbitmq/mnesia/rabbit@rabbit-hostname/msg_stores/vhosts" format. So ideally shouldn't the "move_messages_to_vhost_store" migration function returned nothing to migrate and skipped migration? Is that understanding correct?

[mkuratczyk]

Yes, I believe that's correct. You can follow the code here
https://github.com/rabbitmq/rabbitmq-server/blob/v3.10.x/deps/rabbit/src/rabbit_variable_queue.erl#L3398-L3444

[CassieZAHNG]

Checked the above link only pre-3.7.0 version required to msg_store migrate, not sure why 3.10 still got the issue. Now upgrade to 3.12, the issues are gone. Thanks for the prompt help!