Upgrading clusters managed by K8S Cluster Operator

[schlopa]

Hi everyone,

I've got a problem upgrading my cluster.
I am currently running rabbitmq-cluster-operator:1.8.1 with two nodes on rabbitmq:3.8.19-management-alpine at the moment.

My goal is to update to the newest version cluster-operator:2.3.0 and nodes on 3.12.

From what I understand, I probably should update step by step to avoid a downtime of the cluster. And here is my problem.

When I am going to update the cluster-operator to the next version 1.9.0 it will cause a restart of my nodes. So far so good. But this restart is not a rolling update or anything, the cluster is unavailable until all nodes are back again. The second one starts terminating when the first one is not ready.

Whats my mistake here?

Regards
Patrick

[mkuratczyk]

From https://www.rabbitmq.com/clustering.html#node-count

Two node clusters are highly recommended against since it's impossible for cluster nodes to identify a majority and form a consensus in case of connectivity loss. 

[schlopa]

Thanks for your quick response!
....yes, that did it....
For what ever reason we had a two nodes environment locally.