Ra 1.2: leader operations fail with a timeout

[jingxu2021]

Hi,

I have 2 nodes and 3 clusters running on them. Node 1 is the leader in all 3 clusters. I stopped the node 1, then node 2 became leaders for 2 clusters, but node 2 in the last cluster is in state 'pre_vote', and seems stuck there.
If I run ra:members(cluster_3), I got {timeout, cluster_3}.
Then I tried everything I could imagine, it just stuck there. I tried, restart_server, stop_server/restart_server, start_or_restart_cluster. I always got error already_started.
I will try delete_cluster/start_cluster. Update: delete_cluster got {error,{no_more_servers_to_try,[{timeout,{cluster_3, node_2}}]}}
Could you please suggest how could I recover from this situation? I'm on v1.1.2, I know it's pretty old version, but that's what I have.
Thank you so much.
Jing

[michaelklishin]

This means the cluster does not have an elected leader or quorum online. Or never had them.

[michaelklishin]

We cannot suggest much with this amount of information. Consider gathering node logs and state.

Ra 1.2.x is several minors and that be major behind.

[michaelklishin]

Without any code or logs, #179, #251, #264 look potentially relevant.

#179 mentions how to enable logging, including debug logging.

[jingxu2021]

Thanks so much for all these information. I will enable logging and try my best to provide more information. I appreciate your help.
If I could get out of this stuck situation, that's fine for me. But I couldn't find any API which can help so far.

[jingxu2021]

Here is what I found:
1, if both nodes are in 'follower' state, trigger_election() can recover the cluster
2, if a node is in 'pre_vote' state, stop_server() can change its state to 'follower', then trigger_election() can recover the cluster.
Thanks.

[michaelklishin]

There were many changes around member state, and some upcoming, so I don't think this is relevant any more in the 2.10.x era of Ra.