Impact of network latency

[jingxu2021]

Describe the bug

We have 6 nodes across internet, network latency is up to 65ms, sometimes 100ms. Looks like this impacts raft significantly. Sometimes raft forms two clusters. Just wondering if my observation is true? Is there anything I can do to mitigate this impact?
Thanks a lot!

Reproduction steps

1. this can be simulated in lab by adding latency to network interface like eth0
2. in lab we have 4 nodes, and this impact was observed, sometimes took 6 minutes to reach a good status.

...

Expected behavior

Looking for a mitigation.

Additional context

No response

[michaelklishin]

Ra and Raft in general were not designed for inter-node latencies of 100 ms.

Ra's failure detector defaults were not picked for that kind of latencies, so false positives are quite likely.

When deploying to WAN networks with that kind of latency, use something else. Or use sites with under 5ms latency, those are known to work fine with Ra and the systems build on top of it.