The etcd cluster is unavailable.

[Black-max12138]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via [email protected].
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

In a 3-node etcd cluster, when one node's etcd encounters issues and cannot provide service, the other two nodes also fail to provide service. Using the etcdctl tool to retrieve data results in continuous timeout failures.
Partial error logs are as follows:

What did you expect to happen?

For an etcd cluster with a scale of 3 nodes, it should be able to continue providing normal services even when one etcd node fails.

How can we reproduce it (as minimally and precisely as possible)?

For an etcd cluster with a 3-node scale, stop one etcd node.

Anything else we need to know?

No response

Etcd version (please run commands below)

$ etcd --version
# paste output here

$ etcdctl version
# paste output here

3.5.11

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

{"level":"warn","ts":"2025-07-18T10:07:06.901656Z","caller":"rafthttp/http.go:413","msg":"failed to find remote peer in cluster","local-member-id":"9db5890b6392f376","remote-peer-id-stream-handler":"9db5890b6392f376","remote-peer-id-from":"9db5890b6392f376","cluster-id":"df0158f521ff1360"}

{"level":"info","ts":"2025-07-19T06:21:09.706405Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"9db5890b6392f376 is starting a new election at term 2"}
{"level":"info","ts":"2025-07-19T06:21:09.706427Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"9db5890b6392f376 became pre-candidate at term 2"}
{"level":"info","ts":"2025-07-19T06:21:09.706475Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"9db5890b6392f376 received MsgPreVoteResp from 9db5890b6392f376 at term 2"}
{"level":"info","ts":"2025-07-19T06:21:09.706549Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"9db5890b6392f376 [logterm: 2, index: 465746] sent MsgPreVote request to 5f23d73292959784 at term 2"}
{"level":"info","ts":"2025-07-19T06:21:09.70656Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"9db5890b6392f376 [logterm: 2, index: 465746] sent MsgPreVote request to 9d8e4977362dd913 at term 2"}
{"level":"warn","ts":"2025-07-19T06:21:09.706833Z","caller":"etcdserver/server.go:1015","msg":"rejected Raft message to mismatch member","local-member-id":"9db5890b6392f376","mismatch-member-id":"9d8e4977362dd913"}
{"level":"warn","ts":"2025-07-19T06:21:09.706852Z","caller":"rafthttp/http.go:145","msg":"failed to process Raft message","local-member-id":"9db5890b6392f376","error":"cannot process message to mismatch member"}

{"level":"info","ts":"2025-07-19T06:23:03.452392Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"9d8e4977362dd913 is starting a new election at term 2"}
{"level":"info","ts":"2025-07-19T06:23:03.452434Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"9d8e4977362dd913 became pre-candidate at term 2"}
{"level":"info","ts":"2025-07-19T06:23:03.452469Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"9d8e4977362dd913 received MsgPreVoteResp from 9d8e4977362dd913 at term 2"}
{"level":"info","ts":"2025-07-19T06:23:03.452484Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"9d8e4977362dd913 [logterm: 2, index: 465746] sent MsgPreVote request to 5f23d73292959784 at term 2"}
{"level":"info","ts":"2025-07-19T06:23:03.452491Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"9d8e4977362dd913 [logterm: 2, index: 465746] sent MsgPreVote request to 9db5890b6392f376 at term 2"}
{"level":"warn","ts":"2025-07-19T06:23:03.452929Z","caller":"etcdserver/server.go:1015","msg":"rejected Raft message to mismatch member","local-member-id":"9d8e4977362dd913","mismatch-member-id":"9db5890b6392f376"}
{"level":"warn","ts":"2025-07-19T06:23:03.452953Z","caller":"rafthttp/http.go:145","msg":"failed to process Raft message","local-member-id":"9d8e4977362dd913","error":"cannot process message to mismatch member"}

[tjungblu]

Using the etcdctl tool to retrieve data results in continuous timeout failures.

that depends on the configuration of etcdctl, did you configure all three endpoints?

Please add all debug information required above, commands and the logs of all members, it is not possible to reproduce anything with this information.

[ahrtr]

It seems the members are not belonging to the same cluster.

pls consider generating a reporting following https://github.com/ahrtr/etcd-diagnosis?tab=readme-ov-file#example-1-online-analysis