Cluster member doesn't join back after its data was reset #20586

teapot9 · 2025-08-30T20:43:03Z

teapot9
Aug 30, 2025

I'm working on setting up an etcd cluster using cluster discovery (with an existing etcd cluster).

I tested the following scenario: the server etcd-testing3 had to be reinstalled due to failure.

After server reset, the node etcd-testing3 was started with the same configuration used before:

/usr/local/bin/etcd --name etcd-testing3 --data-dir=/etcd-data --advertise-client-urls http://etcd-testing3:2379 --listen-client-urls http://0.0.0.0:2379 --initial-advertise-peer-urls http://etcd-testing3:2380 --listen-peer-urls http://0.0.0.0:2380 --discovery-token 5a4a785f-c9d9-4b69-8473-b0573c5681e1 --discovery-endpoints http://tasks.etcd-init-cluster:2579 --discovery-insecure-transport --discovery-insecure-skip-tls-verify

When I list the cluster members, node 3 is still listed as member:

+------------------+---------+---------------+---------------------------+---------------------------+------------+
|        ID        | STATUS  |     NAME      |        PEER ADDRS         |       CLIENT ADDRS        | IS LEARNER |
+------------------+---------+---------------+---------------------------+---------------------------+------------+
| 6fa6da62f2b4add9 | started | etcd-testing3 | http://etcd-testing3:2380 | http://etcd-testing3:2379 |      false |
| ea34b7c7249d2743 | started | etcd-testing1 | http://etcd-testing1:2380 | http://etcd-testing1:2379 |      false |
| f99c4a5e43d12e88 | started | etcd-testing2 | http://etcd-testing2:2380 | http://etcd-testing2:2379 |      false |
+------------------+---------+---------------+---------------------------+---------------------------+------------+

However, it looks like the node 3 has started a cluster and a life on its own, as the node 1 & 2 says they can't communicate with node 3 and they failed to find remote peer in cluster.

The node 3 has logged the following:

{"level":"warn","ts":"2025-08-30T19:10:12.681342Z","caller":"embed/config.go:1209","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2025-08-30T19:10:12.681808Z","caller":"etcdmain/etcd.go:64","msg":"Running: ","args":["/usr/local/bin/etcd","--name","etcd-testing3","--data-dir=/etcd-data","--advertise-client-urls","http://etcd-testing3:2379","--listen-client-urls","http://0.0.0.0:2379","--initial-advertise-peer-urls","http://etcd-testing3:2380","--listen-peer-urls","http://0.0.0.0:2380","--discovery-token","5a4a785f-c9d9-4b69-8473-b0573c5681e1","--discovery-endpoints","http://tasks.etcd-init-cluster:2579","--discovery-insecure-transport","--discovery-insecure-skip-tls-verify"]}
{"level":"info","ts":"2025-08-30T19:10:12.682017Z","caller":"etcdmain/etcd.go:124","msg":"Initialize and start etcd server","data-dir":"/etcd-data","dir-type":"empty"}
{"level":"warn","ts":"2025-08-30T19:10:12.682061Z","caller":"embed/config.go:1209","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2025-08-30T19:10:12.682076Z","caller":"embed/etcd.go:138","msg":"configuring peer listeners","listen-peer-urls":["http://0.0.0.0:2380"]}
{"level":"info","ts":"2025-08-30T19:10:12.682499Z","caller":"embed/etcd.go:146","msg":"configuring client listeners","listen-client-urls":["http://0.0.0.0:2379"]}
{"level":"info","ts":"2025-08-30T19:10:12.682682Z","caller":"embed/etcd.go:323","msg":"starting an etcd server","etcd-version":"3.6.4","git-sha":"5400cdc","go-version":"go1.23.11","go-os":"linux","go-arch":"amd64","max-cpu-set":1,"max-cpu-available":1,"member-initialized":false,"name":"etcd-testing3","data-dir":"/etcd-data","wal-dir":"","wal-dir-dedicated":"","member-dir":"/etcd-data/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":10000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://etcd-testing3:2380"],"listen-peer-urls":["http://0.0.0.0:2380"],"advertise-client-urls":["http://etcd-testing3:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[],"experimental-local-address":"","cors":["*"],"host-whitelist":["*"],"initial-cluster":"etcd-testing3=http://etcd-testing3:2380","initial-cluster-state":"new","initial-cluster-token":"5a4a785f-c9d9-4b69-8473-b0573c5681e1","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"feature-gates":"","initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-interval":"1m0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","discovery-token":"5a4a785f-c9d9-4b69-8473-b0573c5681e1","discovery-endpoints":"http://tasks.etcd-init-cluster:2579","discovery-dial-timeout":"2s","discovery-request-timeout":"5s","discovery-keepalive-time":"2s","discovery-keepalive-timeout":"6s","discovery-insecure-transport":true,"discovery-insecure-skip-tls-verify":true,"discovery-cert":"","discovery-key":"","discovery-cacert":"","discovery-user":"","downgrade-check-interval":"5s","max-learners":1,"v2-deprecation":"write-only"}
{"level":"warn","ts":"2025-08-30T19:10:12.682843Z","caller":"fileutil/fileutil.go:56","msg":"check file permission","error":"directory \"/etcd-data\" exist, but the permission is \"drwxr-xr-x\". The recommended permission is \"-rwx------\" to prevent possible unprivileged access to the data"}
{"level":"info","ts":"2025-08-30T19:10:12.683353Z","logger":"bbolt","caller":"backend/backend.go:203","msg":"Opening db file (/etcd-data/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: hashmap, ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 10737418240, PageSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0xc000052150}"}
{"level":"info","ts":"2025-08-30T19:10:12.687268Z","logger":"bbolt","caller":"[email protected]/db.go:321","msg":"Opening bbolt db (/etcd-data/member/snap/db) successfully"}
{"level":"info","ts":"2025-08-30T19:10:12.687317Z","caller":"storage/backend.go:80","msg":"opened backend db","path":"/etcd-data/member/snap/db","took":"4.062674ms"}
{"level":"info","ts":"2025-08-30T19:10:12.687349Z","caller":"etcdserver/bootstrap.go:220","msg":"restore consistentIndex","index":0}
{"level":"info","ts":"2025-08-30T19:10:12.687371Z","caller":"etcdserver/bootstrap.go:94","msg":"bootstrapping cluster"}
{"level":"info","ts":"2025-08-30T19:10:12.687485Z","caller":"etcdserver/bootstrap.go:340","msg":"Bootstrapping cluster using v3 discovery."}
2025/08/30 19:10:12 WARNING: [core] Adjusting keepalive ping interval to minimum period of 10s
{"level":"info","ts":"2025-08-30T19:10:12.697068Z","caller":"v3discovery/discovery.go:284","msg":"found peer from discovery service","discovery-token":"5a4a785f-c9d9-4b69-8473-b0573c5681e1","discovery-endpoints":"http://tasks.etcd-init-cluster:2579","memberKey":"/_etcd/registry/5a4a785f-c9d9-4b69-8473-b0573c5681e1/members/ea34b7c7249d2743","memberInfo":"etcd-testing1=http://etcd-testing1:2380"}
{"level":"info","ts":"2025-08-30T19:10:12.697275Z","caller":"v3discovery/discovery.go:284","msg":"found peer from discovery service","discovery-token":"5a4a785f-c9d9-4b69-8473-b0573c5681e1","discovery-endpoints":"http://tasks.etcd-init-cluster:2579","memberKey":"/_etcd/registry/5a4a785f-c9d9-4b69-8473-b0573c5681e1/members/f99c4a5e43d12e88","memberInfo":"etcd-testing2=http://etcd-testing2:2380"}
{"level":"info","ts":"2025-08-30T19:10:12.700211Z","caller":"v3discovery/discovery.go:358","msg":"register member itself successfully","discovery-token":"5a4a785f-c9d9-4b69-8473-b0573c5681e1","discovery-endpoints":"http://tasks.etcd-init-cluster:2579","memberKey":"/_etcd/registry/5a4a785f-c9d9-4b69-8473-b0573c5681e1/members/6fa6da62f2b4add9","memberInfo":"etcd-testing3=http://etcd-testing3:2380"}
{"level":"info","ts":"2025-08-30T19:10:12.702105Z","caller":"v3discovery/discovery.go:284","msg":"found peer from discovery service","discovery-token":"5a4a785f-c9d9-4b69-8473-b0573c5681e1","discovery-endpoints":"http://tasks.etcd-init-cluster:2579","memberKey":"/_etcd/registry/5a4a785f-c9d9-4b69-8473-b0573c5681e1/members/6fa6da62f2b4add9","memberInfo":"etcd-testing3=http://etcd-testing3:2380"}
{"level":"info","ts":"2025-08-30T19:10:12.702155Z","caller":"v3discovery/discovery.go:284","msg":"found peer from discovery service","discovery-token":"5a4a785f-c9d9-4b69-8473-b0573c5681e1","discovery-endpoints":"http://tasks.etcd-init-cluster:2579","memberKey":"/_etcd/registry/5a4a785f-c9d9-4b69-8473-b0573c5681e1/members/ea34b7c7249d2743","memberInfo":"etcd-testing1=http://etcd-testing1:2380"}
{"level":"info","ts":"2025-08-30T19:10:12.702165Z","caller":"v3discovery/discovery.go:284","msg":"found peer from discovery service","discovery-token":"5a4a785f-c9d9-4b69-8473-b0573c5681e1","discovery-endpoints":"http://tasks.etcd-init-cluster:2579","memberKey":"/_etcd/registry/5a4a785f-c9d9-4b69-8473-b0573c5681e1/members/f99c4a5e43d12e88","memberInfo":"etcd-testing2=http://etcd-testing2:2380"}
{"level":"info","ts":"2025-08-30T19:10:12.702184Z","caller":"v3discovery/discovery.go:145","msg":"discovery joined cluster successfully","discovery-token":"5a4a785f-c9d9-4b69-8473-b0573c5681e1","discovery-endpoints":"http://tasks.etcd-init-cluster:2579","cluster":"etcd-testing1=http://etcd-testing1:2380,etcd-testing2=http://etcd-testing2:2380,etcd-testing3=http://etcd-testing3:2380"}
{"level":"info","ts":"2025-08-30T19:10:12.702482Z","caller":"etcdserver/bootstrap.go:101","msg":"bootstrapping storage"}
{"level":"info","ts":"2025-08-30T19:10:12.710731Z","caller":"etcdserver/bootstrap.go:109","msg":"bootstrapping raft"}
{"level":"info","ts":"2025-08-30T19:10:12.711232Z","caller":"etcdserver/bootstrap.go:499","msg":"starting local member","local-member-id":"6fa6da62f2b4add9","cluster-id":"512db839d32901"}
{"level":"info","ts":"2025-08-30T19:10:12.711497Z","caller":"etcdserver/server.go:312","msg":"bootstrap successfully"}
{"level":"info","ts":"2025-08-30T19:10:12.711706Z","logger":"raft","caller":"[email protected]/raft.go:1981","msg":"6fa6da62f2b4add9 switched to configuration voters=()"}
{"level":"info","ts":"2025-08-30T19:10:12.711775Z","logger":"raft","caller":"[email protected]/raft.go:897","msg":"6fa6da62f2b4add9 became follower at term 0"}
{"level":"info","ts":"2025-08-30T19:10:12.711795Z","logger":"raft","caller":"[email protected]/raft.go:493","msg":"newRaft 6fa6da62f2b4add9 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]"}
{"level":"info","ts":"2025-08-30T19:10:12.711829Z","logger":"raft","caller":"[email protected]/raft.go:897","msg":"6fa6da62f2b4add9 became follower at term 1"}
{"level":"info","ts":"2025-08-30T19:10:12.711900Z","logger":"raft","caller":"[email protected]/raft.go:1981","msg":"6fa6da62f2b4add9 switched to configuration voters=(8045357902857547225)"}
{"level":"info","ts":"2025-08-30T19:10:12.711936Z","logger":"raft","caller":"[email protected]/raft.go:1981","msg":"6fa6da62f2b4add9 switched to configuration voters=(8045357902857547225 16876315769604745027)"}
{"level":"info","ts":"2025-08-30T19:10:12.711968Z","logger":"raft","caller":"[email protected]/raft.go:1981","msg":"6fa6da62f2b4add9 switched to configuration voters=(8045357902857547225 16876315769604745027 17986332780536082056)"}
{"level":"warn","ts":"2025-08-30T19:10:12.757211Z","caller":"auth/store.go:1135","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2025-08-30T19:10:12.765086Z","caller":"mvcc/kvstore.go:408","msg":"kvstore restored","current-rev":1}
{"level":"info","ts":"2025-08-30T19:10:12.772159Z","caller":"storage/quota.go:93","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2025-08-30T19:10:12.772231Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"ea34b7c7249d2743"}
{"level":"info","ts":"2025-08-30T19:10:12.772259Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"6fa6da62f2b4add9","remote-peer-id":"ea34b7c7249d2743"}
{"level":"info","ts":"2025-08-30T19:10:12.772338Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"ea34b7c7249d2743"}
{"level":"info","ts":"2025-08-30T19:10:12.772385Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"6fa6da62f2b4add9","remote-peer-id":"ea34b7c7249d2743","remote-peer-urls":["http://etcd-testing1:2380"]}
{"level":"info","ts":"2025-08-30T19:10:12.772400Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"f99c4a5e43d12e88"}
{"level":"info","ts":"2025-08-30T19:10:12.772419Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"6fa6da62f2b4add9","remote-peer-id":"f99c4a5e43d12e88"}
{"level":"info","ts":"2025-08-30T19:10:12.776154Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"f99c4a5e43d12e88"}
{"level":"info","ts":"2025-08-30T19:10:12.776337Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"6fa6da62f2b4add9","remote-peer-id":"f99c4a5e43d12e88","remote-peer-urls":["http://etcd-testing2:2380"]}
{"level":"info","ts":"2025-08-30T19:10:12.776487Z","caller":"etcdserver/server.go:598","msg":"starting etcd server","local-member-id":"6fa6da62f2b4add9","local-server-version":"3.6.4","cluster-version":"to_be_decided"}
{"level":"info","ts":"2025-08-30T19:10:12.777007Z","caller":"embed/etcd.go:292","msg":"now serving peer/client/metrics","local-member-id":"6fa6da62f2b4add9","initial-advertise-peer-urls":["http://etcd-testing3:2380"],"listen-peer-urls":["http://0.0.0.0:2380"],"advertise-client-urls":["http://etcd-testing3:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[]}
{"level":"info","ts":"2025-08-30T19:10:12.777196Z","caller":"rafthttp/stream.go:168","msg":"started stream writer with remote peer","local-member-id":"6fa6da62f2b4add9","remote-peer-id":"ea34b7c7249d2743"}
{"level":"info","ts":"2025-08-30T19:10:12.777318Z","caller":"rafthttp/stream.go:168","msg":"started stream writer with remote peer","local-member-id":"6fa6da62f2b4add9","remote-peer-id":"ea34b7c7249d2743"}
{"level":"info","ts":"2025-08-30T19:10:12.777424Z","caller":"rafthttp/stream.go:394","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"6fa6da62f2b4add9","remote-peer-id":"ea34b7c7249d2743"}
{"level":"info","ts":"2025-08-30T19:10:12.778102Z","caller":"rafthttp/stream.go:394","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"6fa6da62f2b4add9","remote-peer-id":"ea34b7c7249d2743"}
{"level":"info","ts":"2025-08-30T19:10:12.778245Z","caller":"rafthttp/stream.go:168","msg":"started stream writer with remote peer","local-member-id":"6fa6da62f2b4add9","remote-peer-id":"f99c4a5e43d12e88"}
{"level":"info","ts":"2025-08-30T19:10:12.778262Z","caller":"rafthttp/stream.go:168","msg":"started stream writer with remote peer","local-member-id":"6fa6da62f2b4add9","remote-peer-id":"f99c4a5e43d12e88"}
{"level":"info","ts":"2025-08-30T19:10:12.778289Z","caller":"rafthttp/stream.go:394","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"6fa6da62f2b4add9","remote-peer-id":"f99c4a5e43d12e88"}
{"level":"info","ts":"2025-08-30T19:10:12.778683Z","caller":"rafthttp/stream.go:394","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"6fa6da62f2b4add9","remote-peer-id":"f99c4a5e43d12e88"}
{"level":"info","ts":"2025-08-30T19:10:12.780305Z","caller":"etcdserver/server.go:499","msg":"starting initial election tick advance","election-ticks":10}
{"level":"info","ts":"2025-08-30T19:10:12.780385Z","caller":"fileutil/purge.go:49","msg":"started to purge file","dir":"/etcd-data/member/snap","suffix":"snap.db","max":5,"interval":"30s"}
{"level":"info","ts":"2025-08-30T19:10:12.780416Z","caller":"fileutil/purge.go:49","msg":"started to purge file","dir":"/etcd-data/member/snap","suffix":"snap","max":5,"interval":"30s"}
{"level":"info","ts":"2025-08-30T19:10:12.780424Z","caller":"fileutil/purge.go:49","msg":"started to purge file","dir":"/etcd-data/member/wal","suffix":"wal","max":5,"interval":"30s"}
{"level":"info","ts":"2025-08-30T19:10:12.780658Z","caller":"embed/etcd.go:640","msg":"serving peer traffic","address":"[::]:2380"}
{"level":"info","ts":"2025-08-30T19:10:12.780689Z","caller":"embed/etcd.go:611","msg":"cmux::serve","address":"[::]:2380"}
{"level":"info","ts":"2025-08-30T19:10:12.781632Z","logger":"raft","caller":"[email protected]/raft.go:1981","msg":"6fa6da62f2b4add9 switched to configuration voters=(8045357902857547225 16876315769604745027 17986332780536082056)"}
{"level":"info","ts":"2025-08-30T19:10:12.781942Z","caller":"membership/cluster.go:424","msg":"added member","cluster-id":"512db839d32901","local-member-id":"6fa6da62f2b4add9","added-peer-id":"6fa6da62f2b4add9","added-peer-peer-urls":["http://etcd-testing3:2380"],"added-peer-is-learner":false}
{"level":"info","ts":"2025-08-30T19:10:12.782095Z","logger":"raft","caller":"[email protected]/raft.go:1981","msg":"6fa6da62f2b4add9 switched to configuration voters=(8045357902857547225 16876315769604745027 17986332780536082056)"}
{"level":"info","ts":"2025-08-30T19:10:12.782207Z","caller":"membership/cluster.go:424","msg":"added member","cluster-id":"512db839d32901","local-member-id":"6fa6da62f2b4add9","added-peer-id":"ea34b7c7249d2743","added-peer-peer-urls":["http://etcd-testing1:2380"],"added-peer-is-learner":false}
{"level":"info","ts":"2025-08-30T19:10:12.782338Z","logger":"raft","caller":"[email protected]/raft.go:1981","msg":"6fa6da62f2b4add9 switched to configuration voters=(8045357902857547225 16876315769604745027 17986332780536082056)"}
{"level":"info","ts":"2025-08-30T19:10:12.782467Z","caller":"membership/cluster.go:424","msg":"added member","cluster-id":"512db839d32901","local-member-id":"6fa6da62f2b4add9","added-peer-id":"f99c4a5e43d12e88","added-peer-peer-urls":["http://etcd-testing2:2380"],"added-peer-is-learner":false}
{"level":"info","ts":"2025-08-30T19:10:14.312521Z","logger":"raft","caller":"[email protected]/raft.go:988","msg":"6fa6da62f2b4add9 is starting a new election at term 1"}
{"level":"info","ts":"2025-08-30T19:10:14.312627Z","logger":"raft","caller":"[email protected]/raft.go:930","msg":"6fa6da62f2b4add9 became pre-candidate at term 1"}
{"level":"info","ts":"2025-08-30T19:10:14.313068Z","logger":"raft","caller":"[email protected]/raft.go:1064","msg":"6fa6da62f2b4add9 [logterm: 1, index: 3] sent MsgPreVote request to ea34b7c7249d2743 at term 1"}
{"level":"info","ts":"2025-08-30T19:10:14.313208Z","logger":"raft","caller":"[email protected]/raft.go:1064","msg":"6fa6da62f2b4add9 [logterm: 1, index: 3] sent MsgPreVote request to f99c4a5e43d12e88 at term 1"}
{"level":"info","ts":"2025-08-30T19:10:14.313641Z","logger":"raft","caller":"[email protected]/raft.go:1077","msg":"6fa6da62f2b4add9 received MsgPreVoteResp from 6fa6da62f2b4add9 at term 1"}
{"level":"info","ts":"2025-08-30T19:10:14.313912Z","logger":"raft","caller":"[email protected]/raft.go:1693","msg":"6fa6da62f2b4add9 has received 1 MsgPreVoteResp votes and 0 vote rejections"}
{"level":"info","ts":"2025-08-30T19:10:14.319602Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"f99c4a5e43d12e88"}
{"level":"info","ts":"2025-08-30T19:10:14.319930Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"ea34b7c7249d2743"}

But the node 1 & 2 are spamming the following error:

{"level":"warn","ts":"2025-08-30T20:25:34.588341Z","caller":"rafthttp/http.go:415","msg":"failed to find remote peer in cluster","local-member-id":"f99c4a5e43d12e88","remote-peer-id-stream-handler":"f99c4a5e43d12e88","remote-peer-id-from":"6fa6da62f2b4add9","cluster-id":"512db839d32901"}

I also tried to remove the node 3 from the cluster with etcdctl.
However, it couldn't join the cluster as the member 6fa6da62f2b4add9 was "removed from the cluster", so node 1 & 2 refused it.
Resetting the node 3 again did not help as it did not change member ID (it seems the ID is generated from the member name and it's URLs, so it won't change unless I change the member name).

Is there a way to make my server member of the cluster again?

serathius · 2025-09-01T06:49:07Z

serathius
Sep 1, 2025
Maintainer

Purge the member data again and start it with --initial-cluster-state=existing flag

0 replies

teapot9 · 2025-09-01T09:46:31Z

teapot9
Sep 1, 2025
Author

I purged the member by removing the member directory, and restarted it with --initial-cluster-state=existing, it fails to start with the following output:

{"level":"warn","ts":"2025-09-01T09:28:22.593855Z","caller":"embed/config.go:1209","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2025-09-01T09:28:22.594091Z","caller":"etcdmain/etcd.go:64","msg":"Running: ","args":["/usr/local/bin/etcd","--name","etcd-testing3","--data-dir=/etcd-data","--advertise-client-urls","http://etcd-testing3:2379","--listen-client-urls","http://0.0.0.0:2379","--initial-advertise-peer-urls","http://etcd-testing3:2380","--listen-peer-urls","http://0.0.0.0:2380","--discovery-token","5a4a785f-c9d9-4b69-8473-b0573c5681e1","--discovery-endpoints","http://tasks.etcd-init-cluster:2579","--discovery-insecure-transport","--discovery-insecure-skip-tls-verify","--initial-cluster-state=existing"]}
{"level":"info","ts":"2025-09-01T09:28:22.594197Z","caller":"etcdmain/etcd.go:107","msg":"server has already been initialized","data-dir":"/etcd-data","dir-type":"member"}
{"level":"warn","ts":"2025-09-01T09:28:22.594214Z","caller":"embed/config.go:1209","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2025-09-01T09:28:22.594226Z","caller":"embed/etcd.go:138","msg":"configuring peer listeners","listen-peer-urls":["http://0.0.0.0:2380"]}
{"level":"info","ts":"2025-09-01T09:28:22.594454Z","caller":"embed/etcd.go:146","msg":"configuring client listeners","listen-client-urls":["http://0.0.0.0:2379"]}
{"level":"info","ts":"2025-09-01T09:28:22.594515Z","caller":"embed/etcd.go:323","msg":"starting an etcd server","etcd-version":"3.6.4","git-sha":"5400cdc","go-version":"go1.23.11","go-os":"linux","go-arch":"amd64","max-cpu-set":1,"max-cpu-available":1,"member-initialized":false,"name":"etcd-testing3","data-dir":"/etcd-data","wal-dir":"","wal-dir-dedicated":"","member-dir":"/etcd-data/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":10000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://etcd-testing3:2380"],"listen-peer-urls":["http://0.0.0.0:2380"],"advertise-client-urls":["http://etcd-testing3:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[],"experimental-local-address":"","cors":["*"],"host-whitelist":["*"],"initial-cluster":"etcd-testing3=http://etcd-testing3:2380","initial-cluster-state":"existing","initial-cluster-token":"5a4a785f-c9d9-4b69-8473-b0573c5681e1","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"feature-gates":"","initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-interval":"1m0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","discovery-token":"5a4a785f-c9d9-4b69-8473-b0573c5681e1","discovery-endpoints":"http://tasks.etcd-init-cluster:2579","discovery-dial-timeout":"2s","discovery-request-timeout":"5s","discovery-keepalive-time":"2s","discovery-keepalive-timeout":"6s","discovery-insecure-transport":true,"discovery-insecure-skip-tls-verify":true,"discovery-cert":"","discovery-key":"","discovery-cacert":"","discovery-user":"","downgrade-check-interval":"5s","max-learners":1,"v2-deprecation":"write-only"}
{"level":"warn","ts":"2025-09-01T09:28:22.594564Z","caller":"fileutil/fileutil.go:56","msg":"check file permission","error":"directory \"/etcd-data\" exist, but the permission is \"drwxr-xr-x\". The recommended permission is \"-rwx------\" to prevent possible unprivileged access to the data"}
{"level":"info","ts":"2025-09-01T09:28:22.595233Z","logger":"bbolt","caller":"backend/backend.go:203","msg":"Opening db file (/etcd-data/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: hashmap, ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 10737418240, PageSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0xc000052148}"}
{"level":"info","ts":"2025-09-01T09:28:22.596442Z","logger":"bbolt","caller":"[email protected]/db.go:321","msg":"Opening bbolt db (/etcd-data/member/snap/db) successfully"}
{"level":"info","ts":"2025-09-01T09:28:22.596588Z","caller":"storage/backend.go:80","msg":"opened backend db","path":"/etcd-data/member/snap/db","took":"1.446943ms"}
{"level":"info","ts":"2025-09-01T09:28:22.596635Z","caller":"etcdserver/bootstrap.go:220","msg":"restore consistentIndex","index":0}
{"level":"info","ts":"2025-09-01T09:28:22.596655Z","caller":"etcdserver/bootstrap.go:232","msg":"recovered v3 backend","backend-size-bytes":20480,"backend-size":"20 kB","backend-size-in-use-bytes":12288,"backend-size-in-use":"12 kB"}
{"level":"warn","ts":"2025-09-01T09:28:22.596667Z","caller":"schema/schema.go:39","msg":"Failed to detect storage schema version. Please wait till wal snapshot before upgrading cluster."}
{"level":"info","ts":"2025-09-01T09:28:22.596684Z","caller":"etcdserver/bootstrap.go:94","msg":"bootstrapping cluster"}
{"level":"error","ts":"2025-09-01T09:28:22.599508Z","caller":"etcdserver/server.go:309","msg":"bootstrap failed","error":"cannot fetch cluster info from peer urls: could not retrieve cluster information from the given URLs","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver.NewServer\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:309\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\tgo.etcd.io/etcd/server/v3/embed/etcd.go:262\ngo.etcd.io/etcd/server/v3/etcdmain.startEtcd\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:207\ngo.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:114\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:31\nruntime.main\n\truntime/proc.go:272"}
{"level":"info","ts":"2025-09-01T09:28:22.599764Z","caller":"embed/etcd.go:426","msg":"closing etcd server","name":"etcd-testing3","data-dir":"/etcd-data","advertise-peer-urls":["http://etcd-testing3:2380"],"advertise-client-urls":["http://etcd-testing3:2379"]}
{"level":"info","ts":"2025-09-01T09:28:22.599943Z","caller":"embed/etcd.go:428","msg":"closed etcd server","name":"etcd-testing3","data-dir":"/etcd-data","advertise-peer-urls":["http://etcd-testing3:2380"],"advertise-client-urls":["http://etcd-testing3:2379"]}
{"level":"fatal","ts":"2025-09-01T09:28:22.600031Z","caller":"etcdmain/etcd.go:183","msg":"discovery failed","error":"cannot fetch cluster info from peer urls: could not retrieve cluster information from the given URLs","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:183\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:31\nruntime.main\n\truntime/proc.go:272"}

Nodes 1 & 2 are not logging anything so it looks like node 3 doesn't try to connect to them.

I checked that the cluster infromation is available in the discovery cluster with etcdctl get --prefix /_etcd/registry/5a4a785f-c9d9-4b69-8473-b0573c5681e1:

/_etcd/registry/5a4a785f-c9d9-4b69-8473-b0573c5681e1/_config/size
3
/_etcd/registry/5a4a785f-c9d9-4b69-8473-b0573c5681e1/members/6fa6da62f2b4add9
etcd-testing3=http://etcd-testing3:2380
/_etcd/registry/5a4a785f-c9d9-4b69-8473-b0573c5681e1/members/ea34b7c7249d2743
etcd-testing1=http://etcd-testing1:2380
/_etcd/registry/5a4a785f-c9d9-4b69-8473-b0573c5681e1/members/f99c4a5e43d12e88
etcd-testing2=http://etcd-testing2:2380

0 replies

teapot9 · 2025-09-01T22:19:45Z

teapot9
Sep 1, 2025
Author

After a reset of the etcd cluster, I could not reproduce this exact issue.
I can now purge a member and it will resync with the cluster on next restart.
I suspect this issue was caused by playing around a lot with members, deleting nodes, removing members.

However, this is still the case currently: once I remove a member from the cluster with etcdctl member remove, it will not be able to join the cluster again.
This will result in the member being rejected by other members, even after a purge, with the following error:

{"level":"warn","ts":"2025-09-01T15:51:25.429298Z","caller":"rafthttp/http.go:396","msg":"rejected stream from remote peer because it was removed","local-member-id":"ea34b7c7249d2743","remote-peer-id-stream-handler":"ea34b7c7249d2743","remote-peer-id-from":"6fa6da62f2b4add9"}

This isn't a huge problem though, I just have to make sure not to member remove any member.

6 replies

teapot9 Sep 2, 2025
Author

The problem is that even if I purge member data, it will still keep the same member ID, so it cannot rejoin since this member ID is in the members_removed list

serathius Sep 2, 2025
Maintainer

Something doesn't add up. Member only knows its ID by reading it from disk. New ID is generated using cluster name,peer URLs and timestamp now. So there is no way for member with purged local data to try to rejoin cluster with the same ID.

teapot9 Sep 2, 2025
Author

This is what I though reading the doc. However, I found out my member ID is effectively always the same, as if the timestamp wasn't used when starting a member with discovery mode.

I found the function generating the member ID:

etcd/server/etcdserver/api/membership/member.go

Line 63 in 991ea7e

    
           func computeMemberID(peerURLs types.URLs, clusterName string, now *time.Time) types.ID {

Now, I simulated if the timestamp is nil:

$ printf 'http://etcd-testing3:23805a4a785f-c9d9-4b69-8473-b0573c5681e1' | sha1sum
6fa6da62f2b4add9ec966adb6b160b57fa44c009  -

The member ID of node 3 is 6fa6da62f2b4add9 which matches the result of my test.

serathius Sep 2, 2025
Maintainer

Oh, I didn't noticed you used discovery. That makes sense that removed member got the same identity as it was able to read it from discovery. In my opinion discovery is good for demos if you want to have easy setup, and should not be used for production, as you never want to depend on third party system.

teapot9 Sep 2, 2025
Author

This means a member once removed can never join again with discovery, which is why I concluded that I should avoid etcdctl member remove.

For discovery I use a dedicated cluster that I manage myself, I'm trying to automate scaling up the cluster using discovery.

Cluster member doesn't join back after its data was reset #20586

Uh oh!

teapot9 Aug 30, 2025

Replies: 3 comments · 6 replies

Uh oh!

Uh oh!

serathius Sep 1, 2025 Maintainer

Uh oh!

teapot9 Sep 1, 2025 Author

Uh oh!

teapot9 Sep 1, 2025 Author

Uh oh!

teapot9 Sep 2, 2025 Author

Uh oh!

Uh oh!

serathius Sep 2, 2025 Maintainer

Uh oh!

Uh oh!

teapot9 Sep 2, 2025 Author

Uh oh!

serathius Sep 2, 2025 Maintainer

Uh oh!

teapot9 Sep 2, 2025 Author

teapot9
Aug 30, 2025

Replies: 3 comments 6 replies

serathius
Sep 1, 2025
Maintainer

teapot9
Sep 1, 2025
Author

teapot9
Sep 1, 2025
Author

teapot9 Sep 2, 2025
Author

serathius Sep 2, 2025
Maintainer

teapot9 Sep 2, 2025
Author

serathius Sep 2, 2025
Maintainer

teapot9 Sep 2, 2025
Author