etcd leader is abnormal and reports raft: stopped during removing one follower

[okJiang]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via etcd maintainers.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

The etcd leader was temporarily unavailable and a re-election occurred.

What did you expect to happen?

leader provides services normally

How can we reproduce it (as minimally and precisely as possible)?

These are the steps it triggers, but I haven't been able to reproduce it stably.

There is a three nodes etcd cluster, then

1. add two etcd node
2. remove a non-leader node

Anything else we need to know?

No response

Etcd version (please run commands below)

v0.5.0-alpha.5.0.20220915004622-85b640cee793

v3.4.21

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

Leader Log

[2025/06/15 10:08:56.179 +05:30] [INFO] [raft.go:1523] ["68c09489d90f1839 switched to configuration voters=(2220708991424795239 3923816963460778184 7548196295246026809 15571147237970210531)"]
[2025/06/15 10:08:56.179 +05:30] [INFO] [peer.go:333] ["stopping remote peer"] [remote-peer-id=1d93c79b4a8bacec]
[2025/06/15 10:08:56.179 +05:30] [WARN] [stream.go:291] ["closed TCP streaming connection with remote peer"] [stream-writer-type="stream MsgApp v2"] [remote-peer-id=1d93c79b4a8bacec]
[2025/06/15 10:08:56.179 +05:30] [WARN] [stream.go:301] ["stopped TCP streaming connection with remote peer"] [stream-writer-type="stream MsgApp v2"] [remote-peer-id=1d93c79b4a8bacec]
[2025/06/15 10:08:56.179 +05:30] [WARN] [stream.go:291] ["closed TCP streaming connection with remote peer"] [stream-writer-type="stream Message"] [remote-peer-id=1d93c79b4a8bacec]
[2025/06/15 10:08:56.179 +05:30] [WARN] [stream.go:301] ["stopped TCP streaming connection with remote peer"] [stream-writer-type="stream Message"] [remote-peer-id=1d93c79b4a8bacec]
[2025/06/15 10:08:56.179 +05:30] [INFO] [pipeline.go:86] ["stopped HTTP pipelining with remote peer"] [local-member-id=68c09489d90f1839] [remote-peer-id=1d93c79b4a8bacec]
[2025/06/15 10:08:56.180 +05:30] [WARN] [stream.go:436] ["lost TCP streaming connection with remote peer"] [stream-reader-type="stream MsgApp v2"] [local-member-id=68c09489d90f1839] [remote-peer-id=1d93c79b4a8bacec] [error="context canceled"]
[2025/06/15 10:08:56.180 +05:30] [WARN] [peer_status.go:68] ["peer became inactive (message send to peer failed)"] [peer-id=1d93c79b4a8bacec] [error="failed to read 1d93c79b4a8bacec on stream MsgApp v2 (context canceled)"]
[2025/06/15 10:08:56.180 +05:30] [INFO] [stream.go:459] ["stopped stream reader with remote peer"] [stream-reader-type="stream MsgApp v2"] [local-member-id=68c09489d90f1839] [remote-peer-id=1d93c79b4a8bacec]
[2025/06/15 10:08:56.180 +05:30] [WARN] [stream.go:436] ["lost TCP streaming connection with remote peer"] [stream-reader-type="stream Message"] [local-member-id=68c09489d90f1839] [remote-peer-id=1d93c79b4a8bacec] [error="context canceled"]
[2025/06/15 10:08:56.180 +05:30] [INFO] [peer.go:340] ["stopped remote peer"] [remote-peer-id=1d93c79b4a8bacec]
[2025/06/15 10:08:56.181 +05:30] [WARN] [http.go:425] ["rejected stream from remote peer because it was removed"] [local-member-id=68c09489d90f1839] [remote-peer-id-stream-handler=68c09489d90f1839] [remote-peer-id-from=1d93c79b4a8bacec]
[2025/06/15 10:08:56.181 +05:30] [WARN] [http.go:425] ["rejected stream from remote peer because it was removed"] [local-member-id=68c09489d90f1839] [remote-peer-id-stream-handler=68c09489d90f1839] [remote-peer-id-from=1d93c79b4a8bacec]
[2025/06/15 10:08:56.914 +05:30] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=endpoint://{leader_address}:2379] [attempt=0] [error="rpc error: code = Unknown desc = raft: stopped"]
[2025/06/15 10:08:56.914 +05:30] [WARN] [tso.go:370] ["save timestamp failed"] [] [dc-location=global] [timestamp-path=timestamp] [error="rpc error: code = Unknown desc = raft: stopped"]

Follower Log

[2025/06/15 10:08:56.179 +05:30] [INFO] [raft.go:1523] ["367432bd2ebb48c8 switched to configuration voters=(2220708991424795239 3923816963460778184 7548196295246026809 15571147237970210531)"]
[2025/06/15 10:08:56.179 +05:30] [INFO] [cluster.go:422] ["removed member"] [cluster-id=2630c7cb1030b311] [local-member-id=367432bd2ebb48c8] [removed-remote-peer-id=1d93c79b4a8bacec] [removed-remote-peer-urls="[http://{removed_member_address}:2380]"]
[2025/06/15 10:08:56.179 +05:30] [INFO] [peer.go:333] ["stopping remote peer"] [remote-peer-id=1d93c79b4a8bacec]
[2025/06/15 10:08:56.180 +05:30] [WARN] [stream.go:291] ["closed TCP streaming connection with remote peer"] [stream-writer-type="stream MsgApp v2"] [remote-peer-id=1d93c79b4a8bacec]
[2025/06/15 10:08:56.180 +05:30] [WARN] [stream.go:301] ["stopped TCP streaming connection with remote peer"] [stream-writer-type="stream MsgApp v2"] [remote-peer-id=1d93c79b4a8bacec]
[2025/06/15 10:08:56.180 +05:30] [WARN] [stream.go:291] ["closed TCP streaming connection with remote peer"] [stream-writer-type="stream Message"] [remote-peer-id=1d93c79b4a8bacec]
[2025/06/15 10:08:56.180 +05:30] [WARN] [stream.go:301] ["stopped TCP streaming connection with remote peer"] [stream-writer-type="stream Message"] [remote-peer-id=1d93c79b4a8bacec]
[2025/06/15 10:08:56.180 +05:30] [INFO] [pipeline.go:86] ["stopped HTTP pipelining with remote peer"] [local-member-id=367432bd2ebb48c8] [remote-peer-id=1d93c79b4a8bacec]
[2025/06/15 10:08:56.180 +05:30] [WARN] [stream.go:436] ["lost TCP streaming connection with remote peer"] [stream-reader-type="stream MsgApp v2"] [local-member-id=367432bd2ebb48c8] [remote-peer-id=1d93c79b4a8bacec] [error="context canceled"]
[2025/06/15 10:08:56.180 +05:30] [WARN] [peer_status.go:68] ["peer became inactive (message send to peer failed)"] [peer-id=1d93c79b4a8bacec] [error="failed to read 1d93c79b4a8bacec on stream MsgApp v2 (context canceled)"]
[2025/06/15 10:08:56.180 +05:30] [INFO] [stream.go:459] ["stopped stream reader with remote peer"] [stream-reader-type="stream MsgApp v2"] [local-member-id=367432bd2ebb48c8] [remote-peer-id=1d93c79b4a8bacec]
[2025/06/15 10:08:56.180 +05:30] [WARN] [stream.go:436] ["lost TCP streaming connection with remote peer"] [stream-reader-type="stream Message"] [local-member-id=367432bd2ebb48c8] [remote-peer-id=1d93c79b4a8bacec] [error="context canceled"]
[2025/06/15 10:08:56.180 +05:30] [INFO] [stream.go:459] ["stopped stream reader with remote peer"] [stream-reader-type="stream Message"] [local-member-id=367432bd2ebb48c8] [remote-peer-id=1d93c79b4a8bacec]
[2025/06/15 10:08:56.180 +05:30] [INFO] [peer.go:340] ["stopped remote peer"] [remote-peer-id=1d93c79b4a8bacec]
[2025/06/15 10:08:56.180 +05:30] [INFO] [transport.go:369] ["removed remote peer"] [local-member-id=367432bd2ebb48c8] [removed-remote-peer-id=1d93c79b4a8bacec]
[2025/06/15 10:08:56.182 +05:30] [WARN] [http.go:425] ["rejected stream from remote peer because it was removed"] [local-member-id=367432bd2ebb48c8] [remote-peer-id-stream-handler=367432bd2ebb48c8] [remote-peer-id-from=1d93c79b4a8bacec]
[2025/06/15 10:08:56.182 +05:30] [WARN] [http.go:425] ["rejected stream from remote peer because it was removed"] [local-member-id=367432bd2ebb48c8] [remote-peer-id-stream-handler=367432bd2ebb48c8] [remote-peer-id-from=1d93c79b4a8bacec]
[2025/06/15 10:08:56.951 +05:30] [WARN] [util.go:121] ["failed to apply request"] [took=32.477µs] [request="header:<ID:1745588778383329725 > lease_revoke:<id:48c895ce25c26258>"] [response=size:29] [error="lease not found"]
[2025/06/15 10:08:56.965 +05:30] [INFO] [raft.go:865] ["367432bd2ebb48c8 [term: 13] received a MsgVote message with higher term from d817d3e9d884eee3 [term: 14]"]
[2025/06/15 10:08:56.965 +05:30] [INFO] [raft.go:706] ["367432bd2ebb48c8 became follower at term 14"]
[2025/06/15 10:08:56.965 +05:30] [INFO] [raft.go:966] ["367432bd2ebb48c8 [logterm: 13, index: 177637438, vote: 0] cast MsgVote for d817d3e9d884eee3 [logterm: 13, index: 177637438] at term 14"]
[2025/06/15 10:08:56.965 +05:30] [INFO] [node.go:333] ["raft.node: 367432bd2ebb48c8 lost leader 68c09489d90f1839 at term 14"]
[2025/06/15 10:08:56.966 +05:30] [INFO] [node.go:327] ["raft.node: 367432bd2ebb48c8 elected leader d817d3e9d884eee3 at term 14"]
[2025/06/15 10:08:57.138 +05:30] [INFO] [health_checker.go:386] ["remove stale etcd client"] [endpoint=http://{removed_member_address}:2379] [source=election-etcd-client]
[2025/06/15 10:08:57.138 +05:30] [INFO] [health_checker.go:386] ["remove stale etcd client"] [endpoint=http://{removed_member_address}:2379] [source=server-etcd-client]
[2025/06/15 10:08:57.151 +05:30] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=endpoint://{self}.svc:2379] [attempt=0] [error="rpc error: code = Unavailable desc = etcdserver: server stopped"]
[2025/06/15 10:08:57.787 +05:30] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=endpoint://{self}:2379] [attempt=0] [error="rpc error: code = Unavailable desc = etcdserver: server stopped"]
[2025/06/15 10:08:57.942 +05:30] [INFO] [raft.go:865] ["367432bd2ebb48c8 [term: 14] received a MsgVote message with higher term from 1ed18b0fd7641267 [term: 15]"]
[2025/06/15 10:08:57.942 +05:30] [INFO] [raft.go:706] ["367432bd2ebb48c8 became follower at term 15"]
[2025/06/15 10:08:57.942 +05:30] [INFO] [raft.go:966] ["367432bd2ebb48c8 [logterm: 14, index: 177637456, vote: 0] cast MsgVote for 1ed18b0fd7641267 [logterm: 14, index: 177637456] at term 15"]
[2025/06/15 10:08:57.942 +05:30] [INFO] [node.go:333] ["raft.node: 367432bd2ebb48c8 lost leader d817d3e9d884eee3 at term 15"]
[2025/06/15 10:08:57.942 +05:30] [INFO] [node.go:327] ["raft.node: 367432bd2ebb48c8 elected leader 1ed18b0fd7641267 at term 15"]
[2025/06/15 10:08:57.943 +05:30] [WARN] [util.go:121] ["failed to apply request"] [took=58.981µs] [request="header:<ID:17213768716050668389 > lease_revoke:<id:183993bed37249cd>"] [response=size:29] [error="lease not found"]
[2025/06/15 10:08:58.138 +05:30] [WARN] [util.go:121] ["failed to apply request"] [took=18.136µs] [request="header:<ID:1745588778383329756 > lease_revoke:<id:48c895ce277fd8d1>"] [response=size:29] [error="lease not found"]

[okJiang]

I'd really appreciate any help or suggestions. Thank you in advance!

[jmhbnz]

Hi @okJiang - Thanks for your question, reviewing the etcd version it appears you are running etcd v3.4.21. This release is almost three years old. We have fixed countless bugs and security issues over the last three years. Please upgrade to a modern release of etcd as soon as possible and let us know if the issue persists.

[okJiang]

Hi @jmhbnz , we also encountered the same problem in v3.4.31, which is a version from a year ago. Could you provide support and help investigate? Thank you very much

first time
leader

{"level":"INFO","time":"2025/09/10 20:01:35.434 +00:00","caller":"raft.go:1523","message":"cc33ef0f9744a578 switched to configuration voters=(7726458759518237103 14714367257790948728)"}
{"level":"INFO","time":"2025/09/10 20:01:35.436 +00:00","caller":"cluster.go:425","message":"removed member","cluster-id":"3440cd1929201c0f","local-member-id":"cc33ef0f9744a578","removed-remote-peer-id":"b227f5b331b04ad9","removed-remote-peer-urls":["xxxx"]}
{"level":"INFO","time":"2025/09/10 20:01:35.436 +00:00","caller":"peer.go:333","message":"stopping remote peer","remote-peer-id":"b227f5b331b04ad9"}
{"level":"WARN","time":"2025/09/10 20:01:35.436 +00:00","caller":"server.go:916","message":"rejected Raft message from removed member","local-member-id":"cc33ef0f9744a578","removed-member-id":"b227f5b331b04ad9"}
{"level":"WARN","time":"2025/09/10 20:01:35.436 +00:00","caller":"peer.go:179","message":"failed to process Raft message","error":"cannot process message from removed member"}
{"level":"WARN","time":"2025/09/10 20:01:35.437 +00:00","caller":"stream.go:291","message":"closed TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","remote-peer-id":"b227f5b331b04ad9"}
{"level":"WARN","time":"2025/09/10 20:01:35.437 +00:00","caller":"stream.go:301","message":"stopped TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","remote-peer-id":"b227f5b331b04ad9"}
{"level":"WARN","time":"2025/09/10 20:01:35.438 +00:00","caller":"stream.go:291","message":"closed TCP streaming connection with remote peer","stream-writer-type":"stream Message","remote-peer-id":"b227f5b331b04ad9"}
{"level":"WARN","time":"2025/09/10 20:01:35.438 +00:00","caller":"stream.go:301","message":"stopped TCP streaming connection with remote peer","stream-writer-type":"stream Message","remote-peer-id":"b227f5b331b04ad9"}
{"level":"INFO","time":"2025/09/10 20:01:35.438 +00:00","caller":"pipeline.go:86","message":"stopped HTTP pipelining with remote peer","local-member-id":"cc33ef0f9744a578","remote-peer-id":"b227f5b331b04ad9"}
{"level":"WARN","time":"2025/09/10 20:01:35.438 +00:00","caller":"stream.go:436","message":"lost TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"cc33ef0f9744a578","remote-peer-id":"b227f5b331b04ad9","error":"context canceled"}
{"level":"WARN","time":"2025/09/10 20:01:35.438 +00:00","caller":"peer_status.go:68","message":"peer became inactive (message send to peer failed)","peer-id":"b227f5b331b04ad9","error":"failed to read b227f5b331b04ad9 on stream MsgApp v2 (context canceled)"}
{"level":"INFO","time":"2025/09/10 20:01:35.438 +00:00","caller":"stream.go:459","message":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"cc33ef0f9744a578","remote-peer-id":"b227f5b331b04ad9"}
{"level":"WARN","time":"2025/09/10 20:01:35.438 +00:00","caller":"stream.go:436","message":"lost TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"cc33ef0f9744a578","remote-peer-id":"b227f5b331b04ad9","error":"context canceled"}
{"level":"INFO","time":"2025/09/10 20:01:35.438 +00:00","caller":"stream.go:459","message":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"cc33ef0f9744a578","remote-peer-id":"b227f5b331b04ad9"}
{"level":"INFO","time":"2025/09/10 20:01:35.438 +00:00","caller":"peer.go:340","message":"stopped remote peer","remote-peer-id":"b227f5b331b04ad9"}
{"level":"INFO","time":"2025/09/10 20:01:35.438 +00:00","caller":"transport.go:369","message":"removed remote peer","local-member-id":"cc33ef0f9744a578","removed-remote-peer-id":"b227f5b331b04ad9"}
{"level":"INFO","time":"2025/09/10 20:01:35.438 +00:00","caller":"server.go:2068","message":"applied a configuration change through raft","local-member-id":"cc33ef0f9744a578","raft-conf-change":"ConfChangeRemoveNode","raft-conf-change-node-id":"b227f5b331b04ad9"}
{"level":"WARN","time":"2025/09/10 20:01:35.453 +00:00","caller":"config_logging.go:287","message":"rejected connection","remote-addr":"10.9.183.116:33386","server-name":"xxxxx","error":"EOF"}
{"level":"WARN","time":"2025/09/10 20:01:35.459 +00:00","caller":"http.go:425","message":"rejected stream from remote peer because it was removed","local-member-id":"cc33ef0f9744a578","remote-peer-id-stream-handler":"cc33ef0f9744a578","remote-peer-id-from":"b227f5b331b04ad9"}
{"level":"ERROR","time":"2025/09/10 20:01:36.294 +00:00","caller":"etcd_kv.go:111","message":"save to etcd meet error","key":"/pd/7413512787457025942/raft/min_resolved_ts","value":"664d4dec5c4080a","error":"[PD:etcd:ErrEtcdKVPut]rpc error: code = Unknown desc = raft: stopped: rpc error: code = Unknown desc = raft: stopped","errorVerbose":"[PD:etcd:ErrEtcdKVPut]rpc error: code = Unknown desc = raft: stopped: rpc error: code = Unknown desc = raft: stopped\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByCause\n\t/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:307\ngithub.com/tikv/pd/pkg/storage/kv.(*etcdKVBase).Save\n\t/mnt/tidb/pd/pkg/storage/kv/etcd_kv.go:110\ngithub.com/tikv/pd/pkg/storage/endpoint.(*StorageEndpoint).SaveMinResolvedTS\n\t/mnt/tidb/pd/pkg/storage/endpoint/min_resolved_ts.go:53\ngithub.com/tikv/pd/server/cluster.(*RaftCluster).runMinResolvedTSJob\n\t/mnt/tidb/pd/server/cluster/cluster.go:2323\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
{"level":"ERROR","time":"2025/09/10 20:01:36.392 +00:00","caller":"etcd_kv.go:111","message":"save to etcd meet error","key":"/pd/7413512787457025942/gc/safe_point/service/backup-stream-pitr-106840","value":"{\"service_id\":\"backup-stream-pitr-106840\",\"expired_at\":1757541696,\"safe_point\":460727078149685521}","error":"[PD:etcd:ErrEtcdKVPut]rpc error: code = Unknown desc = raft: stopped: rpc error: code = Unknown desc = raft: stopped","errorVerbose":"[PD:etcd:ErrEtcdKVPut]rpc error: code = Unknown desc = raft: stopped: rpc error: code = Unknown desc = raft: stopped\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByCause\n\t/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:307\ngithub.com/tikv/pd/pkg/storage/kv.(*etcdKVBase).Save\n\t/mnt/tidb/pd/pkg/storage/kv/etcd_kv.go:110\ngithub.com/tikv/pd/pkg/storage/endpoint.saveJSONInTxn\n\t/mnt/tidb/pd/pkg/storage/endpoint/util.go:58\ngithub.com/tikv/pd/pkg/storage/endpoint.(*StorageEndpoint).saveJSON\n\t/mnt/tidb/pd/pkg/storage/endpoint/util.go:50\ngithub.com/tikv/pd/pkg/storage/endpoint.(*StorageEndpoint).SaveServiceGCSafePoint\n\t/mnt/tidb/pd/pkg/storage/endpoint/gc_safe_point.go:172\ngithub.com/tikv/pd/pkg/gc.(*SafePointManager).UpdateServiceGCSafePoint\n\t/mnt/tidb/pd/pkg/gc/safepoint.go:93\ngithub.com/tikv/pd/server.(*GrpcServer).UpdateServiceGCSafePoint\n\t/mnt/tidb/pd/server/grpc_service.go:2168\ngithub.com/pingcap/kvproto/pkg/pdpb._PD_UpdateServiceGCSafePoint_Handler.func1\n\t/go/pkg/mod/github.com/pingcap/[email protected]/pkg/pdpb/pdpb.pb.go:10049\ngithub.com/grpc-ecosystem/go-grpc-prometheus.init.(*ServerMetrics).UnaryServerInterceptor.func3\n\t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server_metrics.go:107\ngo.etcd.io/etcd/etcdserver/api/v3rpc.Server.ChainUnaryServer.func6.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:48\ngo.etcd.io/etcd/etcdserver/api/v3rpc.Server.newUnaryInterceptor.func4\n\t/go/pkg/mod/go.etcd.io/[email protected]/etcdserver/api/v3rpc/interceptor.go:71\ngo.etcd.io/etcd/etcdserver/api/v3rpc.Server.ChainUnaryServer.func6.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:48\ngo.etcd.io/etcd/etcdserver/api/v3rpc.Server.newLogUnaryInterceptor.func3\n\t/go/pkg/mod/go.etcd.io/[email protected]/etcdserver/api/v3rpc/interceptor.go:78\ngo.etcd.io/etcd/etcdserver/api/v3rpc.Server.ChainUnaryServer.func6\n\t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:53\ngithub.com/pingcap/kvproto/pkg/pdpb._PD_UpdateServiceGCSafePoint_Handler\n\t/go/pkg/mod/github.com/pingcap/[email protected]/pkg/pdpb/pdpb.pb.go:10051\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1386\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1797\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1027\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
{"level":"INFO","time":"2025/09/10 20:01:37.172 +00:00","caller":"lease.go:190","message":"stop lease keep alive worker","purpose":"leader election"}

second time

leader

{"level":"INFO","time":"2025/09/10 21:12:07.493 +00:00","caller":"raft.go:1523","message":"cc33ef0f9744a578 switched to configuration voters=(9054114933046903267 14714367257790948728)"}
{"level":"INFO","time":"2025/09/10 21:12:07.494 +00:00","caller":"cluster.go:425","message":"removed member","cluster-id":"3440cd1929201c0f","local-member-id":"cc33ef0f9744a578","removed-remote-peer-id":"6b39e54e24ca31af","removed-remote-peer-urls":["xxxxx"]}
{"level":"INFO","time":"2025/09/10 21:12:07.494 +00:00","caller":"peer.go:333","message":"stopping remote peer","remote-peer-id":"6b39e54e24ca31af"}
{"level":"WARN","time":"2025/09/10 21:12:07.495 +00:00","caller":"stream.go:291","message":"closed TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","remote-peer-id":"6b39e54e24ca31af"}
{"level":"WARN","time":"2025/09/10 21:12:07.495 +00:00","caller":"stream.go:301","message":"stopped TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","remote-peer-id":"6b39e54e24ca31af"}
{"level":"WARN","time":"2025/09/10 21:12:07.495 +00:00","caller":"stream.go:291","message":"closed TCP streaming connection with remote peer","stream-writer-type":"stream Message","remote-peer-id":"6b39e54e24ca31af"}
{"level":"WARN","time":"2025/09/10 21:12:07.495 +00:00","caller":"stream.go:301","message":"stopped TCP streaming connection with remote peer","stream-writer-type":"stream Message","remote-peer-id":"6b39e54e24ca31af"}
{"level":"INFO","time":"2025/09/10 21:12:07.495 +00:00","caller":"pipeline.go:86","message":"stopped HTTP pipelining with remote peer","local-member-id":"cc33ef0f9744a578","remote-peer-id":"6b39e54e24ca31af"}
{"level":"WARN","time":"2025/09/10 21:12:07.495 +00:00","caller":"stream.go:436","message":"lost TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"cc33ef0f9744a578","remote-peer-id":"6b39e54e24ca31af","error":"context canceled"}
{"level":"WARN","time":"2025/09/10 21:12:07.495 +00:00","caller":"peer_status.go:68","message":"peer became inactive (message send to peer failed)","peer-id":"6b39e54e24ca31af","error":"failed to read 6b39e54e24ca31af on stream MsgApp v2 (context canceled)"}
{"level":"INFO","time":"2025/09/10 21:12:07.495 +00:00","caller":"stream.go:459","message":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"cc33ef0f9744a578","remote-peer-id":"6b39e54e24ca31af"}
{"level":"WARN","time":"2025/09/10 21:12:07.495 +00:00","caller":"stream.go:436","message":"lost TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"cc33ef0f9744a578","remote-peer-id":"6b39e54e24ca31af","error":"context canceled"}
{"level":"INFO","time":"2025/09/10 21:12:07.495 +00:00","caller":"stream.go:459","message":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"cc33ef0f9744a578","remote-peer-id":"6b39e54e24ca31af"}
{"level":"INFO","time":"2025/09/10 21:12:07.495 +00:00","caller":"peer.go:340","message":"stopped remote peer","remote-peer-id":"6b39e54e24ca31af"}
{"level":"INFO","time":"2025/09/10 21:12:07.495 +00:00","caller":"transport.go:369","message":"removed remote peer","local-member-id":"cc33ef0f9744a578","removed-remote-peer-id":"6b39e54e24ca31af"}
{"level":"WARN","time":"2025/09/10 21:12:07.514 +00:00","caller":"http.go:425","message":"rejected stream from remote peer because it was removed","local-member-id":"cc33ef0f9744a578","remote-peer-id-stream-handler":"cc33ef0f9744a578","remote-peer-id-from":"6b39e54e24ca31af"}
{"level":"WARN","time":"2025/09/10 21:12:07.520 +00:00","caller":"http.go:425","message":"rejected stream from remote peer because it was removed","local-member-id":"cc33ef0f9744a578","remote-peer-id-stream-handler":"cc33ef0f9744a578","remote-peer-id-from":"6b39e54e24ca31af"}
{"level":"ERROR","time":"2025/09/10 21:12:08.088 +00:00","caller":"etcd_kv.go:111","message":"save to etcd meet error","key":"/pd/7413512787457025942/raft/min_resolved_ts","value":"664d5e0704408b7","error":"[PD:etcd:ErrEtcdKVPut]rpc error: code = Unknown desc = raft: stopped: rpc error: code = Unknown desc = raft: stopped","errorVerbose":"[PD:etcd:ErrEtcdKVPut]rpc error: code = Unknown desc = raft: stopped: rpc error: code = Unknown desc = raft: stopped\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByCause\n\t/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:307\ngithub.com/tikv/pd/pkg/storage/kv.(*etcdKVBase).Save\n\t/mnt/tidb/pd/pkg/storage/kv/etcd_kv.go:110\ngithub.com/tikv/pd/pkg/storage/endpoint.(*StorageEndpoint).SaveMinResolvedTS\n\t/mnt/tidb/pd/pkg/storage/endpoint/min_resolved_ts.go:53\ngithub.com/tikv/pd/server/cluster.(*RaftCluster).runMinResolvedTSJob\n\t/mnt/tidb/pd/server/cluster/cluster.go:2323\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
{"level":"ERROR","time":"2025/09/10 21:12:08.804 +00:00","caller":"etcd_kv.go:111","message":"save to etcd meet error","key":"/pd/7413512787457025942/gc/safe_point/service/backup-stream-pitr-106911","value":"{\"service_id\":\"backup-stream-pitr-106911\",\"expired_at\":1757545928,\"safe_point\":460728151327901748}","error":"[PD:etcd:ErrEtcdKVPut]rpc error: code = Unknown desc = raft: stopped: rpc error: code = Unknown desc = raft: stopped","errorVerbose":"[PD:etcd:ErrEtcdKVPut]rpc error: code = Unknown desc = raft: stopped: rpc error: code = Unknown desc = raft: stopped\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByCause\n\t/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:307\ngithub.com/tikv/pd/pkg/storage/kv.(*etcdKVBase).Save\n\t/mnt/tidb/pd/pkg/storage/kv/etcd_kv.go:110\ngithub.com/tikv/pd/pkg/storage/endpoint.saveJSONInTxn\n\t/mnt/tidb/pd/pkg/storage/endpoint/util.go:58\ngithub.com/tikv/pd/pkg/storage/endpoint.(*StorageEndpoint).saveJSON\n\t/mnt/tidb/pd/pkg/storage/endpoint/util.go:50\ngithub.com/tikv/pd/pkg/storage/endpoint.(*StorageEndpoint).SaveServiceGCSafePoint\n\t/mnt/tidb/pd/pkg/storage/endpoint/gc_safe_point.go:172\ngithub.com/tikv/pd/pkg/gc.(*SafePointManager).UpdateServiceGCSafePoint\n\t/mnt/tidb/pd/pkg/gc/safepoint.go:93\ngithub.com/tikv/pd/server.(*GrpcServer).UpdateServiceGCSafePoint\n\t/mnt/tidb/pd/server/grpc_service.go:2168\ngithub.com/pingcap/kvproto/pkg/pdpb._PD_UpdateServiceGCSafePoint_Handler.func1\n\t/go/pkg/mod/github.com/pingcap/[email protected]/pkg/pdpb/pdpb.pb.go:10049\ngithub.com/grpc-ecosystem/go-grpc-prometheus.init.(*ServerMetrics).UnaryServerInterceptor.func3\n\t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server_metrics.go:107\ngo.etcd.io/etcd/etcdserver/api/v3rpc.Server.ChainUnaryServer.func6.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:48\ngo.etcd.io/etcd/etcdserver/api/v3rpc.Server.newUnaryInterceptor.func4\n\t/go/pkg/mod/go.etcd.io/[email protected]/etcdserver/api/v3rpc/interceptor.go:71\ngo.etcd.io/etcd/etcdserver/api/v3rpc.Server.ChainUnaryServer.func6.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:48\ngo.etcd.io/etcd/etcdserver/api/v3rpc.Server.newLogUnaryInterceptor.func3\n\t/go/pkg/mod/go.etcd.io/[email protected]/etcdserver/api/v3rpc/interceptor.go:78\ngo.etcd.io/etcd/etcdserver/api/v3rpc.Server.ChainUnaryServer.func6\n\t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:53\ngithub.com/pingcap/kvproto/pkg/pdpb._PD_UpdateServiceGCSafePoint_Handler\n\t/go/pkg/mod/github.com/pingcap/[email protected]/pkg/pdpb/pdpb.pb.go:10051\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1386\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1797\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1027\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
{"level":"INFO","time":"2025/09/10 21:12:08.812 +00:00","caller":"grpc_service.go:2173","message":"update service GC safe point","service-id":"backup-stream-pitr-106911","expire-at":1757545928,"safepoint":460728151327901748}
{"level":"WARN","time":"2025/09/10 21:12:08.870 +00:00","caller":"tso.go:370","message":"save timestamp failed","dc-location":"global","timestamp-path":"timestamp","error":"rpc error: code = Unknown desc = raft: stopped","errorVerbose":"rpc error: code = Unknown desc = raft: stopped\ngithub.com/tikv/pd/pkg/storage/kv.(*SlowLogTxn).Commit\n\t/mnt/tidb/pd/pkg/storage/kv/etcd_kv.go:188\ngithub.com/tikv/pd/pkg/storage/kv.(*etcdTxn).commit\n\t/mnt/tidb/pd/pkg/storage/kv/etcd_kv.go:284\ngithub.com/tikv/pd/pkg/storage/kv.(*etcdKVBase).RunInTxn\n\t/mnt/tidb/pd/pkg/storage/kv/etcd_kv.go:213\ngithub.com/tikv/pd/pkg/storage/endpoint.(*StorageEndpoint).SaveTimestamp\n\t/mnt/tidb/pd/pkg/storage/endpoint/tso.go:72\ngithub.com/tikv/pd/pkg/tso.(*timestampOracle).UpdateTimestamp\n\t/mnt/tidb/pd/pkg/tso/tso.go:369\ngithub.com/tikv/pd/pkg/tso.(*GlobalTSOAllocator).UpdateTSO\n\t/mnt/tidb/pd/pkg/tso/global_allocator.go:200\ngithub.com/tikv/pd/pkg/tso.(*AllocatorManager).updateAllocator\n\t/mnt/tidb/pd/pkg/tso/allocator_manager.go:816\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
{"level":"WARN","time":"2025/09/10 21:12:08.870 +00:00","caller":"allocator_manager.go:817","message":"failed to update allocator's timestamp","dc-location":"global","name":"xxxx","error":"rpc error: code = Unknown desc = raft: stopped","errorVerbose":"rpc error: code = Unknown desc = raft: stopped\ngithub.com/tikv/pd/pkg/storage/kv.(*SlowLogTxn).Commit\n\t/mnt/tidb/pd/pkg/storage/kv/etcd_kv.go:188\ngithub.com/tikv/pd/pkg/storage/kv.(*etcdTxn).commit\n\t/mnt/tidb/pd/pkg/storage/kv/etcd_kv.go:284\ngithub.com/tikv/pd/pkg/storage/kv.(*etcdKVBase).RunInTxn\n\t/mnt/tidb/pd/pkg/storage/kv/etcd_kv.go:213\ngithub.com/tikv/pd/pkg/storage/endpoint.(*StorageEndpoint).SaveTimestamp\n\t/mnt/tidb/pd/pkg/storage/endpoint/tso.go:72\ngithub.com/tikv/pd/pkg/tso.(*timestampOracle).UpdateTimestamp\n\t/mnt/tidb/pd/pkg/tso/tso.go:369\ngithub.com/tikv/pd/pkg/tso.(*GlobalTSOAllocator).UpdateTSO\n\t/mnt/tidb/pd/pkg/tso/global_allocator.go:200\ngithub.com/tikv/pd/pkg/tso.(*AllocatorManager).updateAllocator\n\t/mnt/tidb/pd/pkg/tso/allocator_manager.go:816\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}

follower

{"level":"INFO","time":"2025/09/10 21:12:07.494 +00:00","caller":"raft.go:1523","message":"7da6ad8f93d201e3 switched to configuration voters=(9054114933046903267 14714367257790948728)"}
{"level":"INFO","time":"2025/09/10 21:12:07.496 +00:00","caller":"cluster.go:425","message":"removed member","cluster-id":"3440cd1929201c0f","local-member-id":"7da6ad8f93d201e3","removed-remote-peer-id":"6b39e54e24ca31af","removed-remote-peer-urls":["xxxxx"]}
{"level":"INFO","time":"2025/09/10 21:12:07.496 +00:00","caller":"peer.go:333","message":"stopping remote peer","remote-peer-id":"6b39e54e24ca31af"}
{"level":"WARN","time":"2025/09/10 21:12:07.496 +00:00","caller":"stream.go:291","message":"closed TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","remote-peer-id":"6b39e54e24ca31af"}
{"level":"WARN","time":"2025/09/10 21:12:07.496 +00:00","caller":"stream.go:301","message":"stopped TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","remote-peer-id":"6b39e54e24ca31af"}
{"level":"WARN","time":"2025/09/10 21:12:07.496 +00:00","caller":"stream.go:291","message":"closed TCP streaming connection with remote peer","stream-writer-type":"stream Message","remote-peer-id":"6b39e54e24ca31af"}
{"level":"WARN","time":"2025/09/10 21:12:07.496 +00:00","caller":"stream.go:301","message":"stopped TCP streaming connection with remote peer","stream-writer-type":"stream Message","remote-peer-id":"6b39e54e24ca31af"}
{"level":"INFO","time":"2025/09/10 21:12:07.496 +00:00","caller":"pipeline.go:86","message":"stopped HTTP pipelining with remote peer","local-member-id":"7da6ad8f93d201e3","remote-peer-id":"6b39e54e24ca31af"}
{"level":"WARN","time":"2025/09/10 21:12:07.496 +00:00","caller":"stream.go:436","message":"lost TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"7da6ad8f93d201e3","remote-peer-id":"6b39e54e24ca31af","error":"context canceled"}
{"level":"WARN","time":"2025/09/10 21:12:07.496 +00:00","caller":"peer_status.go:68","message":"peer became inactive (message send to peer failed)","peer-id":"6b39e54e24ca31af","error":"failed to read 6b39e54e24ca31af on stream MsgApp v2 (context canceled)"}
{"level":"INFO","time":"2025/09/10 21:12:07.496 +00:00","caller":"stream.go:459","message":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"7da6ad8f93d201e3","remote-peer-id":"6b39e54e24ca31af"}
{"level":"WARN","time":"2025/09/10 21:12:07.496 +00:00","caller":"stream.go:436","message":"lost TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"7da6ad8f93d201e3","remote-peer-id":"6b39e54e24ca31af","error":"context canceled"}
{"level":"INFO","time":"2025/09/10 21:12:07.496 +00:00","caller":"stream.go:459","message":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"7da6ad8f93d201e3","remote-peer-id":"6b39e54e24ca31af"}
{"level":"INFO","time":"2025/09/10 21:12:07.496 +00:00","caller":"peer.go:340","message":"stopped remote peer","remote-peer-id":"6b39e54e24ca31af"}
{"level":"INFO","time":"2025/09/10 21:12:07.496 +00:00","caller":"transport.go:369","message":"removed remote peer","local-member-id":"7da6ad8f93d201e3","removed-remote-peer-id":"6b39e54e24ca31af"}
{"level":"INFO","time":"2025/09/10 21:12:07.496 +00:00","caller":"server.go:2068","message":"applied a configuration change through raft","local-member-id":"7da6ad8f93d201e3","raft-conf-change":"ConfChangeRemoveNode","raft-conf-change-node-id":"6b39e54e24ca31af"}
{"level":"WARN","time":"2025/09/10 21:12:07.520 +00:00","caller":"http.go:425","message":"rejected stream from remote peer because it was removed","local-member-id":"7da6ad8f93d201e3","remote-peer-id-stream-handler":"7da6ad8f93d201e3","remote-peer-id-from":"6b39e54e24ca31af"}
{"level":"WARN","time":"2025/09/10 21:12:07.520 +00:00","caller":"http.go:425","message":"rejected stream from remote peer because it was removed","local-member-id":"7da6ad8f93d201e3","remote-peer-id-stream-handler":"7da6ad8f93d201e3","remote-peer-id-from":"6b39e54e24ca31af"}
{"level":"INFO","time":"2025/09/10 21:12:08.874 +00:00","caller":"leadership.go:375","message":"current leadership is deleted","revision":507897290,"leader-key":"/pd/7413512787457025942/leader","purpose":"leader election"}
{"level":"ERROR","time":"2025/09/10 21:12:08.874 +00:00","caller":"client.go:161","message":"region sync with leader meet error","error":"[PD:grpc:ErrGRPCRecv]receive response error: rpc error: code = Canceled desc = context canceled"}
{"level":"WARN","time":"2025/09/10 21:12:08.887 +00:00","caller":"util.go:121","message":"failed to apply request","took":"9.922µs","request":"header:<ID:11923448466809704286 username:\"xxxx\" auth_revision:1 > lease_revoke:<id:31af92fd9445b9b0>","response":"size:30","error":"lease not found"}
{"level":"INFO","time":"2025/09/10 21:12:09.065 +00:00","caller":"raft.go:1348","message":"7da6ad8f93d201e3 [term 10] received MsgTimeoutNow from cc33ef0f9744a578 and starts an election to get leadership."}
{"level":"INFO","time":"2025/09/10 21:12:09.065 +00:00","caller":"raft.go:719","message":"7da6ad8f93d201e3 became candidate at term 11"}
{"level":"INFO","time":"2025/09/10 21:12:09.065 +00:00","caller":"raft.go:830","message":"7da6ad8f93d201e3 received MsgVoteResp from 7da6ad8f93d201e3 at term 11"}
{"level":"INFO","time":"2025/09/10 21:12:09.065 +00:00","caller":"raft.go:817","message":"7da6ad8f93d201e3 [logterm: 10, index: 507918085] sent MsgVote request to cc33ef0f9744a578 at term 11"}
{"level":"INFO","time":"2025/09/10 21:12:09.065 +00:00","caller":"node.go:333","message":"raft.node: 7da6ad8f93d201e3 lost leader cc33ef0f9744a578 at term 11"}
{"level":"INFO","time":"2025/09/10 21:12:09.067 +00:00","caller":"raft.go:830","message":"7da6ad8f93d201e3 received MsgVoteResp from cc33ef0f9744a578 at term 11"}
{"level":"INFO","time":"2025/09/10 21:12:09.067 +00:00","caller":"raft.go:1295","message":"7da6ad8f93d201e3 has received 2 MsgVoteResp votes and 0 vote rejections"}
{"level":"INFO","time":"2025/09/10 21:12:09.067 +00:00","caller":"raft.go:771","message":"7da6ad8f93d201e3 became leader at term 11"}
{"level":"INFO","time":"2025/09/10 21:12:09.067 +00:00","caller":"node.go:327","message":"raft.node: 7da6ad8f93d201e3 elected leader 7da6ad8f93d201e3 at term 11"}
{"level":"INFO","time":"2025/09/10 21:12:09.875 +00:00","caller":"server.go:1669","message":"pd leader has changed, try to re-campaign a pd leader"}
{"level":"INFO","time":"2025/09/10 21:12:09.875 +00:00","caller":"server.go:1706","message":"start to campaign PD leader","campaign-leader-name":"xxxx"}
{"level":"INFO","time":"2025/09/10 21:12:09.883 +00:00","caller":"lease.go:66","message":"lease granted","lease-id":2700076429954928489,"lease-timeout":3,"purpose":"leader election"}
{"level":"INFO","time":"2025/09/10 21:12:09.884 +00:00","caller":"leadership.go:182","message":"check campaign resp","resp":null}
{"level":"ERROR","time":"2025/09/10 21:12:09.889 +00:00","caller":"server.go:1712","message":"campaign PD leader meets error due to etcd error","campaign-leader-name":"xxxxxx","error":"[PD:etcd:ErrEtcdTxnInternal]rpc error: code = Unknown desc = raft: stopped: rpc error: code = Unknown desc = raft: stopped","errorVerbose":"[PD:etcd:ErrEtcdTxnInternal]rpc error: code = Unknown desc = raft: stopped: rpc error: code = Unknown desc = raft: stopped\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByCause\n\t/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:307\ngithub.com/tikv/pd/pkg/election.(*Leadership).Campaign\n\t/mnt/tidb/pd/pkg/election/leadership.go:185\ngithub.com/tikv/pd/pkg/member.(*EmbeddedEtcdMember).CampaignLeader\n\t/mnt/tidb/pd/pkg/member/member.go:198\ngithub.com/tikv/pd/server.(*Server).campaignLeader\n\t/mnt/tidb/pd/server/server.go:1707\ngithub.com/tikv/pd/server.(*Server).leaderLoop\n\t/mnt/tidb/pd/server/server.go:1701\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
{"level":"INFO","time":"2025/09/10 21:12:09.895 +00:00","caller":"server.go:1706","message":"start to campaign PD leader","campaign-leader-name":"xxxx"}
{"level":"ERROR","time":"2025/09/10 21:12:09.895 +00:00","caller":"server.go:1712","message":"campaign PD leader meets error due to etcd error","campaign-leader-name":"xxxx","error":"[PD:etcd:ErrEtcdGrantLease]rpc error: code = Unknown desc = raft: stopped: rpc error: code = Unknown desc = raft: stopped","errorVerbose":"[PD:etcd:ErrEtcdGrantLease]rpc error: code = Unknown desc = raft: stopped: rpc error: code = Unknown desc = raft: stopped\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByCause\n\t/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:307\ngithub.com/tikv/pd/pkg/election.(*lease).Grant\n\t/mnt/tidb/pd/pkg/election/lease.go:61\ngithub.com/tikv/pd/pkg/election.(*Leadership).Campaign\n\t/mnt/tidb/pd/pkg/election/leadership.go:171\ngithub.com/tikv/pd/pkg/member.(*EmbeddedEtcdMember).CampaignLeader\n\t/mnt/tidb/pd/pkg/member/member.go:198\ngithub.com/tikv/pd/server.(*Server).campaignLeader\n\t/mnt/tidb/pd/server/server.go:1707\ngithub.com/tikv/pd/server.(*Server).leaderLoop\n\t/mnt/tidb/pd/server/server.go:1701\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
{"level":"INFO","time":"2025/09/10 21:12:09.897 +00:00","caller":"server.go:1706","message":"start to campaign PD leader","campaign-leader-name":"xxxxx"}