ETCD real-time cross domain disaster recovery

[nancylcl]

故障集群名：cluster1；故障集群DB名：db1；新集群名：cluster2；新集群DB名：db2
我将故障集群(cluster1)的Learner节点的db(这里称其为db1)同步到新集群(cluster2)的data-dir路径下，重启新集群；为什么在新集群虽然可以查看到数据，但是无法put、del；我再将新集群的db恢复到db2，然后又将新集群db修改成db1，重启服务后发现不仅可以查到数据，也可以put，del。这个为什么呢？又该如何优化呢？

[jmhbnz]

Hey @nancylcl - Can you please update your post to English. The language used in this discussion forum is English.

[nancylcl]

Ok, ok, I‘am sorry

Fault cluster name: cluster1; Fault cluster DB name: db1; New group name: cluster2; New cluster DB name: db

I will synchronize the db (referred to as db1) of the Learner node in the failed cluster (cluster1) to the data dir path of the new cluster (cluster2), and restart the new cluster; Why is it that although data can be viewed in the new cluster, it cannot be put or del; I will restore the db of the new cluster to db2, and then modify the db of the new cluster to db1. After restarting the service, I found that not only can I find data, but I can also put and del. In fact, restarting the cluster multiple times can also solve this problem, but why can the first method solve it? Is there a solution that can be optimized that requires multiple restarts?

[jmhbnz]

Hey @nancylcl - Thanks for posting in english. I'm not sure I understand the issue. I think you will need to provide a lot more detail for us to understand better.

Consider posting the configuration you are using for your etcd clusters and the step by step commands you are running in sequence, along with their output so we can understand what you are trying to do.

[nancylcl]

Sorry, maybe my expression wasn't clear enough

In other words, after replacing the db of the failed cluster Learner node with the new cluster db, I restart the new cluster and find that data can be queried but cannot be written normally; After multiple restarts of the cluster, write and delete operations can only proceed normally. Why is this? Is there a solution that can be optimized?

[nancylcl]

The operation process is as follows: Taking a node in a cluster as an example

First：Backup the new cluster db file
mv /var/lib/etcd/member/snap/db{,.bak}
Second：Backup the new cluster db file and upload the db file of the faulty cluster Learner to the new cluster "/var/lib/etc/member/snap/" directory
Third：chmod 600 /var/lib/etcd/member/snap/db && chown nobody:nobody /var/lib/etcd/member/snap/db
Fourth：systemctl restart etcd.service
etcdctl --endpoints=http://10.152.24.101:2379 put test 123
It was found that the query can be performed normally, but a "context deadline exceeded" error will be reported when writing.
After multiple restarts of the cluster, it can be written normally.

[ahrtr]

Why is it that although data can be viewed in the new cluster, it cannot be put or del;

What's the error message? Is it "etcdserver: rpc not supported for learner" ?

FYI.

etcd/server/etcdserver/api/v3rpc/util.go

Lines 141 to 150 in 262a3a1

	func isRPCSupportedForLearner(req any) bool {
	switch r := req.(type) {
	case *pb.StatusRequest:
	return true
	case *pb.RangeRequest:
	return r.Serializable
	default:
	return false
	}
	}

Please follow https://etcd.io/docs/v3.5/op-guide/recovery/#restoring-a-cluster to recovery the cluster.

[nancylcl]

error message:
{"level":"warn","ts":"2023-10-19T10:14:05.320+0800","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-44d5eef8-f10b-4f85-b7c2-e487b1544ab3/10.152.24.101:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Error: context deadline exceeded