Purpose of peer versus client separation

[camac2025]

Hello all,

I don't quite see this covered explicitly in the docs, but apologies if it is.

I am trying to understand the distinction between peers and client connections in etcd. The background is that we run two separate etcd clusters to support our patroni implementations. In one cluster we have VMs with single interfaces and no problems: peer and client advertised and listening URLs are all on the same IP, differentiated by port.

Our other cluster is constrained in that all VMs are multihomed with separate administrative and data interfaces. Without going down the rabbit hole of our configuration, we are having trouble configuring etcd to pass healthchecks. My assumption so far has been that the peer IP is for exchanging all significant cluster traffic in terms of replicating data and RAFT comms. It seems that the client IP is used for healthchecks (etcdctl endpoint health and status) and for clients such as patroni to connect.

If this is correct, can I also confirm that if I configure a node to listen on all interfaces (with ETCD_LISTEN_CLIENT_URLS="0.0.0.0" and by advertising both client URLs) then clients such as patroni and etcdctl should be able to connect to either interface? I have configured this scenario, confirmed with ss that etcd is listening on 0.0.0.0:*, and confirmed with 'member list' that both interfaces are advertised - however, I find that I am failing healthchecks from other nodes despite the fact that they have a route to one or both interfaces and can successfully ping and curl to both peer and client ports.

Thanks for any insights!

[camac2025]

Any detail I can add here to encourage a response? Accurately reproducing our actual environment for replication would require a couple of VLANs and wireguard tunnels so I don't think it would be practical. I realise the solution may well be in the detail of how we have done this, but an answer to those questions would help I think:

• what in principle is the idea behind separating client and peer addresses?
• do healthchecks use the client address?
• should a node advertising two client addresses on different IPs and listening on both respond to client/healthcheck requests on either address or is there any sort of priority mechanism?

Thanks!

[camac2025]

One last bump - no-one able to shed any light on the design thinking here?

[vivekpatani]

Let me try helping you, but someone more informed might come along, and provide an even better perspective.

the design thinking here

The separation serves distinct architectural purposes as you've correctly identified:

• Peer URLs --listen-peer-urls and --initial-advertise-peer-urls (default: 2380): Used for Raft, handle cluster membership changes, leader election, log replication, and used by the rafthttp transport layer for inter-node comms, etc.

• Client URLs --listen-client-urls and --advertise-client-urls (default 2379): Used by client apps (like Patroni, etcdctl) to connect to etcd, handle gRPC and HTTP API requests for KV ops, serve health check endpoints /health, /livez, /readyz, etc.

It seems that the client IP is used for healthchecks (etcdctl endpoint health and status) and for clients such as patroni to connect.

Yes, health checks use client URLs. Looking at the health check implementation -

etcd/server/etcdserver/api/etcdhttp/health.go

Lines 58 to 73 in 52ce705

	// HandleHealth registers metrics and health handlers. it checks health by using v3 range request
	// and its corresponding timeout.
	func HandleHealth(lg *zap.Logger, mux *http.ServeMux, srv ServerHealth) {
	mux.Handle(PathHealth, NewHealthHandler(lg, func(ctx context.Context, excludedAlarms StringSet, serializable bool) Health {
	if h := checkAlarms(lg, srv, excludedAlarms); h.Health != "true" {
	return h
	}
	if h := checkLeader(lg, srv, serializable); h.Health != "true" {
	return h
	}
	return checkAPI(ctx, lg, srv, serializable)
	}))
	installLivezEndpoints(lg, mux, srv)
	installReadyzEndpoints(lg, mux, srv)
	}

are served on client URLs, hc can be serializable (local node only) or linearizable (requires cluster consensus). The checkAPI function performs health checks using Range requests.

either address or is there any sort of priority mechanism

If my understanding serves me right, there is no priority mechanism.

Source:

etcd/server/embed/etcd.go

Lines 657 to 670 in 52ce705

	sctxs = make(map[string]*serveCtx)
	for _, u := range append(cfg.ListenClientUrls, cfg.ListenClientHttpUrls...) {
	if u.Scheme == "http" || u.Scheme == "unix" {
	if !cfg.ClientTLSInfo.Empty() {
	cfg.logger.Warn("scheme is http or unix while key and cert files are present; ignoring key and cert files", zap.String("client-url", u.String()))
	}
	if cfg.ClientTLSInfo.ClientCertAuth {
	cfg.logger.Warn("scheme is http or unix while --client-cert-auth is enabled; ignoring client cert auth for this URL", zap.String("client-url", u.String()))
	}
	}
	if (u.Scheme == "https" || u.Scheme == "unixs") && cfg.ClientTLSInfo.Empty() {
	return nil, fmt.Errorf("TLS key/cert (--cert-file, --key-file) must be provided for client url %s with HTTPS scheme", u.String())
	}
	}

[camac2025]

Thanks for taking the time here. My Go is not really strong enough but this encourages me to try to get some answers from the code.

[vivekpatani]

Good luck, I can try and pitch in if you need additional help. @camac2025