timed out publish local member to cluster through raft

[godbestway]

Hi etcd community,

I’m trying to deploy an etcd node inside a Kubernetes cluster and have it join an existing etcd cluster that is running on three external VMs. To expose the external etcd service, I set up a load balancer in front of those VMs.

However, when my etcd node in Kubernetes attempts to connect to the external cluster through the load balancer, it always times out.

I got the logs like this
`
{"level":"info","ts":"2025-07-19T15:56:39.819778Z","caller":"etcdserver/server.go:798","msg":"initialized peer connections; fast-forwarding election ticks","local-member-id":"9c590d54aba2444b","forward-ticks":8,"forward-duration":"800ms","election-ticks":10,"election-timeout":"1s","active-remote-members":2}
{"level":"info","ts":"2025-07-19T15:56:39.822073Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"912320f6905136a4"}
{"level":"info","ts":"2025-07-19T15:56:39.822103Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"9c590d54aba2444b","remote-peer-id":"912320f6905136a4"}
{"level":"info","ts":"2025-07-19T15:56:39.822870Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"9c590d54aba2444b","remote-peer-id":"912320f6905136a4"}
{"level":"info","ts":"2025-07-19T15:56:39.822913Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"9c590d54aba2444b","remote-peer-id":"25f42ca9e477cc46"}
{"level":"info","ts":"2025-07-19T15:56:39.919419Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"9c590d54aba2444b [term: 0] received a MsgHeartbeat message with higher term from 912320f6905136a4 [term: 21423]"}
{"level":"info","ts":"2025-07-19T15:56:39.919457Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"9c590d54aba2444b became follower at term 21423"}
{"level":"info","ts":"2025-07-19T15:56:39.919476Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 9c590d54aba2444b elected leader 912320f6905136a4 at term 21423"}
{"level":"warn","ts":"2025-07-19T15:56:46.742568Z","caller":"etcdserver/server.go:2165","msg":"failed to publish local member to cluster through raft","local-member-id":"9c590d54aba2444b","local-member-attributes":"{Name:dev-cluster-etcd ClientURLs:[https://dev-cluster-etcd.xxx.de:2379]}","request-path":"/0/members/9c590d54aba2444b/attributes","publish-timeout":"7s","error":"etcdserver: request timed out, possibly due to connection lost"}
{"level":"warn","ts":"2025-07-19T15:56:53.743527Z","caller":"etcdserver/server.go:2165","msg":"failed to publish local member to cluster through raft","local-member-id":"9c590d54aba2444b","local-member-attributes":"{Name:dev-cluster-etcd ClientURLs:[https://dev-cluster-etcd.xxx.de:2379]}","request-path":"/0/members/9c590d54aba2444b/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}

`

These are my Settings

apiVersion: v1
data:
  ETCD_ADVERTISE_CLIENT_URLS: https://dev-cluster-etcd.xxx.de:2379
  ETCD_INITIAL_ADVERTISE_PEER_URLS: https://dev-cluster-etcd.xxx.de:2380
  ETCD_INITIAL_CLUSTER: dc01e-dc=https://dc01e-dc.xxx.de:2380,fs01e-dc=https://fs01e-dc.xxx.de:2380,int01e-dc=https://int01e-dc.xxx.de:2380,dev-cluster-etcd=https://dev-cluster-etcd.xxx.de:2380
  ETCD_NAME: dev-cluster-etcd
kind: ConfigMap
metadata:
  name: env-vv-etcd

apiVersion: v1
kind: Service
metadata:
  annotations:
    loadbalancer.openstack.org/keep-floatingip: "true"
    service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
  labels:
    app.kubernetes.io/name: etcd
    app.kubernetes.io/version: 3.5.21
  name: etcd
spec:
  externalTrafficPolicy: Local
  loadBalancerIP: ""
  ports:
  - name: client
    port: 2379
    targetPort: client
  - name: peer
    port: 2380
    targetPort: peer
  publishNotReadyAddresses: true
  selector:
    app.kubernetes.io/name: etcd
  sessionAffinity: None
  type: LoadBalancer

`

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app.kubernetes.io/name: etcd
  name: dev-cluster-etcd
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: etcd
  template:
    metadata:
      labels:
        app.kubernetes.io/name: etcd
    spec:
      affinity:
        nodeAffinity: null
        podAffinity: null
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: etcd
              topologyKey: kubernetes.io/hostname
            weight: 100
      automountServiceAccountToken: false
      containers:
      - env:
        - name: MY_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: ETCDCTL_API
          value: "3"
        - name: ETCD_ON_K8S
          value: "no"
        - name: ETCD_INITIAL_CLUSTER_TOKEN
          value: "dev"
        - name: ETCD_ROOT_PASSWORD
          value: "test-etcd"
        - name: ETCD_DATA_DIR
          value: /bitnami/etcd/data
        - name: ETCD_LISTEN_CLIENT_URLS
          value: https://0.0.0.0:2379
        - name: ETCD_LISTEN_PEER_URLS
          value: https://0.0.0.0:2380
        - name: ETCD_TRUSTED_CA_FILE
          value: /etc/etcd/certs/ca/bundle.pem
        - name: ETCD_PEER_TRUSTED_CA_FILE
          value: /etc/etcd/certs/ca/bundle.pem
        - name: ETCD_INITIAL_CLUSTER_STATE
          value: existing
        - name: ETCD_CERT_FILE
          value: /opt/bitnami/etcd/certs/client/star_xxx_de_ecc_pub.pem
        - name: ETCD_KEY_FILE
          value: /opt/bitnami/etcd/certs/client/star_xxx_de.pem
        - name: ETCD_CLIENT_CERT_AUTH
          value: "true"
        - name: ETCD_PEER_CERT_FILE
          value: /opt/bitnami/etcd/certs/peer/star_xxx_de_ecc_pub.pem
        - name: ETCD_PEER_KEY_FILE
          value: /opt/bitnami/etcd/certs/peer/star_xxx_de.pem
        - name: ETCD_PEER_CLIENT_CERT_AUTH
          value: "true"
        - name: ETCD_AUTH_TOKEN_TTL
          value: "3600"
        envFrom:
        - configMapRef:
            name: env-vv-etcd
        image: docker.io/bitnami/etcd:3.5.21-debian-12-r1
        imagePullPolicy: Always
        livenessProbe:
          exec:
            command:
            - /opt/bitnami/scripts/etcd/healthcheck.sh
          failureThreshold: 5
          initialDelaySeconds: 60
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 5
        name: etcd
        ports:
        - containerPort: 2379
          name: client
          protocol: TCP
        - containerPort: 2380
          name: peer
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - /opt/bitnami/scripts/etcd/healthcheck.sh
          failureThreshold: 5
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources:
          limits:
            cpu: 500m
            memory: 512Mi
          requests:
            cpu: 100m
            memory: 256Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          privileged: false
          readOnlyRootFilesystem: true
          runAsGroup: 1001
          runAsNonRoot: true
          runAsUser: 1001
          seLinuxOptions: {}
          seccompProfile:
            type: RuntimeDefault
        volumeMounts:
        - mountPath: /opt/bitnami/etcd/conf/
          name: empty-dir
          subPath: app-conf-dir
        - mountPath: /tmp
          name: empty-dir
          subPath: tmp-dir
        - mountPath: /bitnami/etcd/data
          name: data
          subPath: etcd-data
        - mountPath: /opt/bitnami/etcd/certs/client/
          name: etcd-client-certs
          readOnly: true
        - mountPath: /opt/bitnami/etcd/certs/peer/
          name: etcd-peer-certs
          readOnly: true
        - mountPath: /etc/etcd/
          name: etcd-certs
          readOnly: true
        - mountPath: /etc/etcd/certs/ca
          name: cacerts
          readOnly: true
      securityContext:
        fsGroup: 1001
        fsGroupChangePolicy: Always
        runAsUser: 1001
        supplementalGroups: []
        sysctls: []
      topologySpreadConstraints:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/name: etcd
        maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
      volumes:
      - emptyDir: {}
        name: empty-dir
      - name: etcd-client-certs
        secret:
          defaultMode: 256
          secretName: etcd-certs
      - name: etcd-peer-certs
        secret:
          defaultMode: 256
          secretName: etcd-certs
      - name: etcd-certs
        secret:
          secretName: etcd-certs
      - configMap:
          name: vv-cacerts
        name: cacerts
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 8Gi
`

Could someone help me understand what might be causing the timeout? And I have already checked the network issues, there's no deloyed networkpolicies and the firewalld on the other side allow everything. So i am really confused about this issue.

And I also tried many combination of the port 2380, like this
https://dev-cluster-etcd-0.etcd-headless.dev-etcd.svc.cluster.local:2380
But it also not work

Thanks in advance for any advice!