help request: request etcd endpoint timeout error

[idb-root]

Current Behavior

/usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua init
/usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd
request etcd endpoint "http://apisix-etcd:2379/v3/kv/put" error, timeout
trying to initialize the data of etcd
Warning! Request etcd endpoint 'http://apisix-etcd:2379/v3/kv/put' error, timeout, retry time=1
Warning! Request etcd endpoint 'http://apisix-etcd:2379/v3/kv/put' error, timeout, retry time=1
Warning! Request etcd endpoint 'http://apisix-etcd:2379/v3/kv/put' error, timeout, retry time=2

In this Kubernetes cluster, APISIX 3.14.1 is deployed via Helm, but one of the APISIX pods cannot start and keeps crashing. The pod logs show:

• apisix.lua init_etcd
• request to http://apisix-etcd:2379/v3/kv/put times out
• container exits with code 1 → CrashLoopBackOff

At the same time, if I run a temporary/debug pod in the same namespace and connect to apisix-etcd:2379, the connection works and etcdctl put/get succeeds.

So the endpoints are reachable, but APISIX startup (init_etcd) still fails due to request timeouts.

Expected Behavior

No response

Error Logs

No response

Steps to Reproduce

apiVersion: apps/v1
kind: Deployment
metadata:
  name: apisix
  namespace: default
  labels:
    app.kubernetes.io/instance: apisix
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: apisix
    app.kubernetes.io/version: 3.2.0
    helm.sh/chart: apisix-1.3.1
  annotations:
    deployment.kubernetes.io/revision: "<REDACTED>"
    meta.helm.sh/release-name: apisix
    meta.helm.sh/release-namespace: default
spec:
  replicas: 2
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600
  selector:
    matchLabels:
      app.kubernetes.io/instance: apisix
      app.kubernetes.io/name: apisix
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: apisix
        app.kubernetes.io/name: apisix
      annotations:
        checksum/config: "<REDACTED>"
        redeploy-timestamp: "<REDACTED>"
    spec:
      serviceAccountName: default
      dnsPolicy: ClusterFirst
      terminationGracePeriodSeconds: 30

      initContainers:
        - name: wait-etcd
          image: "<REDACTED_REGISTRY>/busybox:1.28"
          imagePullPolicy: IfNotPresent
          command:
            - sh
            - -c
            - >
              until nc -z apisix-etcd.default.svc.cluster.local 2379; do
                echo waiting for etcd `date`;
                sleep 2;
              done;

      containers:
        - name: apisix
          image: "<REDACTED_REGISTRY>/apisix:3.14.1-debian"
          imagePullPolicy: Always
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 30"]

          ports:
            - name: http
              containerPort: 9080
              protocol: TCP
            - name: tls
              containerPort: 9443
              protocol: TCP
            - name: admin
              containerPort: 9180
              protocol: TCP

          readinessProbe:
            tcpSocket:
              port: 9080
            initialDelaySeconds: 90
            periodSeconds: 30
            timeoutSeconds: 1
            successThreshold: 1
            failureThreshold: 9

          resources: {}

          volumeMounts:
            - name: apisix-config
              mountPath: /usr/local/apisix/conf/config.yaml
              subPath: config.yaml

            # Custom plugins / extra mounted files (all from ConfigMaps)
            - name: plugin-cm-1
              mountPath: /usr/local/apisix/apisix/plugins/<REDACTED_PLUGIN_1>.lua
              subPath: <REDACTED_PLUGIN_1>.lua
            - name: plugin-cm-2
              mountPath: /usr/local/apisix/apisix/plugins/<REDACTED_PLUGIN_2>.lua
              subPath: <REDACTED_PLUGIN_2>.lua
            - name: plugin-cm-3
              mountPath: /usr/local/apisix/apisix/plugins/<REDACTED_PLUGIN_3>.lua
              subPath: <REDACTED_PLUGIN_3>.lua
            - name: plugin-cm-4
              mountPath: /usr/local/apisix/apisix/plugins/<REDACTED_PLUGIN_4>.lua
              subPath: <REDACTED_PLUGIN_4>.lua
            - name: plugin-cm-5
              mountPath: /usr/local/apisix/apisix/plugins/<REDACTED_PLUGIN_5>.lua
              subPath: <REDACTED_PLUGIN_5>.lua
            - name: extra-file-cm
              mountPath: /usr/local/openresty/lualib/resty/<REDACTED_FILE>.lua
              subPath: <REDACTED_FILE>.lua

      volumes:
        - name: apisix-config
          configMap:
            name: apisix
            defaultMode: 420

        - name: plugin-cm-1
          configMap:
            name: apisix-plugin-new
            defaultMode: 420
        - name: plugin-cm-2
          configMap:
            name: apisix-plugin-new
            defaultMode: 420
        - name: plugin-cm-3
          configMap:
            name: apisix-plugin-new
            defaultMode: 420
        - name: plugin-cm-4
          configMap:
            name: apisix-plugin-new
            defaultMode: 420
        - name: plugin-cm-5
          configMap:
            name: apisix-plugin-new
            defaultMode: 420
        - name: extra-file-cm
          configMap:
            name: apisix-plugin-new
            defaultMode: 420

Environment

• APISIX version (run apisix version):
• Operating system (run uname -a):
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[Baoyuantop]

if I run a temporary/debug pod in the same namespace and connect to apisix-etcd:2379, the connection works and etcdctl put/get succeeds.

What is the approximate processing time for the request? Have you tried testing etcd connectivity from the APISIX pod?