Skip to content

Flannel DaemonSet fails to start due to iptables-restore error during chain teardown #2305

@chymy

Description

@chymy

kubernetes version: 1.34.0
flannel version: 0.27.4
flannel-cni-plugin version: 1.8.0

When deploying Flannel with the provided configuration, the flannel pods fail to start with iptables-restore errors during the cleanup of previous iptables rules.

kube-flannel-cfg ConfigMap:

apiVersion: v1
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "172.16.0.0/12",
      "EnableNFTables": false,
      "SubnetLen": 24,
      "Backend": {
        "Type": "vxlan"
      }
    }
kind: ConfigMap
metadata:
  labels:
    app: flannel
    k8s-app: flannel
    tier: node
  name: kube-flannel-cfg
  namespace: kube-system

DaemonSet yaml:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: flannel
    k8s-app: flannel
    tier: node
  name: kube-flannel
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: flannel
      k8s-app: flannel
  template:
    metadata:
      labels:
        app: flannel
        k8s-app: flannel
        tier: node
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      containers:
      - args:
        - --ip-masq
        - --kube-subnet-mgr
        command:
        - /opt/bin/flanneld
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: EVENT_QUEUE_DEPTH
          value: "5000"
        - name: CONT_WHEN_CACHE_NOT_READY
          value: "false"
        image: ...
        name: kube-flannel
        securityContext:
          capabilities:
            add:
            - NET_ADMIN
            - NET_RAW
          privileged: false
        volumeMounts:
        - mountPath: /run/flannel
          name: run
        - mountPath: /etc/kube-flannel/
          name: flannel-cfg
        - mountPath: /run/xtables.lock
          name: xtables-lock
      hostNetwork: true
      initContainers:
      - args:
        - -f
        - /flannel
        - /opt/cni/bin/flannel
        command:
        - cp
        image: ...
        name: install-cni-plugin
        volumeMounts:
        - mountPath: /opt/cni/bin
          name: cni-plugin
      - args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        command:
        - cp
        image: ...
        name: install-cni
        volumeMounts:
        - mountPath: /etc/cni/net.d
          name: cni
        - mountPath: /etc/kube-flannel/
          name: flannel-cfg
      priorityClassName: system-node-critical
      serviceAccountName: flannel
      tolerations:
      - effect: NoSchedule
        operator: Exists
      volumes:
      - hostPath:
          path: /run/flannel
        name: run
      - hostPath:
          path: /opt/cni/bin
        name: cni-plugin
      - hostPath:
          path: /etc/cni/net.d
        name: cni
      - configMap:
          name: kube-flannel-cfg
        name: flannel-cfg
      - hostPath:
          path: /run/xtables.lock
          type: FileOrCreate
        name: xtables-lock
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 20%
    type: RollingUpdate

Log Output:

I1128 08:54:03.668577       1 iptables.go:50] Starting flannel in iptables mode...
W1128 08:54:03.668609       1 main.go:577] no subnet found for key: FLANNEL_NETWORK in file: /run/flannel/subnet.env
W1128 08:54:03.668621       1 main.go:577] no subnet found for key: FLANNEL_SUBNET in file: /run/flannel/subnet.env
W1128 08:54:03.668632       1 main.go:612] no subnet found for key: FLANNEL_IPV6_NETWORK in file: /run/flannel/subnet.env
W1128 08:54:03.668658       1 main.go:612] no subnet found for key: FLANNEL_IPV6_SUBNET in file: /run/flannel/subnet.env
I1128 08:54:03.668666       1 iptables.go:101] Current network or subnet (172.16.0.0/12, 172.16.3.0/24) is not equal to previous one (0.0.0.0/0, 0.0.0.0/0), trying to recycle old iptables rules
I1128 08:54:03.757999       1 iptables_restore.go:94] trying to run with payload *nat
-D POSTROUTING -m comment --comment "flanneld masq" -j FLANNEL-POSTRTG
-D FLANNEL-POSTRTG -s 0.0.0.0/0 -d 0.0.0.0/0 -m comment --comment "flanneld masq" -j RETURN
-D FLANNEL-POSTRTG -s 0.0.0.0/0 -d 0.0.0.0/0 -m comment --comment "flanneld masq" -j RETURN
COMMIT
E1128 08:54:03.759545       1 iptables.go:450] Failed to teardown iptables: unable to teardown iptables: unable to run iptables-restore (, iptables-restore: line 4 failed: Bad rule (does a matching rule exist in that chain?).
): exit status 1
E1128 08:54:03.759574       1 main.go:414] Failed to setup masq rules, unable to teardown iptables: unable to run iptables-restore (, iptables-restore: line 4 failed: Bad rule (does a matching rule exist in that chain?).
): exit status 1
I1128 08:54:03.759619       1 main.go:503] Stopping shutdownHandler...

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions