UPF Pod stuck in PodInitializing state

[nobody8v]

The UPF Pod is persistently stuck in PodInitializing state. This happened after upgrading the cluster to Kubernetes version v1.31.3+rke2r1.

I can't access to logs of any container in the pod and these are the Pod Events:

Events:
  Type     Reason                           Age                From               Message
  ----     ------                           ----               ----               -------
  Normal   Scheduled                        13s                default-scheduler  Successfully assigned aether-5gc/upf-0 to new-k8s-original-aether
  Normal   AddedInterface                   12s                multus             Add eth0 [10.42.0.68/32] from k8s-pod-network
  Normal   AddedInterface                   12s                multus             Add access [192.168.252.3/24] from aether-5gc/access-net
  Normal   AddedInterface                   12s                multus             Add core [192.168.250.3/24] from aether-5gc/core-net
  Normal   Pulled                           12s                kubelet            Container image "registry.aetherproject.org/proxy/omecproject/upf-epc-bess:rel-2.0.0" already present on machine
  Normal   Created                          12s                kubelet            Created container bess-init
  Normal   Started                          11s                kubelet            Started container bess-init
  Warning  FailedToRetrieveImagePullSecret  10s (x2 over 12s)  kubelet            Unable to retrieve some image pull secrets (aether.registry); attempting to pull the image may not succeed.
  Normal   Pulled                           10s                kubelet            Container image "registry.aetherproject.org/proxy/omecproject/upf-epc-bess:rel-2.0.0" already present on machine
  Normal   Created                          10s                kubelet            Created container bessd
  Normal   Started                          10s                kubelet            Started container bessd

I also attach the SMF and kube-proxy logs, that maybe are helpful. The IP that appears in SMF logs is the svc of the UPF

SMF logs

2025-04-25T18:25:15.141Z        INFO    message/send.go:152     sent PFCP Association Request to NodeID[10.43.63.31]    {"component": "SMF", "category": "PFCP"}
2025-04-25T18:25:25.152Z        INFO    message/send.go:152     sent PFCP Association Request to NodeID[10.43.63.31]    {"component": "SMF", "category": "PFCP"}
2025-04-25T18:25:35.163Z        INFO    message/send.go:152     sent PFCP Association Request to NodeID[10.43.63.31]    {"component": "SMF", "category": "PFCP"}
2025-04-25T18:25:45.175Z        INFO    message/send.go:152     sent PFCP Association Request to NodeID[10.43.63.31]    {"component": "SMF", "category": "PFCP"}
2025-04-25T18:25:55.185Z        INFO    message/send.go:152     sent PFCP Association Request to NodeID[10.43.63.31]    {"component": "SMF", "category": "PFCP"}

kube-proxy logs

        error adding entry 10.43.97.37,sctp:38412, error: exit status 1 (ipset v7.16: Syntax error: cannot parse 'sctp' as a protocol
        )
 > ipSetEntry="10.43.97.37,sctp:38412" ipSet="KUBE-CLUSTER-IP"
E0425 19:09:01.373602       1 ipset.go:180] "Failed to add ip set entry to ip set" err=<
        error adding entry 10.43.31.218,sctp:38412, error: exit status 1 (ipset v7.16: Syntax error: cannot parse 'sctp' as a protocol
        )
 > ipSetEntry="10.43.31.218,sctp:38412" ipSet="KUBE-CLUSTER-IP"
E0425 19:09:31.338130       1 proxier.go:1445] "Failed to execute iptables-restore" err=<
        exit status 2: Warning: Extension MARK revision 0 not supported, missing kernel module?
        ip6tables-restore v1.8.9 (nf_tables): unknown option "--xor-mark"
        Error occurred at line: 11
        Try `ip6tables-restore -h' or 'ip6tables-restore --help' for more information.
 > ipFamily="IPv6" rules="*nat\n:KUBE-SERVICES - [0:0]\n:KUBE-POSTROUTING - [0:0]\n:KUBE-NODE-PORT - [0:0]\n:KUBE-LOAD-BALANCER - [0:0]\n:KUBE-MARK-MASQ - [0:0]\n-A KUBE-SERVICES -s ::1/128 -j RETURN\n-A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT\n-A KUBE-LOAD-BALANCER -j KUBE-MARK-MASQ\n-A KUBE-POSTROUTING -m mark ! --mark 0x00004000/0x00004000 -j RETURN\n-A KUBE-POSTROUTING -j MARK --xor-mark 0x00004000\n-A KUBE-POSTROUTING -m comment --comment \"kubernetes service traffic requiring SNAT\" -j MASQUERADE --random-fully\n-A KUBE-MARK-MASQ -j MARK --or-mark 0x00004000\nCOMMIT\n*filter\n:KUBE-FORWARD - [0:0]\n:KUBE-NODE-PORT - [0:0]\n:KUBE-PROXY-FIREWALL - [0:0]\n:KUBE-SOURCE-RANGES-FIREWALL - [0:0]\n:KUBE-IPVS-FILTER - [0:0]\n:KUBE-IPVS-OUT-FILTER - [0:0]\n-A KUBE-SOURCE-RANGES-FIREWALL -j DROP\n-A KUBE-FORWARD -m comment --comment \"kubernetes forwarding rules\" -m mark --mark 0x00004000/0x00004000 -j ACCEPT\n-A KUBE-FORWARD -m comment --comment \"kubernetes forwarding conntrack rule\" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT\n-A KUBE-NODE-PORT -m comment --comment \"Kubernetes health check node port\" -m set --match-set KUBE-6-HEALTH-CHECK-NODE-PORT dst -j ACCEPT\n-A KUBE-IPVS-FILTER -m set --match-set KUBE-6-LOAD-BALANCER dst,dst -j RETURN\n-A KUBE-IPVS-FILTER -m set --match-set KUBE-6-CLUSTER-IP dst,dst -j RETURN\n-A KUBE-IPVS-FILTER -m set --match-set KUBE-6-EXTERNAL-IP dst,dst -j RETURN\n-A KUBE-IPVS-FILTER -m set --match-set KUBE-6-EXTERNAL-IP-LOCAL dst,dst -j RETURN\n-A KUBE-IPVS-FILTER -m set --match-set KUBE-6-HEALTH-CHECK-NODE-PORT dst -j RETURN\n-A KUBE-IPVS-FILTER -m conntrack --ctstate NEW -m set --match-set KUBE-6-IPVS-IPS dst -j REJECT\nCOMMIT\n"
E0425 19:09:31.496568       1 ipset.go:180] "Failed to add ip set entry to ip set" err=<
        error adding entry 10.42.0.49,sctp:38412,10.42.0.49, error: exit status 1 (ipset v7.16: Syntax error: cannot parse 'sctp' as a protocol
        )
 > ipSetEntry="10.42.0.49,sctp:38412,10.42.0.49" ipSet="KUBE-LOOP-BACK"
E0425 19:09:31.497525       1 ipset.go:180] "Failed to add ip set entry to ip set" err=<
        error adding entry 10.42.0.54,sctp:38412,10.42.0.54, error: exit status 1 (ipset v7.16: Syntax error: cannot parse 'sctp' as a protocol
        )
 > ipSetEntry="10.42.0.54,sctp:38412,10.42.0.54" ipSet="KUBE-LOOP-BACK"
E0425 19:09:31.498495       1 ipset.go:180] "Failed to add ip set entry to ip set" err=<
        error adding entry 192.168.12.16,sctp:38412, error: exit status 1 (ipset v7.16: Syntax error: cannot parse 'sctp' as a protocol
        )
 > ipSetEntry="192.168.12.16,sctp:38412" ipSet="KUBE-EXTERNAL-IP"
E0425 19:09:31.503230       1 ipset.go:180] "Failed to add ip set entry to ip set" err=<
        error adding entry 10.43.97.37,sctp:38412, error: exit status 1 (ipset v7.16: Syntax error: cannot parse 'sctp' as a protocol
        )
 > ipSetEntry="10.43.97.37,sctp:38412" ipSet="KUBE-CLUSTER-IP"
E0425 19:09:31.503543       1 ipset.go:180] "Failed to add ip set entry to ip set" err=<
        error adding entry 10.43.31.218,sctp:38412, error: exit status 1 (ipset v7.16: Syntax error: cannot parse 'sctp' as a protocol
        )
 > ipSetEntry="10.43.31.218,sctp:38412" ipSet="KUBE-CLUSTER-IP"

Thanks in advance for the help.

[samanthacs01]

I have had the same problem and I still haven't been able to find a solution

[gab-arrobo]

From the logs (upf-epc-bess:rel-2.0.0), I see you are not using the latest version of OnRamp. Can you please give it a try using the latest version (i.e., Helm Charts version 2.2.6)?

[nobody8v]

I will try it and share the response. Thanks you!

[gab-arrobo]

@nobody8v, any update about this issue?

[nobody8v]

We update to the latest version of OnRamp and it seems to work

[gab-arrobo]

We update to the latest version of OnRamp and it seems to work

Then, is it ok to close the issue?