Adopt CNI-aware conformance for networking (include Cilium KPR)

[batistein]

Summary

Running Cilium in kube-proxy-free (KPR) mode causes a small number of upstream SIG-Network and KaaS NetworkPolicy conformance tests to fail, despite the fact that the cluster’s user-visible networking works correctly. This is not unique to us: multiple vendors and upstream projects have had to skip or adjust these tests when using Cilium’s eBPF datapath instead of kube-proxy. I propose that SCS adopt a CNI-aware conformance policy, certifying a list of known-good CNIs (including Cilium KPR) and excluding a narrow, documented set of tests that are not portable across datapaths, or clearly marking them as optional — mirroring how the Cilium project itself handles certain upstream tests.
See: cilium/cilium#29524

Failures

The tests that fail are those tied to kube-proxy’s implementation details (e.g. checking kube-proxy’s own metrics endpoint or making timing-sensitive assumptions about Linux conntrack). These details do not affect everyday application connectivity or isolation.

Note: With KPR, Cilium replaces kube‑proxy and programs the kernel with eBPF. That changes how traffic is steered internally, not what users see from Kubernetes networking.

Examples of non‑portable tests

• kube-proxy metrics/health endpoints: Tests that call the kube-proxy /metrics or health endpoints will fail under KPR because kube-proxy isn't running. Cilium provides these functions differently.
• NodePort session affinity 'flip' tests: These tests rapidly toggle session affinity. Kube-proxy uses iptables, whereas Cilium tracks affinity in BPF maps and uses NodePort modes with different reply paths (e.g. SNAT and DSR). The semantics that users care about remain correct, but timing-specific expectations differ.
• HostPort conflict predicate tests: In KPR, HostPort is implemented by Cilium’s datapath scheduler/runtime, and the behaviour of host socket binding can differ without affecting HostPort functionality.
• SCTP NetworkPolicy tests: SCTP is optional and must be explicitly enabled in Cilium. If it is not enabled cluster-wide, SCTP tests should be skipped by design.

See:

• https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/#limitations
• KubeProxy metric tests fails with cilium + kubeproxy replacement kubernetes/kubernetes#126903

Common Issue in the Ecosystem

• Cilium upstream skips specific conformance tests while fixes are implemented
• Vendors report identical failures when using Cilium KPR (e.g. Giant Swarm tracks CNCF test failures attributable to Cilium behaviour).
• Conformance submissions adapt accordingly (e.g. the EKS Anywhere K8s 1.29 conformance PR, with discussions and adjustments around network conformance details).

Proposal (CNI‑aware conformance)

1. Publish an 'Approved CNI' list for SCS, including Cilium KPR.
2. Maintain a Skip/Optional list for each CNI, with a reason for each one and upstream reference. Initial list for Cilium KPR:
   • NodePort session-affinity flip tests (timing/conntrack assumptions).
   • HostPort conflict predicate tests (portmap versus datapath implementation).
   • SCTP-specific tests, unless SCTP is enabled in the product profile.
   • kube-proxy metric/health endpoint tests (component absent under KPR).
3. Document that user-visible semantics remain in scope, including pod-to-pod, DNS, ClusterIP/NodePort/LB reachability and L3/L4 NetworkPolicy.

[mbuechse]

@batistein Not sure we really want to drop SCTP when it can be activated with Cilium. Would it be possible for you to activate SCTP?

[fkr]

@batistein I tried pinging you via @janiskemper - we'd like to proceed on this and kinda need your input :)

[mbuechse]

@batistein @janiskemper Can you please respond here? I'm being told that not supporting SCTP is quite the loss. If it can be enabled, this would be desirable. What is your stance here?

[batistein]

Hi @mbuechse, sorry for being radio silent here — my capacity is currently very limited.

We evaluated whether we could enable SCTP, but unfortunately that’s not possible. In practice, SCTP is barely used nowadays; it’s considered a niche protocol. Since we're running fully with eBPF, only TCP, UDP, and ICMP are supported. Enabling SCTP would require an additional kernel module.

Furthermore, as far as I know, SCTP conntrack support is also limited, which means the implementation on the Cilium side is not fully covered either. See: cilium/cilium#20490

I also checked, and the common approach is to skip the SCTPConnectivity tests.