resolve implementation-defined behavior around hairpin packets in CNP

[danwinship]

There is some unspecified/implementation-defined behavior involving hairpin packets in NetworkPolicy, which is currently also implementation-defined in ClusterNetworkPolicy... We should try to make them well-defined.

Consider a Service X with endpoint Pods A and B, which are both fully isolated.

1. Simple hairpinning: If pod A connects to its own podIP, does this succeed despite NetworkPolicy? ("Yes"?)
2. Service hairpinning: If pod A connects to Service X and the service proxy redirects the packet back to pod A, does this succeed despite NetworkPolicy? ("Yes"?) (EDIT: or "No"?)
3. Service hairpinning miss: If pod A connects to Service X and the service proxy redirects the packet to pod B, does this succeed despite NetworkPolicy? ("No"?)
4. Service masquerading: If, due to hairpinning or other reasons, the service proxy decides to masquerade a packet, should that packet then be matched against rules for its original source IP or the masqueraded IP? ("Uh....")

I think everyone agrees the answer to the first question is "yes" and the third one is "no". This means the answer to the second question must be inconsistent with one of them. I think these days, everyone says that the answer to the second question is "yes" (obeying the general rule that "NetworkPolicy doesn't see Services", and thus the exception for hairpinned packets from the first case applies). It's possible I'm wrong and Cilium does things differently though? (From some quick searches, it looks like Cilium used to do things differently, but now does things the same way everyone else does.)

The fourth case seems unrelated, but I believe that Calico and kube-network-policies don't specifically handle the second case, and they only get it "right" because kube-proxy is masquerading the packet to the node IP, and so then it hits the magic "all packets from the local node are allowed" rule for kubelet probes. So it's worth thinking about whether this is a bug or a feature...

[fasaxc]

From memory (could do with double checking on a live system to make sure), I think this depends on Calico's dataplane configuration:

• Calico + kube-proxy: the packet leaves the pod, then, in order,
  • kube-proxy DNATs the traffic
  • Calico applies policy: policy sees src == dest == pod.
  • kube-proxy SNATs the traffic
• Calico in BPF mode: if connect-time load balancing is enabled, traffic doesn't even leave the pod, it would be short circuited and connect back to itself within the pod. No policy.
• Calico in BPF mode: CTLB disabled, traffic leaves the pod, hits our BPF program; I think we apply DNAT, then policy, then SNAT.

If a proxy intercepts traffic, it'll depend on where it re-injects the packet (and which parts of the tuple it preserves when it does so).

[danwinship]

@fasaxc that sounds like "yes" for case 2 in all 3 configurations though?

[fasaxc]

Service hairpinning: If pod A connects to Service X and the service proxy redirects the packet back to pod A, does this succeed despite NetworkPolicy? ("Yes"?)

Assuming "service proxy" is typically kube-proxy...

I think, in *tables mode (and BPF mode with CTLB disabled) we do apply policy to this traffic. That doesn't often cause confusion because typically all the pods backing a service are labeled the same way (a deployment) and you'd naturally use that labeling to make both your service and your netpol.

It's only the CTLB case where it gets short circuited.

Service masquerading: If, due to hairpinning or other reasons, the service proxy decides to masquerade a packet, should that packet then be matched against rules for its original source IP or the masqueraded IP? ("Uh....")

In Calico's case, in *tables mode, the source IP that we see depends on whether the SNAT happened on this node or on a remote node. If it was a remote node then the source IP is lost. If it was this node, policy happens before SNAT so we see the original IP.

In BPF mode, when we're doing the job of kube-proxy, we preserve source IP so that policy always sees the original source IP. Of course, if there was another proxy/ingress/gateway involved then it may terminate traffic and we'd see its IP as source. (And external load balancers may not preserve source IP.)

[danwinship]

I think, in *tables mode (and BPF mode with CTLB disabled)

(CTLB = "connect-time load balancing" = implementing service proxying by intercepting connect() calls rather than DNAT-ing.)

we do apply policy to [hairpin service] traffic.

I'm starting to think that applying network policy to hairpin service connections actually makes more sense than not applying it (particularly given that NP has to be applied in the non-hairpin case of the same service, case 3).

In the logical model of k8s networking, service proxying and NetworkPolicy are things that happen outside of the pod network namespace. PodIP-to-podIP traffic isn't subject to NetworkPolicy because the packet never leaves the pod network namespace, but podIP-to-clusterIP-to-podIP should be subject to NP, because it does leave the pod network namespace. (Even if services/NP are actually implemented inside the pod netns, they are still logically outside of it, so the implementation should just make the podIP-to-podIP and podIP-to-serviceIP-to-podIP cases behave differently.)

In Calico's case, in *tables mode, the source IP that we see depends on whether the SNAT happened on this node or on a remote node. If it was a remote node then the source IP is lost. If it was this node, policy happens before SNAT so we see the original IP.

Oh, oh, right... I was thinking of openshift-sdn's behavior, where SNAT happened before NP because of the way the split between iptables rules and OVS flows worked.

OK, so in that case, I feel like:

• Simple hairpinning: podIP-to-same-podIP is never subject to NetworkPolicy. (If you implement NetworkPolicy outside the pod network namespace, this behavior is automatic, but if you implement NetworkPolicy inside the pod network namespace, you may need to add an exception for this case, so it "looks like" you're implementing NetworkPolicy outside the namespace.)
• Service hairpinning: podIP-to-clusterIP-to-same-podIP is subject to the NetworkPolicy rules that would match podIP-to-same-podIP, even though direct podIP-to-same-podIP wouldn't be. (Again, this would be the automatic/default behavior if services and NP are implemented outside the pod network namespace, but should be implemented as a special case when services/NP are implemented inside the pod network namespace.)
• Service hairpinning miss: podIP-to-clusterIP-to-different-podIP is obviously subject to NP.
• Service masquerading: In the specific case of masquerading podIP-to-clusterIP-to-same-podIP, the implementation must make sure that it matches the packet against podIP-to-same-podIP rules, not against maqueradedIP-to-pod-IP rules. In other cases (even podIP-to-NodePort-to-same-podIP) it is implementation-defined whether the packet is matched against the original or masqueraded IP.

[bowei]

My take is that we should try to think about it in terms of identities to avoid getting anchored on what implementations do. Also, luckily this lines up with what you have written above...

If you look at it from the identity level, 1 + 2 for sure is OK because the (unwritten?) rule is that you should always be able to communicate with yourself.

I'm a little confused why case 3 involves hair pinning. Isn't this just the normal case?

For 4), it sounds like it is feasible for us to require all implementations to respect the identity abstraction? At least within a cluster, it seems we should try to make this possible.

[danwinship]

Case 3 involves potential hairpinning (A→X→A) that didn't happen (it went A→X→B instead, hence "hairpinning miss").

The point is that it's weird that in the example given, a connection from A to X will succeed half the time and fail half the time, depending on whether it ends up as A→X→A or A→X→B.

Though I guess that can happen even without hairpinning if you isolate B and not A for some reason. (Imagine rolling out an update and accidentally changing the labels on the pods when you do.) Then C→X→A succeeds and C→X→B fails for all clients. So I guess "2 and 3 should be consistent" is a weak argument.

[danwinship]

If you look at it from the identity level, 1 + 2 for sure is OK because the (unwritten?) rule is that you should always be able to communicate with yourself.

Hm... That's not the unwritten rule I would have said. 🙂

I feel that in 1, the unwritten rule is that NetworkPolicy is something that happens outside the pod, so traffic that doesn't leave the pod is not affected by NetworkPolicy. (Alternatively, and I think equivalently, the packet neither enters nor leaves the pod, so neither ingress nor egress rules match it.)

But by that argument, in case 2, NetworkPolicy should be enforced. Except there's an unwritten rule that "Service proxying doesn't affect NetworkPolicy" or "A connection from A-to-Service-to-B is allowed if and only if a connection from A-to-B is allowed", which implies that NetworkPolicy shouldn't be enforced. Depending on how/where you implement the exception from the first case...

[npinaeva]

I am not sure about
case2: Service hairpinning: If pod A connects to Service X and the service proxy redirects the packet back to pod A, does this succeed despite NetworkPolicy? ("Yes"?)
Why would we want this to work? If the pod is using k8s service abstraction instead of its own IP then k8s network policies can apply to that? And then we could have consistent "No" behaviour for case 2 and case 3? It is also more consistent with the load balancing behaviour, because if the traffic goes to another service backend pod it will be denied, so the service connections will be inconsistent depending on which backend was chosen.
Ignoring that it could break someone relying on the existing behaviour, but I think the point of this issue is to figure out the desired behaviour first

[danwinship]

Ignoring that it could break someone relying on the existing behaviour

We're not changing the behavior of NetworkPolicy though, just ClusterNetworkPolicy.

I think I may have been confused about what the standard behavior was in the service hairpin case, and it seems like everyone might be happier with "network policy gets enforced, which means that pod-to-same-pod and pod-to-service-IP-to-same-pod behave differently, which is fine".