clarify connection-based vs packet-based behavior

[danwinship]

NetworkPolicy was originally intended to apply only to connection establishment, not to every packet in a connection. However, like so many other things, this was not documented, and there is inconsistency between implementations currently. (For example, I know that kube-network-policies only checks NP on connection establishment, but ovn-kubernetes checks it on every packet.)

Only enforcing on connection establishment is more efficient, which is probably why we intended it to work that way. But there are users who definitely want the opposite semantics ("when I change policies, any existing connections that are no longer in-policy should be dropped"). That doesn't actually necessarily imply checking every packet: you could do it by deleting conntrack records for out-of-policy connections when policies change. (I think that might actually be what OVN does.)

We should probably say:

• ANP should use packet-based semantics.
• NP implementations may use either semantics
• Future DNP or whatever should be consistent with ANP.

?

(cc @bowei re: #303 (comment))

[aojea]

ANP should use packet-based semantics.

I think we should make a larger study of the ecosystem to understand how this is widely implemented across the industry, if per example, AWS, Azure, GCE, Cisco, Juniper, ... implement network policies that are connection based, I do not see why we need to implement something different

[aojea]

This is the classic Stateful vs Stateless debate on firewalling.
Most network policy implementations out there are already stateful for performance reasons, keeping track of a connection from start to finish instead of inspecting every single packet.

I think that the real question isn't if they're stateful (connection vs packet based), but rather how a stateful system should react when an administrator changes the rules in the middle of an active connection.

I don't think we have to force a slow, per-packet check to make this happen. A stateful system can still handle this efficiently. When a policy changes, the implementation can simply "forget" the state of any existing connections that are no longer allowed. The next time a packet from that old connection shows up, the system will see it as "new," check it against the new rules, and block it.

This way, administrators get the immediate enforcement they expect, and we still keep the performance benefits of a stateful design.

I think instead of "ANP should use packet-based semantics", it should be

"ANP should be stateful and must re-evaluate active connections when policies change."

[danwinship]

I think we should make a larger study of the ecosystem to understand how this is widely implemented across the industry

I was originally in favor of pushing everyone to the "connection-based" behavior, but there are OpenShift customers who would apparently be pretty upset if we didn't have the "out-of-policy connections get broken immediately when policies change" behavior, whereas it seems unlikely that anyone is strongly in favor of the opposite behavior, so "packet-based" seemed easier to standardize on.

When a policy changes, the implementation can simply "forget" the state of any existing connections that are no longer allowed.

(Yeah, I mentioned that (and I think that's what OVN actually does).)

I think instead of "ANP should use packet-based semantics", it should be

"ANP should be stateful and must re-evaluate active connections when policies change."

👍

This is the classic Stateful vs Stateless debate on firewalling.

Not entirely; NP implementations have to be stateful anyway since there's always an implied "plus the corresponding return traffic" on every rule.

[npinaeva]

In the context of protocols extension: if we ever implement ICMP or any other connection-less protocol, we have to use connection-less logic?

[danwinship]

I would say that you don't really need to think about it in that case. But yeah, we need to avoid implying that "ANP only applies to connection-based protocols".