Fully Integrated Network Model with Cilium

[M4t7e]

Hello guys, I have the idea to introduce a fully integrated network model (aka flat model) for Cilium. The current Cilium integration doesn't take advantage of many of the most popular benefits, which is why people usually go for it. I would like to change that and try to achieve a best practice configuration, comparable to the big public clouds (e.g. GKE, EKS). See: https://cloud.google.com/kubernetes-engine/docs/concepts/gke-compare-network-models#fully-integrated-model

There is also a problem currently with a decoupled K3s/Hetzner CCM IPAM and the Cilium IPAM (possibly also with Calico, but I would leave Calico aside for now). Cilium completely ignores the K3s/CCM IPAM and uses its own IPAM instead. This can cause routing issues if the Cilium overlay is disabled or you want to natively route from pods to the outside of the cluster without maquerading. -> The back routes for various pod CIDRs pointing to the nodes (configured by Hetzner CCM) may differ from what Cilium assigned to them.

Goals:

• Enable Kubernetes host-scope IPAM mode (only K3s/CCM IPAM should be used) so that Cilium does not make its own IPAM
• Avoid NATing and overlay tunneling for better performance and less resource usage
• Enable native routing without masquerading to achieve a fully integrated network model
• Enable eBPF wherever meaningful to gain better performance (one of the biggest Cilium selling points)

Current Draft:

# Enable Kubernetes host-scope IPAM mode (required for K3s + Hetzner CCM)
# Ref: https://docs.cilium.io/en/stable/network/concepts/ipam/kubernetes/#configuration
ipam:
  mode: kubernetes
k8s:
  requireIPv4PodCIDR: true

# Replace kube-proxy with Cilium
# Ref: https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/
kubeProxyReplacement: true

# Enable Tunnel Mode or Native Routing Mode (supported by Hetzner CCM Route Controller)
# Ref: https://github.com/hetznercloud/hcloud-cloud-controller-manager/blob/main/docs/deploy_with_networks.md
#      https://github.com/hetznercloud/hcloud-cloud-controller-manager/blob/v1.17.1/hack/dev-up.sh#L144
#      https://docs.cilium.io/en/stable/network/concepts/routing/#native-routing
routingMode: "${var.cilium_routing_mode}"
%{if var.cilium_routing_mode == "native"~}
ipv4NativeRoutingCIDR: "${local.native_routing_ipv4_cidr}"
%{endif~}

endpointRoutes:
  # Enable use of per endpoint routes instead of routing via the cilium_host interface.
  enabled: true

loadBalancer:
  # Enable LoadBalancer & NodePort XDP Acceleration (direct routing (routingMode=native) is recommended to achieve optimal performance)
  # Ref: https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/#loadbalancer-nodeport-xdp-acceleration
  acceleration: native

bpf:
  # Enable eBPF-based Masquerading ("The eBPF-based implementation is the most efficient implementation")
  # Ref: https://docs.cilium.io/en/stable/network/concepts/masquerading/#ebpf-based
  masquerade: true

%{if var.enable_wireguard~}
encryption:
  enabled: true
  type: wireguard
%{endif~}

MTU: 1450

Remarks:

• devices is removed, because Cilium needs full control over all interfaces. Only eth1 would not work for most of these settings.
• I would also make ipv4NativeRoutingCIDR freely configurable (default network_ipv4_cidr), so that bigger routed network setups are possible (e.g. connect K3s Hetzner Network via VPN to other Projects, Clouds, DCs, etc.)

Cilium Status Output

Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
KVStore:                Ok   Disabled
Kubernetes:             Ok   1.26 (v1.26.7+k3s1) [linux/arm64]
Kubernetes APIs:        ["EndpointSliceOrEndpoint", "cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumCIDRGroup", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:   Strict   [eth0 167.235.251.118, eth1 10.0.127.229 (Direct Routing)]
Host firewall:          Disabled
CNI Chaining:           none
Cilium:                 Ok   1.14.0 (v1.14.0-b5013e15)
NodeMonitor:            Listening for events on 2 CPUs with 64x4096 of shared memory
Cilium health daemon:   Ok   
IPAM:                   IPv4: 5/254 allocated from 10.0.64.0/24, 
Allocated addresses:
  10.0.64.135 (health)
  10.0.64.148 (kube-system/hcloud-csi-node-v8dsx)
  10.0.64.155 (system-upgrade/system-upgrade-controller-68597fb8b-dqbp7)
  10.0.64.17 (router)
  10.0.64.250 (kube-system/kured-nccql)
IPv4 BIG TCP:           Disabled
IPv6 BIG TCP:           Disabled
BandwidthManager:       Disabled
Host Routing:           BPF
Masquerading:           BPF   [eth0, eth1]   10.0.0.0/17 [IPv4: Enabled, IPv6: Disabled]
Clock Source for BPF:   ktime
Controller Status:      32/32 healthy
  [...]
Proxy Status:            OK, ip 10.0.64.17, 0 redirects active on ports 10000-20000, Envoy: embedded
Global Identity Range:   min 256, max 65535
Hubble:                  Ok   Current/Max Flows: 4095/4095 (100.00%), Flows/s: 5.26   Metrics: Disabled
KubeProxyReplacement Details:
  Status:                 Strict
  Socket LB:              Enabled
  Socket LB Tracing:      Enabled
  Socket LB Coverage:     Full
  Devices:                eth0 167.235.251.118, eth1 10.0.127.229 (Direct Routing)
  Mode:                   SNAT
  Backend Selection:      Random
  Session Affinity:       Enabled
  Graceful Termination:   Enabled
  NAT46/64 Support:       Disabled
  XDP Acceleration:       Native
  Services:
  - ClusterIP:      Enabled
  - NodePort:       Enabled (Range: 30000-32767) 
  - LoadBalancer:   Enabled 
  - externalIPs:    Enabled 
  - HostPort:       Enabled
BPF Maps:   dynamic sizing: on (ratio: 0.002500)
  [...]
Encryption:                           Disabled        
Cluster health:                       6/6 reachable   (2023-08-01T06:14:24Z)
  [...]

What do you guys think? Does it make sense to go in this direction? Did I miss something or do you already see problems with the current draft? Also, if you see other points to consider here, please do not hesitate to mention them 🙂

PS: I'm not a Cilium expert, so a critical review would be more than welcome! I tried my best to create something good based on many docs, examples and tutorials.

[M4t7e]

Added endpointRoutes.enabled: true (draft updated) since it is now possible to use it together with BPF-based host routing. There was a problem with this combination before and got fixed in cilium 1.14

[M4t7e]

I think it could also be nice to provide an option to control the routingMode. In some edge cases there still could be a good reason to have a tunnel overlay, regardless of its overhead compared to a native routing setup. I updated the draft again and made the ipv4NativeRoutingCIDR configurable as I initially suggested.

[mysticaltech]

Most definitely! Super important.

[M4t7e]

I changed the setting names according to the latest deprecation warnings (see https://docs.cilium.io/en/latest/operations/upgrade/#deprecated-options).

This also gave me the idea that maybe we should control the Cilium version (e.g. cilium_version setting analogous to calico_version) so that we can always guarantee that it will work with the configuration provided by us. This can also help the user to fix a version number if a custom cilium configuration has been provided via cilium_values.

[mysticaltech]

Absolutely, super important! I thought we had the version flag for Cilium! 🤦🏻

[M4t7e]

It seems l7Proxy: false is not needed anymore, when wireguard is enabled. Cilium 1.14 added support for L7 proxy with wireguard: https://isovalent.com/blog/post/cilium-release-114/#h-wireguard-node-to-node-encryption-and-layer-7-policies-support

[mysticaltech]

Super interesting! Go for it @M4t7e, this should be a huge improvement. @kube-hetzner/core @ifeulner FYI.

[M4t7e]

Here is the corresponding PR: #930