|
| 1 | +--- |
| 2 | +layout: blog |
| 3 | +title: "Kubernetes 1.29: New (alpha) Feature, Load Balancer IP Mode for Services" |
| 4 | +date: 2023-12-18 |
| 5 | +slug: kubernetes-1-29-feature-loadbalancer-ip-mode-alpha |
| 6 | +--- |
| 7 | + |
| 8 | +**Author:** [Aohan Yang](https://github.com/RyanAoh) |
| 9 | + |
| 10 | +This blog introduces a new alpha feature in Kubernetes 1.29. |
| 11 | +It provides a configurable approach to define how Service implementations, |
| 12 | +exemplified in this blog by kube-proxy, |
| 13 | +handle traffic from pods to the Service, within the cluster. |
| 14 | + |
| 15 | +## Background |
| 16 | + |
| 17 | +In older Kubernetes releases, the kube-proxy would intercept traffic that was destined for the IP |
| 18 | +address associated with a Service of `type: LoadBalancer`. This happened whatever mode you used |
| 19 | +for `kube-proxy`. |
| 20 | +The interception implemented the expected behavior (traffic eventually reaching the expected |
| 21 | +endpoints behind the Service). The mechanism to make that work depended on the mode for kube-proxy; |
| 22 | +on Linux, kube-proxy in iptables mode would redirecting packets directly to the endpoint; in ipvs mode, |
| 23 | +kube-proxy would configure the load balancer's IP address to one interface on the node. |
| 24 | +The motivation for implementing that interception was for two reasons: |
| 25 | + |
| 26 | +1. **Traffic path optimization:** Efficiently redirecting pod traffic - when a container in a pod sends an outbound |
| 27 | + packet that is destined for the load balancer's IP address - |
| 28 | + directly to the backend service by bypassing the load balancer. |
| 29 | + |
| 30 | +2. **Handling load balancer packets:** Some load balancers send packets with the destination IP set to |
| 31 | +the load balancer's IP address. As a result, these packets need to be routed directly to the correct backend (which |
| 32 | +might not be local to that node), in order to avoid loops. |
| 33 | + |
| 34 | +## Problems |
| 35 | + |
| 36 | +However, there are several problems with the aforementioned behavior: |
| 37 | + |
| 38 | +1. **[Source IP](https://github.com/kubernetes/kubernetes/issues/79783):** |
| 39 | + Some cloud providers use the load balancer's IP as the source IP when |
| 40 | + transmitting packets to the node. In the ipvs mode of kube-proxy, |
| 41 | + there is a problem that health checks from the load balancer never return. This occurs because the reply packets |
| 42 | + would be forward to the local interface `kube-ipvs0`(where the load balancer's IP is bound to) |
| 43 | + and be subsequently ignored. |
| 44 | + |
| 45 | +2. **[Feature loss at load balancer level](https://github.com/kubernetes/kubernetes/issues/66607):** |
| 46 | + Certain cloud providers offer features(such as TLS termination, proxy protocol, etc.) at the |
| 47 | + load balancer level. |
| 48 | + Bypassing the load balancer results in the loss of these features when the packet reaches the service |
| 49 | + (leading to protocol errors). |
| 50 | + |
| 51 | + |
| 52 | +Even with the new alpha behaviour disabled (the default), there is a |
| 53 | +[workaround](https://github.com/kubernetes/kubernetes/issues/66607#issuecomment-474513060) |
| 54 | +that involves setting `.status.loadBalancer.ingress.hostname` for the Service, in order |
| 55 | +to bypass kube-proxy binding. |
| 56 | +But this is just a makeshift solution. |
| 57 | + |
| 58 | +## Solution |
| 59 | + |
| 60 | +In summary, providing an option for cloud providers to disable the current behavior would be highly beneficial. |
| 61 | + |
| 62 | +To address this, Kubernetes v1.29 introduces a new (alpha) `.status.loadBalancer.ingress.ipMode` |
| 63 | +field for a Service. |
| 64 | +This field specifies how the load balancer IP behaves and can be specified only when |
| 65 | +the `.status.loadBalancer.ingress.ip` field is also specified. |
| 66 | + |
| 67 | +Two values are possible for `.status.loadBalancer.ingress.ipMode`: `"VIP"` and `"Proxy"`. |
| 68 | +The default value is "VIP", meaning that traffic delivered to the node |
| 69 | +with the destination set to the load balancer's IP and port will be redirected to the backend service by kube-proxy. |
| 70 | +This preserves the existing behavior of kube-proxy. |
| 71 | +The "Proxy" value is intended to prevent kube-proxy from binding the load balancer's IP address |
| 72 | +to the node in both ipvs and iptables modes. |
| 73 | +Consequently, traffic is sent directly to the load balancer and then forwarded to the destination node. |
| 74 | +The destination setting for forwarded packets varies depending on how the cloud provider's load balancer delivers traffic: |
| 75 | + |
| 76 | +- If the traffic is delivered to the node then DNATed to the pod, the destination would be set to the node's IP and node port; |
| 77 | +- If the traffic is delivered directly to the pod, the destination would be set to the pod's IP and port. |
| 78 | + |
| 79 | +## Usage |
| 80 | + |
| 81 | +Here are the necessary steps to enable this feature: |
| 82 | + |
| 83 | +- Download the [latest Kubernetes project](https://kubernetes.io/releases/download/) (version `v1.29.0` or later). |
| 84 | +- Enable the feature gate with the command line flag `--feature-gates=LoadBalancerIPMode=true` |
| 85 | +on kube-proxy, kube-apiserver, and cloud-controller-manager. |
| 86 | +- For Services with `type: LoadBalancer`, set `ipMode` to the appropriate value. |
| 87 | +This step is likely handled by your chosen cloud-controller-manager during the `EnsureLoadBalancer` process. |
| 88 | + |
| 89 | +## More information |
| 90 | + |
| 91 | +- Read [Specifying IPMode of load balancer status](/docs/concepts/services-networking/service/#load-balancer-ip-mode). |
| 92 | +- Read [KEP-1860](https://kep.k8s.io/1860) - [Make Kubernetes aware of the LoadBalancer behaviour](https://github.com/kubernetes/enhancements/tree/b103a6b0992439f996be4314caf3bf7b75652366/keps/sig-network/1860-kube-proxy-IP-node-binding#kep-1860-make-kubernetes-aware-of-the-loadbalancer-behaviour) _(sic)_. |
| 93 | + |
| 94 | +## Getting involved |
| 95 | + |
| 96 | +Reach us on [Slack](https://slack.k8s.io/): [#sig-network](https://kubernetes.slack.com/messages/sig-network), |
| 97 | +or through the [mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-network). |
| 98 | + |
| 99 | +## Acknowledgments |
| 100 | + |
| 101 | +Huge thanks to [@Sh4d1](https://github.com/Sh4d1) for the original KEP and initial implementation code. |
| 102 | +I took over midway and completed the work. Similarly, immense gratitude to other contributors |
| 103 | +who have assisted in the design, implementation, and review of this feature (alphabetical order): |
| 104 | + |
| 105 | +- [@aojea](https://github.com/aojea) |
| 106 | +- [@danwinship](https://github.com/danwinship) |
| 107 | +- [@sftim](https://github.com/sftim) |
| 108 | +- [@tengqm](https://github.com/tengqm) |
| 109 | +- [@thockin](https://github.com/thockin) |
| 110 | +- [@wojtek-t](https://github.com/wojtek-t) |
0 commit comments