Design questions: Is it possible to, 1) bypass the gateway when using headless services in a multi-cluster setup, and 2) add load balancing support for headless services?

[rng2]

Hi, I’ve been researching how linkerd (and service meshes in general) works with k8s headless services, and have some questions. Apologies beforehand if I miss any design constraints. I’m fairly new to the whole k8s + service mesh space.

First one is about multi-cluster. According to this blog post, multi-cluster-kubernetes-with-headless-services, when routing to a headless service in a remote cluster (say R), traffic has to first go through a dedicated gateway in R. This is because the mirrored Endpoints configure the gateway’s IP as the destination. Is this a fundamental requirement? Afaik, in certain k8s setups like GKE, pod IPs are first-class and are natively routable from anywhere within the VPC. So it seems that having each mirrored Endpoint points directly to its corresponding remote pod should work, in which case we save an extra hop. Is this design possible (assuming the context)? Or maybe we then lose out on some features that require going through the gateway?

Second question is also about headless services, but within a single cluster. The linkerd doc on load balancing states this:

If working with headless services, endpoints of the service cannot be retrieved. Therefore, Linkerd will not perform load balancing and instead route only to the target IP address.

Istio has the same constraint afaik. Again, is this a fundamental limitation with service meshes and k8s? Assuming the sidecar proxy on the client pod can utilize dns for service discovery, it seems that it should be able to retrieve the IPs of all the pods backing the headless service, and then do client-side load balancing between these IPs. I saw this approach mentioned in several blog posts on how to load balance grpc services in k8s in a non-service-mesh world. In fact linkerd is doing the same thing for grpc from my understanding; it just doesn’t support headless services.

Or if this is not possible (maybe the dns step doesn’t work the way I think), can we do something similar to what the Endpoint-mirroring controller for multi-cluster does? If it can discover/mirror all headless service Endpoints from a remote cluster, can’t the same logic be used for a local service? When the doc says

endpoints of the [headless] service cannot be retrieved

what exact part of k8s makes it so?

Or if the problem is that the proxy cannot tell whether the client pod wants to load balance between target pod IPs or to route directly to one at random, is it possible to add a mechanism that lets the client announce its “intent” to linkerd?

[mateiidavid]

Hey @rng2, great questions :) I'll try to answer them as concretely as possible, this is all based on my understanding of headless services after working with them for a while.

Background: in Kubernetes, most workloads are deployed as a set of pods. These usually tend to be ephemeral, which makes the management of IP addresses at higher levels of abstraction a bit of a nightmare to deal with. To work around this, we have clusterIP service objects that select over a group of pods. Since pod IPs change so often, services give us a long-lived DNS hostname (and IP addresses) to target in our applications. Behind the scenes, this target changes to one of the backing pods.

Headless services are an exception to the rule, and are primarily used with StatefulSet objects. The assumption here is that statefulset workloads, such as databases, tend to be long-lived. Since a stateful pod is long-lived, we can count on its IP address being present for a longer duration. Pods in a statefulset also have some interesting properties, such as predictable names and guaranteed scheduling ordering (i.e statefulset-0, statefulset-1). Headless services in these situations will NOT represent an entry point to our group of pods, they simply set up DNS records so that you may access any of the pods through DNS names. So, overly simplified, headless services allow us to create DNS records for pods (statefulset-0.my-headless-service.default.svc.cluster.local). In vanilla Kubernetes routing, you can either hit the headless service itself, in which case it'll route to any of the other hosts, or you can target pods directly by name. This is purposely done so that operators or engineers implement their own discovery mechanisms or load balancing (e.g redis would use a headless service to decide leader election and discover other nodes in a redis cluster).

Load balancing and service discovery: let's say a client A sends a request to a server B. Both of them are injected and in the mesh. Client A sends a request to http://foo:80 which corresponds to the DNS record of a service foo with IP address 10.0.0.1. Client A will build an http client in code, that client will do a DNS lookup and resolve foo to 10.0.0.1 and send a packet to it to establish the underlying connection. Linkerd intercepts the packet, and will do a lookup on 10.0.0.1 to find the service. Once it finds foo, it can also lookup its endpoints. We can do this because the hostname will always resolve to the service IP, which gives us quite a bit of metadata to work with. In normal Kubernetes, the packet would go to your host (node 1), and then it would be changed to an arbitrary endpoint through iptables.

Now, if you work with headless services, the situation is a bit different. Say foo is now a headless service, and it has two pods, pod-0: 10.0.0.1 and pod-1: 10.0.0.2. Your client will send a request to http://foo:80. Since foo does not map to a concrete IP address, your client will try to establish a connection with either 10.0.0.1 or 10.0.0.2. In normal Kubernetes, this would not be routed on the host through iptables -- it goes straight to the pod. With linkerd, we still lookup the service associated with the address, but since there is no service and we only get an endpoint, we route directly to the endpoint. If for example we had a way of extracting the name of the headless service based on the packet we intercepted, we could in theory collect all endpoints, but this introduces all kinds of complex situations, and the proxy imo shouldn't decide to route this request to a different endpoint if it already has an endpoint to return, if that makes sense.

Multicluster: it's true that in some distributions you can take advantage of a flat network topology and have access to any IP address, but we don't make that assumption. Multicluster should work for any arbitrary cluster topology imo, so it makes more sense to go through a gateway. We also embed the gateway's identity in the mirror endpoints so we can do TLS. It wouldn't be a problem I suppose to do that for pods, but it would just again introduce a lot of complexity.

Istio has the same constraint afaik. Again, is this a fundamental limitation with service meshes and k8s? Assuming the sidecar proxy on the client pod can utilize dns for service discovery, it seems that it should be able to retrieve the IPs of all the pods backing the headless service, and then do client-side load balancing between these IPs.

1. DNS will be resolved through your client.
2. We used to utilise DNS for service discovery but have since then moved away from it and are using native k8s resources: Endpoints and EndpointSlices. We also wouldn't be able to really do DNS discovery here, since the IP we get would already belong to an endpoint. If we would look up 10.0.0.1 we wouldn't get foo.default.svc.cluster.local, we'd get pod-0.foo.default.svc.cluster.local.

Generally, I think if you have a headless service, it means you want to control yourself which pod to send traffic to. This is more of an assumption k8s makes and we went along with. If you want to let your service mesh take over load balancing, or do load balancing implicitly through k8s' kube-proxy, then you need to use a clusterIP service. Otherwise, with headless services you can collect all of the IP addresses associated with a headless service's DNS name and maintain your own client side pool/service discovery, which seems to be what certain stores seem to implement.

Let me know if you have any questions. You can read through the research we have done on headless services and multicluster in #5162 (comment) which might be a bit more helpful.